# ATLAS Product Suite > Comprehensive technical documentation for Motion Applied ATLAS. Covering custom displays, data integration, APIs, and developer tools Source: https://atlas.motionapplied.com Full text of the documentation, ordered by site navigation. Each page is preceded by its title and canonical URL. Generated at build time by hooks/llms_txt.py. --- # Overview ATLAS (Advanced Telemetry Linked Acquisition System) is more than just a telemetry tool—it's a modular, open, and scalable data ecosystem built to meet the evolving demands of motorsport and high-performance engineering. Whether you're a race team, OEM, or technology partner, ATLAS empowers you to capture, analyse, and act on data in real time and at scale. ## What is ATLAS? ATLAS is a real-time data acquisition and analytics platform used across Formula 1, Formula E, IndyCar, NASCAR, and beyond. It enables teams to: - Capture telemetry from hundreds of sensors in real time - Distribute data across teams and tools via open APIs and streaming protocols - Analyse performance using custom dashboards, overlays, and integrations with MATLAB, Python, and cloud-native platforms ## Why choose ATLAS?
- __Open by Design__ --- ATLAS offers an open, API-first architecture that integrates easily with existing tools and workflows. - __Built for Performance__ --- It delivers fast startup, high-frequency data capture, and powerful visualisation tools trusted by elite motorsport teams. - __Flexible Deployment__ --- ATLAS supports local, cloud, and hybrid setups with scalable storage and licensing options to fit any team size or budget.
!!! tip "Interested in how you can use ATLAS?" If you're interested in how Motion Applied can advance your data systems, contact [sales@motionapplied.com](mailto:sales@motionapplied.com). --- # Use Cases in Motorsport and Engineering The ATLAS ecosystem is a powerful telemetry and data intelligence platform designed to support high-performance environments. Whether you're tuning a race car or validating aerospace systems, ATLAS provides the tools to ingest, analyse, and act on data in real time. Below are key use cases that illustrate how ATLAS is being applied across motorsport and engineering domains. ## Real-Time Telemetry for Competitive Edge in F1 In Formula 1, milliseconds matter. ATLAS enables teams to stream telemetry from ECUs, sensors, simulators, and wind tunnels with minimal latency. This real-time data flow allows engineers to monitor vehicle performance live during races or testing sessions. Teams use ATLAS to: - Stream live telemetry from ECUs, sensors, and gearboxes. - Analyse powertrain and gearbox performance in real time. - Securely share configuration files and telemetry across consortium members using RDA licensing and encryption. This ensures consistent control across hardware generations and protects proprietary data while enabling collaborative development. ## Formula E and Extreme E: Sustainability and Simulation ATLAS powers telemetry and control systems for Formula E and Extreme E, supporting electric powertrains and sustainable racing initiatives. Use cases include: - Battery management and diagnostics. - Integration with MATLAB/Simulink for predictive modelling. - Remote access to telemetry for offsite analysis. These capabilities help teams optimise energy usage and performance under strict sustainability constraints. ## NASCAR: Custom Telemetry Recorder and GPS Integration ATLAS has been tailored for NASCAR with a dedicated telemetry recorder that supports Vector GPS data and custom configuration formats. Teams benefit from: - Real-time location tracking and performance overlays. - Streamlined data ingestion compatible with ATLAS visualisation tools. This adaptation allows NASCAR teams to leverage ATLAS’s precision analytics in a high-speed oval racing context. --- # Eco-System Overview The ATLAS ecosystem is Motion Applied’s advanced telemetry and data management platform, purpose-built for high-performance motorsport engineering and adaptable to wider industries. It brings together powerful components for **hardware configuration, data distribution, storage, and visualisation**, ensuring that engineers can capture, process, and analyse telemetry data at speed and scale. ## System Monitor – ECU Configuration and Calibration At the foundation of the ecosystem is **System Monitor**, the primary tool for configuring and calibrating ECUs (Electronic Control Units). - Provides a multi-platform [Configuration API](../developer-resources/system-monitor/configuration_api/index.md), that enables seamless integration with client applications in various programming languages. - Supports the creation of **virtual parameters**, which are translated into open formats for downstream use. - Offers a secure and efficient environment for hardware configuration, serving as the **gateway between physical ECUs and the ATLAS data ecosystem**. System Monitor ensures that hardware is correctly defined and calibrated before data is distributed, making it the cornerstone of telemetry workflows in motorsport and other high-performance applications. ## Distribution – ATLAS Data Server, Bridge Service, and Stream API Telemetry flows through ATLAS using a set of tightly integrated services: - [**ATLAS Data Server (ADS)**](../key-functionality/stream/ads/index.md) Acts as the central hub for telemetry ingestion and distribution. ADS records live sessions, manages multicast delivery, and supports daisy-chaining across servers. This enables remote telemetry architectures, where distributed teams can access live data with no loss of performance. - [**Bridge Service**](../key-functionality/stream/open_streaming_architecture/index.md/#bridge-service) Real-time data is processed and distributed using modern broker technology (e.g., Kafka), replacing legacy multicast methods. This service ensures reliable, scalable delivery and supports cloud deployments. Engineers can also create calculated signals derived in real time—that are published back into the stream for consistent downstream use. - [**Stream API**](../developer-resources/secu4/stream_api/index.md) Exposes telemetry in a standardised, engineering-calibrated format using Google Protobuf. The API provides interfaces for both data producers and consumers, enabling seamless integration with external tools and workflows. The Bridge Service further enhances this by converting raw telemetry into usable engineering values while decoupling broker technology. ## Data Storage – SQL Race Sessions and Parquet Sessions ATLAS supports flexible storage of telemetry data in two complementary formats: - [**SQL Race**](../key-functionality/store/sqlrace/index.md) A session-based database optimised for motorsport workflows. SQL Race enables rapid querying and analysis of session data, making it a proven choice for race engineering environments. - [**Parquet**](../key-functionality/stream/open_streaming_architecture/index.md#historical-data-storage-parquet) A modern, columnar format optimised for big data analytics. Parquet sessions are well-suited for integration with external platforms, cloud pipelines, and long-term storage. Files can be exported at configurable intervals, with optional encryption for secure distribution. Together, these formats allow users to work within established motorsport practices while also taking advantage of scalable, analytics-driven workflows. ## [Visualisation – ATLAS Viewer](../key-functionality/analyse/viewer/index.md) At the user-facing end of the ecosystem, the ATLAS Viewer provides powerful tools for real-time visualisation and post-session analysis: - Live streaming of telemetry alongside historical playback - Flexible plotting, overlays, and comparisons across multiple sessions - Integration with external models and simulation data - Support for multiple data formats This combination of performance, flexibility, and usability makes ATLAS Viewer the engineer’s primary workspace for turning raw telemetry into actionable insight. --- # Ecosystem Guide Not sure where to start? This guide will help you find the right ATLAS components for what you're trying to do. No prior knowledge of the software needed. ATLAS is an ecosystem for working with telemetry data - the measurements that come from sensors, controllers, and simulation in high performance engineering environments. ``` mermaid graph LR A["Configure"] --> B["Stream"] B --> C["Store"] B --> E["Enrich"] E --> B B --> D["Analyse"] ``` | Stage | What Happens | Example | |-----------|-----------------|------------| | [Configure](../key-functionality/configure/system-monitor/index.md) | Setup hardware so it knows what to measure | Defining which sensors to log on an ECU | | [Stream](../key-functionality/stream/index.md) | Move live data from hardware to software | Telemetry flowing from a car to the pitwall | | [Enrich](../key-functionality/enrich/index.md) | Run models or calculations on data to create parameters | Running a Simulink aero model against live telemetry | | [Store](../key-functionality/store/index.md) | Save live data for later use | Archiving a race session for post-event review | | [Analyse](../key-functionality/analyse/index.md) | Visualise and explore the data | Overlaying two laps to find where time was lost | ## Storing Data There are two broad choices: files or databases. They are not mutually exclusive, some teams will use both but at different stages. === "Files" Self contained data files on disk. Copy, email, upload, put them on a USB stick. No server required. Pick files when: - Portability matters when you're sharing data with someone without access to your internal IT infrastructure - You frequently work offline with just a laptop - Batch processing is part of your workflow - Long time archiving on a network drive or S3 bucket is cheaper than keeping a database running - Interoperability !!! tip "Which format?" - SSN2 is ATLAS-native file with full session fidelity. Use when data is staying in the ATLAS ecosystem. - CSV is the universal fallback, any tool will work with CSV. But it's large and slow for big data sets and some configuration is necessary. - MATLAB is best if you frequently run models using Simulink and have control-system workflows. - Parquet is best for analytics, data science and cloud pipelines. Some configuration is needed to work with ATLAS. === "SQL Race Database" A purpose built database layer ontop of SQL Server or SQLite. Manage sessions, configuration and data. Pick SQL Race Database when: - You are capturing data from sensors in real-time - You need to browse, filter and sort sessions by metadata instead of folder structures - A team of engineers need concurrent access to the same session === "Other Database Structure - RTA" RTA is an architectural API layer that sits between ATLAS and any existing data store. Pick RTA when: - You already have your databases setup, with data and want to avoid a migration to SQL Server - You want flexibility of scaling and integration with other Cloud services - There are many other tools in your data pipeline that need to query the same data source ## Data Distribution When you need to get data from physical systems to downstream clients. === "ADS + RDS" The traditional ATLAS live data path. ADS accepts incoming telemetry (either by wirelink, radio or ethernet) and multicasts to downstream clients on the same local network. Pick when: - You need low-latency live data on a local network - Data is coming directly from physical systems or vehicles - Downstream consumers are on the same LAN - You want a simple, proven live telemetry path - You do not need cloud distribution or long-term buffering - Consumers are expected to be online at the same time as the data source === "Open Streaming" A Kafka-based streaming layer that decodes raw data from ADS and publishes it to defined Kafka topics. Pick when: - You need to read or write data on Kafka - You want to decouple producers and consumers - Data needs to be consumed by multiple independent downstream systems - You are building event-driven, scalable, or cloud-based architectures - You need replayability or durable streams - Consumers may be remote, offline, or join later - You want to ingest third party data sources === "RTA" A websocket interface for session data. Data can be buffered through Redis so clients can join at any point. Pick when: - You need to distribute session-based data in near real time - Clients need to join late and still receive buffered data - You want a browser- or UI-friendly interface (WebSockets) - You are supporting visualisation tools or dashboards - Kafka is not required, but live-ish access is - You want controlled fan-out without full streaming infrastructure === "Local File Share" Local files that can be sent manually to team members or uploaded to a network drive. Pick when: - Data volumes are small or manageable - Real-time access is not required - You need manual review or ad-hoc sharing - Consumers are humans, not systems - You want the simplest possible distribution mechanism - The workflow is offline, asynchronous, or low frequency ## Calculated Channels When you need to compute derived parameters. === "FDL" Built in expression language for ATLAS Viewer. Pick when: - Your calculations are simple and can be achieved with standard mathematical operations. === "C# .NET Functions" Custom functions written in C# can be embedded in XFN or compiled into a DLL. Pick when: - You have existing C# functions you wish to reuse without rewriting - You need advanced Object Oriented Programming functionality - You require multiple function outputs === "MATLAB Functions" Custom functions defined via .XFN that delegates execution to a MATLAB script. Pick when: - You have existing MATLAB functions you wish to reuse without rewriting into FDL - You want to use native MATLAB functionality to process complicated mathematical equations. === "vTAG Server" Software application that executes Simulink models against data, producing parameters that appear in ATLAS Viewer. Pick when: - You have Simulink models that you want to run - You have hardware running physically and you wish to augment by adding software modelling ontop - You wish to standardise models and ensure all downstream clients have the same virtual outputs - You are running an ADS setup === "Virtual Parameter Service" Microservice that can be deployed alongside Open Streaming to move calculations downstream of clients. Pick when: - You want to co-ordinate consistency with virtuals from a centralised location separate from each client - You need to monitor the throughput of virtual processing - You require performant calculation of virtual parameters separate from the load of downstream clients - You are running the Open Streaming Architecture ## Programmatic Access When you need to read, write or manipulate your data either from inside or outside of the ATLAS Ecosystem. === "SQL Race API" For reading, writing and managing session data. Pick when: - You do NOT need to automate ATLAS interactions. - You need to extract session data, regardless of session type - You do NOT need to load the session into ATLAS. - You want a direct, simpler integration focused on SQLRace session handling without UI or automation complexity. !!! tip "Supported session formats" - SSN - SSN2 - SQL Race Session === "Automation API" For controlling the UI of ATLAS Viewer programmatically. Pick when: - You need to automate ATLAS interactions. - You need to load session data into ATLAS. - The workflow involves programmatic control of ATLAS actions, such as loading or manipulating sessions. !!! tip "Supported session formats" - SSN - SSN2 - SQL Race Session === "Display API" For creating custom display plugins in ATLAS Viewer. Pick when: - You need to automate ATLAS interactions and - You want to add custom displays (visualization or UI enhancements). === "Stream API" For interfacing with the Kafka broker. Pick when: - You are using Kafka - You need live or continuous data streams - You need to consume data from Kafka topics - You need to publish data into Kafka topics - You are building event-driven or real-time systems !!! tip "Support Library" Support library is recommended to be used instead of the Stream API directly as it provides many helpful wrappers which can be used to simplify the process of interfacing with the broker. --- # Quick Start This page contains quick access to getting started with different component in the ATLAS ecosystem. ## Software Installation Guides | Component | Description | |--------------|-------------| | [ATLAS Viewer](../key-functionality/analyse/viewer/quick-start.md) | Visualise an analyse telemetry data. Open session files, build custom workbooks and compare runs. | | [vTAG Server](../key-functionality/enrich/vtag/getting-started.md) | Run Simulink models against live or historical telemetry. Derive new parameters without hardware changes. | | [ATLAS Data Server](../key-functionality/stream/ads/installation.md) | Distribute live telemetry across your network. Connects hardware to ATLAS clients over Ethernet or Wi-Fi. | | [System Monitor](../key-functionality/configure/system-monitor/getting-started/index.md) | Configure, calibrate and tune Motion Applied ECYs. Program units, setup logging and manage sensors. | | [SQL Race Management Studio](../key-functionality/store/sqlrace/installation.md) | Store and manage telemetry session in a central database. Bulk import, organise and share data across your team. | | [Standalone Stream Recorder](../key-functionality/analyse/standalone-stream-recorder/index.md) | Record live Stream API telemetry to SSN2 files or a SQL Race database, without running ATLAS Viewer as the recorder. | ## Developer Guides | Component | Description | |--------------|-------------| | [SQL Race API](../developer-resources/atlas/sql-race/integration.md) | Read and write telemetry sessions, parameters, events and laps directly | | [Display API](../developer-resources/atlas/display-api/initialization.md) | Build custom display plugins that run natively in side ATLAS Viewer | | [Automation API](../developer-resources/atlas/automation-api/getting-started.md) | Control ATLAS Viewer programmatically - load sessions, navigate displays and extract data. | | [Stream API](../developer-resources/secu4/stream_api/reference_docs/getting-started/installation.md) | Read and write to the live telemetry stream over gRPC and Protbuf. | | [Support Library](../developer-resources/secu4/support_library/index.md) | Helper library built ontop of the Stream API for building services. | | [Configuration API](../developer-resources/system-monitor/configuration_api/index.md) | Configure ECUs programmatically via a cross-platform gRPC interface. | | [Remote Telemetry Architecture](../developer-resources/rta/downloads.md) | Connect ATLAS to any existing database. Write a light weight data adapter and serve telemetry over REST. | --- # Security One of the key mechanisms used to protect customer data is RDA (Restricted Data Access). This system ensures that sensitive telemetry and operational data is only accessible to authorised users, maintaining confidentiality and compliance across all environments. ## What is RDA? RDA is a security layer within ATLAS that governs access to protected data streams. It acts as a gatekeeper, filtering telemetry and system data based on user permissions. If a user or team does not have the appropriate access rights, the data is automatically discarded before it reaches them—ensuring that sensitive information remains secure and unseen. ## How RDA Protects You **Access Control:** RDA enforces strict access policies, allowing only authorised personnel to view or interact with protected data. **Data Filtering:** Data streams are filtered in real-time by ATLAS bridge services. Any data tagged with RDA protection is excluded from delivery to unauthorised endpoints. **Third-Party Encryption Support:** RDA can be extended to support encryption from external sources, adding an additional layer of protection for integrated systems. ## Why It Matters Whether you're monitoring system performance, analysing telemetry, or integrating with external platforms, RDA ensures that your data remains protected. It’s part of our commitment to delivering secure, scalable, and trustworthy solutions for every customer. --- # System Requirements ## Operating System * Windows 11 Professional or Enterprise (64 bit) ## Recommended Hardware * 16 GB RAM or higher * Quad-core Intel Core i7 or equivalent * Dedicated GPU - NVidia Quadro M1000M or better ## Virtual Environments ATLAS will run on Server VMs with Remote Desktop. Enable GPU support where possible: * [VMWare](https://www.dell.com/support/article/uk/en/ukdhs1/sln288103/how-to-enable-a-vmware-virtual-machine-for-gpu-pass-through?lang=en) * [Hyper-V](https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/deploy/deploying-graphics-devices-using-dda) * [Citrix](https://www.citrix.com/content/dam/citrix/en_us/documents/go/reviewers-guide-remote-3d-graphics-apps-part-2-vsphere-gpu-passthrough.pdf) --- # Overview ATLAS (Advance Telemetry Linked Acquisition System) is a powerful software platform used to collect, process, and visualise data—especially in high-performance environments like motorsport. But don’t worry: you don’t need to be an engineer to get started. This guide walks you through the essentials. !!! tip "Tips for New Users" - Start with a simple session file and explore the display tools. - Don’t worry about acronyms, focus on what you want to see or understand. - ATLAS is modular, learn one part at a time. ## Core Concepts ### 1. The Journey of Data ATLAS is built around the idea that data flows through a journey—from being captured, to being stored, to being visualised. ATLAS provides tools for each step: - Capture: Collect data from sensors, simulations, or control units. - Store: Save that data in structured formats for later use. - Visualise: Display and analyse the data to make decisions. ### 2. Getting Started When you first open ATLAS, you’ll typically: - Load a historical session file (often called an .ssn file). - Build a display to view the data—think of this like setting up your dashboard. ### 3. Distribute and Capture This part of ATLAS helps you: - Connect to hardware like ECUs (Electronic Control Units). - Export and import configuration files. - Use vTAG, a tool that lets you run virtual models (e.g. from Simulink) and view the results in ATLAS. ### 4. Process and Analyse ATLAS supports: - Real-time data streaming. - Replay of past sessions. - Integration with tools like MATLAB or Python for deeper analysis. You can also compare multiple sessions side-by-side to spot trends or issues. ### 5. Store and Share Data can be stored in: - SQL Race: A database system for large-scale data storage. - SSN2 files: Local files for quick access. You can use SQL Race Management Studio to manage and share data across teams. ### 6. Visualise ATLAS gives you full control over how data is displayed: - Drag-and-drop interface to build custom dashboards. - Zoom, pan, and overlay data from different sources. - Use colour coding and annotations to highlight key events. ## Tools You’ll Encounter ATLAS includes a set of tools that work together to help you collect, analyse, and visualise data—each one designed for a specific part of your workflow. Here's a quick guide to what they do and when to use them. | Tool | What It Does | When to use it | | -------- | ------- | ------- | | [System Monitor](configure/system-monitor/index.md) | Manages projects and hardware connections | When programming ECUs | | [ADS](stream/ads/index.md) | Connects ATLAS to live data sources | When streaming live data from a vehicle or simulator | | [vTAG](enrich/vps/index.md) | Runs virtual models and integrates results | When running Simulink models with real-world data | | [SQL Race](store/sqlrace/index.md) | Stores and manages large datasets | For persistent storage | | [ATLAS](analyse/viewer/index.md) | Lets you create custom workbooks to visualise data | For visualising data in different displays | --- # Overview System Monitor is a software application for interfacing with Electronic Control Units (ECUs) and related systems. It supports trackside support, unit optimisation, dyno testing, development, and production. The application is optimised for Motion Applied systems but can be adapted for third-party controllers with hardware modifications. ## Key Features **ECU Program Management** - Upload and download ECU program files. - Ensure the latest program versions are available. **Tuning** - View and modify editable ECU parameters. - Apply changes instantly (Live Tune) or offline. - Supports automatic tuning with a Testbed. **Parameter Information Display** - Monitor measurement parameters for live feedback. - Access real-time data for diagnostics. **Data Logging Configuration** - Set up and manage data logging on TAG ECUs. - Customise logging for specific needs. **Sensor Configuration** - Calibrate sensors and actuators connected to the ECU. - Configure the DATALab system for advanced sensor management. !!! note Some features may require hardware modifications for third-party controllers. --- # Overview This guide helps you install and start using System Monitor. Below are the minimum and recommended system requirements: | Component | Minimum | Recommended | |-------------------|-------------------------------|-------------------------------| | **Processor** | Intel Core Duo 2 GHz | Intel Core i7 Processor | | **Memory** | 1 GB | 8 GB or higher | | **Disk Space** | HDD with 200 GB free space | High capacity SSD 512 GB+ | | **Operating System** | Windows 10 | Windows 10 | | **Installation/Data Transfer** | CD-ROM drive, FTP site | CD-ROM drive, FTP site | | **Power Supply** | Rapid battery recharge recommended (not trickle charge) while applications are running | Rapid battery recharge recommended (not trickle charge) while applications are running | ## 1. Install System Monitor Download the System Monitor installer from the [Motion Applied Portal](https://portal.mclarenapplied.com/portal/Downloads/Software#System%20Monitor) and run the .msi file to begin the installation. Three installation types are available: - **Typical:** Installs all files for supported ECUs. - **Custom:** Choose which components to install. - **Complete:** Installs all files. System requirements are checked during installation (memory, disk space, processor). Default install directory: `C:\Program Files\McLaren Electronic Systems\System Monitor 8` System Monitor icons are added to the default program folder: `\System Monitor\` You may be prompted to restart your computer after installation. ### Custom Installation Custom Installation lets you select components to install. Default settings match Typical Installation for first-time installs. Changed settings are remembered for future installations. Available options: - **Program Files:** System Monitor, CA-Card embedded code, TAGASAP conversion utility, SaxBasic. - **Help Files:** Install help system. - **Registry:** Modify Windows® Registry for licences and bases. - **Device Drivers:** Install drivers for hardware devices (e.g., CA-Card). - **DESKey Dongle Drivers:** Install drivers for DK2 USB, Parallel, and Network dongles. - **VE-82 Drivers:** Install drivers for VE-82 Pot Board (run `USBDriverInstaller.exe` manually after installation). ## 2. Insert Your CA-Card or Licence Dongle System Monitor can be licensed by: - Communications adapter card (CA-card) - Dongle - Network card address ### Adding a Licence Open the Add Licence dialog via **Help > Add Licence** or **Add Licence** in the Licence Required dialog. !!! note Later versions use a Licence File stored locally, defining teams and options. Licences are managed online at [Motion Applied Portal](https://portal.motionapplied.com/). The window lists added devices (type and serial number). Select a device to view its licence profile. - **Add New:** Browse for Licence Files. - **Remove:** Remove selected licence. - **Export:** Export the licence file. - **Devices:** List connected devices. - **Detect:** Search for accessible devices. - **Active Team:** Select from the drop-down list. - **Licence Details:** View licence profile summary. Applications listed under a team are unrestricted for that team. ![Licence](./assets/licence.png) ## 5. Start System Monitor ## 6. Select a Multi-Application Base or Configure a Unit.Def File Customer bases control file access to prevent incorrect ECU programming. Associate customer bases with a project using a Multi-Application base. - Select **File > Select Base**. - Choose a Multi-Application Base and click **OK**. ## 7. [Create a New Program Version for Each Customer Base](../projects/program-data-versions.md) Define Program and Data Version files for each Application in the Multi-Application Base. - Select **File > New**. - In the dialog, select **Program Version** and click **OK**. - Choose files to include in the Program Version. ## 8. [Create a Project](../projects/file-operations.md) A Project links all files needed to work on an ECU. - Select **File > New**. - Choose **Project** and click **OK**. - Complete the sequence of dialog boxes to define the Project, connection method, Desktop filename, and Program Version for each Customer Base. ## 9. [Connect to the ECU](../projects/ecu-communication.md) Manage the ECU connection the project uses. - Select **ECU > Connect**. - Choose the connection method (IP, HDLC, CAN). - Click **Connect**. --- # RDA The Reduced Data Access feature of System Monitor lets teams restrict sharing of data and setups between engine and chassis applications. Access is determined by the team's licence and can be one of three states: - `FullAccess` - `RestrictedAccess` - `NoAccess` If a team has restricted access to an application, parameters within that application are locked unless they appear on the Parameter Unlock List (PUL). Locked parameters remain stored in session data so users with full access can still view them; they are either visible or invisible. Teams with restricted access cannot add a locked parameter to the Desktop or view/edit its Properties. ## Team Security Team Security is a feature that prevents unauthorised users from accessing data associated with a team. Each team has a unique Team ID which is part of the licence file. More than one team is available for selection as the Active Team and can be changed at any time from the Licence Manager dialog. Team Security can be disabled for a specific Data Version by selecting: **File > Save as** and selecting the Disable Team Security check box in the dialog box. When connecting to an FIA Standard ECU, System Monitor verifies the Team ID of the Active Team. If the Team ID is valid, team functionality is enabled, System Monitor operates normally, and ECU tuning is permitted. If verification fails, ECU tuning is blocked until a **Clear and Reprogram** command is performed. Click **Yes** to clear and reprogram the ECU. After reprogramming, a clear logging command is sent to the ECU. Once the connection is re-established, the Team ID is validated and ECU tuning becomes available. If **No** is selected, access to the following commands is restricted: - ECU Online - ECU Live Updates - ECU Auto Tune - ECU Single Message ## Licensing Access to data is controlled by the licence management scheme. Each licence is a file stored on the local machine. Application qualifiers inside the licence file determine which applications a team has: - Full access to - Restricted access to - No access to (shown by prefixing the application name with `Locked:`) !!! note New licences are requested and generated online on the [Motion Applied Portal](https://portal.mclarenapplied.com/). Authorised users can request System Monitor and ATLAS licences, select a licence profile (teams, application access, ActiveX, etc.) and download the generated licence file. ## Parameter Unlock List (PUL) Each application has an associated Parameter Unlock List (PUL) containing parameters that teams with Restricted Access may access. Teams with Full Access can edit the PUL to add or remove parameters. PUL access is provided from the Unlock List Management tab of the Application Properties dialog. ## On loading a .dtv When a DTV is saved, a checksum of the PUL is stored in the file header. On loading a `.dtv` the stored checksum is compared with the checksum of the PUL on disk: - If the user is not authorised for Full Access: - If the checksums match: the appropriate parameters are unlocked. - If the checksums do not match: the user is prompted to locate an alternative PUL. - If no PUL file is found: all parameters for that application remain locked. - If the user is authorised for Full Access: no checks are made and all parameters are unlocked. Further team security is provided to prevent Data VersionFiles (.dtv) being loaded by other teams. Data Version files contain a list of Team IDs which are verified upon loading. The Data Version file will not load unless a valid Team ID is detected. ## Locked Parameters Teams with Restricted Access see parameter icons with a locked symbol superimposed (for example, a locked Measurement Parameter in the Explorer). Locked parameters are not removed from display windows they were previously added to — they are simply invisible. In a display window a locked icon appears in the status column and `-` is shown in the value column. --- # Overview System Monitor is a software tool used for real-time data monitoring, parameter editing, and logging in motorsport applications. It interfaces with ECUs and other data sources to provide engineers with flexible visualisation and control. ## Features - Real-time acquisition and display of measurement data - Editing of calibration parameters - Customisable desktop layout with multiple window types - Logging and export of data - Integration with Data Wizard and other analysis tools --- # Menus | Menu | Description | Shortcut | |---|---|---| | File | Work on Project files. Includes printing. | ALT+F | | Edit | Work on parameters and their values. Includes Parameter Explorer commands. | ALT+E | | View | Control the appearance of the Desktop. | ALT+V | | ECU | Work with ECUs. Shown when a Project is open. | ALT+U | | Logging | Work with logging configuration. Shown when a Project is open. | ALT+L | | Tools | Access additional tools within System Monitor. | ALT+T | | Data Wizard | Work with Data Wizard. Shown when a Project is open. | ALT+W | | Desktop | Create and edit Pages and Windows. Shown when a Project is open. | ALT+D | ### File Menu Options to manage files and projects. ![File](./assets/file.png) ### Edit Menu Options to edit projects and parameters. ![Edit](./assets/edit.png) ### View Menu Options to enable and disable different displays and toolbars. ![View](./assets/view.png) ### ECU Menu Options to interact with the connected ECU. ![ECU](./assets/ecu.png) ### Logging Menu Access logging configuration and controls. ![Logging](./assets/logging.png) ### Tools Menu Options to access additional tools. ![Tools](./assets/tools.png) ### Data Wizard Menu Access to Data Wizard features. Data Wizard compares the contents of the Edit Buffer with one or more previously created Data Versions, known as Candidate Data Versions. Using the Merge facility parameters in the current Data Version can be overwritten with those in the Candidate Data Versions. ![Data Wizard](./assets/data-wizard.png) ### Desktop Menu Options to customise the desktop layout. A Desktop consists of one or more pages that contain Windows. Every Project has a Desktop associated with it. Any Project can use any Desktop in the Multi-Application Base but can only have one Desktop loaded at any one time. ![Desktop](./assets/desktop.png) --- # Windows and Parameters ## Windows System Monitor supports multiple window types for displaying and interacting with parameters. ### Editable Windows Used to view and modify calibration parameters. These windows display parameter names, current values, units, and status indicators. Changes can be made directly in the value fields. Each type of Editable Parameter has a specific type of Window. The following types of Editable Parameter are supported: - Scalar - 1-axis map - 2-axis map - Array - String ### Measurement Windows Used to monitor real-time data from sensors and ECUs. These windows can display values numerically or graphically (e.g. strip charts, bar graphs). The data from Measurement Parameters can be displayed in the following types of display: - Big Number - Numeric - Bar Graph - Oscilloscope - Grid ### Creating Windows New windows can be created via the Desktop menu. Users can choose the window type and assign parameters during creation. ### Window Properties Each window has configurable properties: - **Title**: Custom name for the window - **Colours**: Background and text colour settings - **Axis Settings**: For graphical windows, axis ranges and labels can be defined - **Update Rate**: Determines how frequently data is refreshed ### Window Types - **Numeric Display**: Shows parameter values in a grid - **Strip Chart**: Plots values over time - **Bar Graph**: Displays values as vertical bars - **Gauge**: Simulates analogue gauge behaviour - **Status Indicator**: Shows boolean or state-based parameters ## Parameters Parameters are central to System Monitor’s functionality. They are either editable (calibration) or measurement (read-only): - Editable Parameters The information used by the embedded control system in the ECU to control its performance. Also known as Configurable or Tune data. Examples are: fuel injection quantity look-up tables; engine idle speeds; ignition maps; etc. Tuning is the process of optimising the values of Editable Parameters - Measurement Parameters The information gathered by the ECU from sensors or generated by its own operation. Also known as On-Line information. Examples are: engine speed; water temperature; etc. Logging or Data Acquisition is the process of obtaining this information. System Monitor includes tools to control Logging. ### Editable Parameters Any aspect of an ECU that can be changed in System Monitor is made available as an Editable Parameter. The type of Parameter is defined in the ASAP file by the ECU manufacturer. The ASAP file also permits or prevents System Monitor from changing the values of the Parameter. Editable parameters can be modified and sent to the ECU. They include: - **Name** - **Current Value** - **Units** - **Limits** - **Status** (e.g. locked/unlocked) Changes are made in editable windows and transmitted via the ECU menu. ### Measurement Parameters Any item of data that can be reported by the ECU, is available in the form of a Measurement Parameter. All Measurement Parameters are treated the same in System Monitor but they can be grouped according to their source. ### Parameter Properties Each parameter includes metadata: - **Data Type**: Integer, float, boolean, etc. - **Units** - **Conversion**: Raw to engineering units - **Update Rate** - **Display Format** ### Virtual Parameters A Virtual Parameter is a Measurement Parameter whose value is the result of applying mathematical operations to one or more existing Measurement Parameters. For example, a differential pressure parameter may be created from the difference of two absolute pressure values. These are useful for derived metrics or combining multiple inputs. ### Parameter Explorer A tool for browsing all available parameters. Supports filtering by name, type, and source. Parameters can be dragged into windows for display or editing. --- # Configuration and Advanced Settings ## Configuration System Monitor allows users to customise the desktop layout, window behaviour, and display settings. ### Desktop Layout Windows can be arranged freely across the desktop. Users can: - Create multiple pages to organise windows by function - Save and load desktop configurations - Cascade or tile windows for visibility ### Pages Pages act as tabs or layers within the desktop. Each page can contain a unique set of windows. Useful for separating calibration, diagnostics, and logging views. ### Fonts Font size and style can be adjusted globally or per window. This improves readability, especially in high-density layouts. ### Axis Properties Graphical windows (e.g. strip charts) allow axis configuration: - **Range**: Set min/max values - **Labels**: Custom axis labels - **Scaling**: Linear or logarithmic ### Conversion Properties Parameters may require conversion from raw values to engineering units. Conversion settings include: - **Scaling factors** - **Offsets** - **Lookup tables** (if supported) ## Advanced Tools ### Shortcuts System Monitor supports keyboard shortcuts for common actions: - **Ctrl+S**: Save desktop - **Ctrl+L**: Start/stop logging - **Ctrl+E**: Open parameter explorer Shortcuts can be customised in the settings menu. ### Mathematical Operators Used in virtual parameters and data analysis. Supported operators include: - Basic arithmetic: `+`, `-`, `*`, `/` - Logical: `AND`, `OR`, `NOT` - Comparison: `>`, `<`, `==` ### Number Formats Parameters can be displayed in various formats: - Decimal - Hexadecimal - Scientific notation Format is set per parameter or window. ### Parameter Unlock List Some parameters are locked by default. Unlocking requires appropriate permissions and may involve entering a password or key. ### Advanced Settings To open Advanced Settings, go to **Tools > Advanced Settings**. Some settings take effect immediately, some require reconnecting to the ECU, and some require restarting System Monitor — this is called out per setting below. Settings are grouped by area to make them easier to find. !!! note "Version note" Settings and defaults below reflect System Monitor 8.87.2. If you are running an older version, a setting's default may differ from what is shown here. #### Startup and project behaviour | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Enable Multiple Instances | Y / N | Allow multiple instances of System Monitor to run simultaneously on the PC. | Y | No | | Open most recently used project | Y / N | Open the most recently used project when System Monitor starts. | N | No | | Go online when opening project | Y / N | Put the ECU On-Line automatically when a project is opened. | N | No | | Warn if Program Versions do not match | Y / N | If Y, a mismatch prevents ECU being put On-Line and shows a message box; if N the message is ignored. | N | No | | Prompt for unit connection on opening project | Y / N | If N, automatically attempt connection to the last unit connection used for the project. | Y | No | | Enable livetune on opening project | Y / N | If Y, start livetune immediately after project open and unit connection. | N | No | | Default Explorer display to collapsed view | Y / N | If Y, the Parameter Explorer starts collapsed instead of expanded. | N | No | #### Data versions and team security | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Search base for versions | Y / N | Controls search base for versions (see ECU Search). | N | No | | Format data version text | Y / N | Control formatting of data version text (see ECU Bar). | N | Restart | | Enable Data Version Security | Y / N | Enable Team Security globally for data versions. | Y | No | | Directory for Parameter Unlock List files | Path | Directory used to list parameters accessible by restricted teams. Leave blank to use System Monitor's default location. | (blank) | No | | Search base for non-active versions | Y / N | When N, multi-application base is not searched for non-active application DTVs. | Y | No | | Enable Parameter Offset | Y / N | If N, disable the zero offset button and value offset input in parameter properties. | Y | No | | Auto save DTV changes | Y / N | Auto-save modified data versions when necessary without prompting; saved to next incremented version. | N | No | | Enable Enhanced RDA DTV unlocking | Y / N | Allow RDA settings on Enhanced RDA2 DTVs to be disabled. | N | No | | Disable DTV backup | Y / N | If Y, DTV backup files are not created/updated after each edit (performance improvement). | N | Restart | | Default Virtual Parameter format to XML | Y / N | If N, revert Virtual Parameter file save format to historical binary VPM format. | Y | Restart | | Disable Search for DTV on opening PGV | Y / N | If Y, disable multi-application search after PGV change (may improve performance). | N | No | | Use Encrypted Parameter Unlock List Files | Y / N | If Y, Parameter Unlock List files are read/written in an encrypted format. | N | No | #### Backups and file locations | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Save Document Backups | Y / N | Automatically save backup copies of documents. | N | No | | Directory for backup files | Path | Override default location for backup files. Leave blank to use System Monitor's default location. | (blank) | No | | Directory for configuration and macro files | Path | Location of logging configuration files and macros. Leave blank to use System Monitor's default location. | (blank) | No | | Directory for exporting logging files | Path | Location for exported logging configuration files. Leave blank to use System Monitor's default location. | (blank) | No | | Ident translation file | Path | Pathname of the file used to translate parameter identifiers when they change between versions. | (blank) | Restart | | File for sensor calibration | Path (.prm) | Pathname of the .prm sensor calibration file; System Monitor requires the correct path to locate it. Leave blank to use System Monitor's default location. | (blank) | No | | File for unit IP addresses | Path | Pathname of the Unit Definition file used to resolve unit IP addresses. Leave blank to use System Monitor's default location. | (blank) | No | #### Logging configuration | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Logging Config ID Seed | 0 – 3 | Seed applied to Logging Config ID to help ensure unique config IDs between units. | 0 | No | | Limit Logging To MAX_REFRESH | Y / N | If Y, enforce maximum refresh rates for parameter logging; if N, allow rates above max with warning. | N | No | | Session Row Logging Interval (ms) | Numeric (ms) | Interval for session row logging in milliseconds. | 1000 | No | | Directory for log files | Path | Location for System Monitor's own log files. Leave blank to use System Monitor's default location. | (blank) | Reconnect | | Enable log file per project | Y / N | Create log files unique to each project name to aid identification when switching projects. | N | Restart | | Automatic IP Message Logging | Y / N | If Y, a new message log is started when the project is opened or unit selection changes; if N, logging is controlled by the Record button. | N | Reconnect | | Message log level (for debugging) | Levels / Not Logging | Enable logging of System Monitor messages for debugging. | Not Logging | Reconnect | | Message log file size (MB) | Numeric (MB) | Sets the size limit for the message log. | (blank) | Reconnect | | Export Extended MESL Config Binary | Y / N | Enable MCB Writer to export a binary MCB containing logging/telemetry description and IDs. | N | No | | Log Single Messages | Y / N | If Y, log single messages from Active-X or GUI to individual log files. | N | No | #### Sensors | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Enable New Sensor Subset Upload | Y / N | Enable upload of new sensor subsets. | N | No | | Prompt New Sensor Subset Upload | Y / N | Enable a prompt when uploading new sensor subsets. | N | No | | Zero Offset Sampling Time (ms) | Numeric (ms) | Duration for sampling to compute an average zero offset value. | 5000 | No | #### Comms and ECU link | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Timeout before link lost (ms) | Numeric (ms) | Time System Monitor will attempt to send a message before treating the link as lost. | 2000 | No | | Read Ram View Timeout | Numeric (ms) | Timeout for Read RAM view operations. | 100 | No | | Use extended read RAM messages | Y / N | Use extended read RAMs if the unit supports them (faster large reads). | Y | Reconnect | | Use extended write RAM messages | Y / N | Use extended write RAMs if the unit supports them (faster large writes). | N | No | | Enable FIA lock checking | Y / N | Check for FIA-locked ECUs before reprogramming and prevent programming of not-permitted versions. | N | No | | Default message timeout (ms) | Numeric (ms) | Timeout for messages sent to the unit. | 200 | No | | Disable Encrypted Comms | Y / N | If Y, disables encrypted comms (enables network logging but disables some programming features). | N | Reconnect | | Preferred port for unit link | Port (hex/dec) | Default starting port for unit connection; if unavailable, the port is incremented and retried. | 0xC000 (49152) | No | | Increase ECU Bar display contrast | Y / N | Change ECU Bar text colours for greater contrast in bright conditions. | N | Reconnect | #### ECU programming and control | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Enforce Prime Parameter requirements | Y / N | Allow Prime Parameters to be toggled per Application via the Parameter Explorer context menu. | Y | No | | Enable Matlab Export after each parameter tune | Y / N | Control writing of Tune Log entries and Matlab export after parameter tunes. | N | No | | Enable Matlab Export after DataWizard Merge | Y / N | Control Matlab/Tune Log export after DataWizard merges. | N | No | | ECU Read Only mode | Y / N | When Y, user cannot modify values on the ECU. | N | No | | Prompt for confirmation on Remote Reset | Y / N | If N, remote reset executes without a confirmation prompt. | N | No | | Enable Multi Document Options in Open Dialog | Y / N | When N, open dialog does not offer multi-document options for CAN and remote logging configs. | Y | No | | Minimum default time before leave boot (ms) | Numeric (ms) | Minimum time before sending the leave boot message at end of programming. | 0 | No | | Enable Enhanced Slow Row | Y / N | Enables the enhanced slow row acquisition mechanism. | Y | Restart | #### Miscellaneous and integration | Setting Name | Options Available | Description of Setting | Default | Restart/Reconnect required | |---|---:|---|---|---| | Enable export of associated documents with project export | Y / N | Include Program Version documents when exporting a project to reduce manual inclusion. | N | No | | Enable Unit.Def configuration | Y / N | Replace Customer Base project management with Unit.Def file management. | N | Restart | | Suppress Licence Expiry Warnings | Y / N | If Y, System Monitor does not show warnings when a licence is due to expire. | N | No | | Enable Remote Configuration Upload | Y / N | Allow remote configuration to be uploaded from the ECU. | Y | No | | Enable FFC creation for PGV and Logging Configuration | Y / N | Enable creation of an FFC file alongside PGV and Logging Configuration files. | N | No | | Suppress 'Not live-tuneable' warning during import | Y / N | If Y, suppresses the warning normally shown when importing parameters that are not live-tuneable. | N | No | | Auto-increment filename for all numerically terminated files on save | Y / N | If Y, automatically increments a trailing number in the filename each time a file is saved. | N | No | | Enable Logging of API Calls | Y / N | Enable logging of calls made through System Monitor's API. | N | No | --- # Overview System Monitor has various safeguards to ensure that only the correct data and programs are transferred to an ECU. This is done either by using the Customer Base, which links the ASAP project to the file directories used by System Monitor or Unit.DefManagement. Each control system has associated ASAP and Hex files, which should be supplied by the ECU manufacturer. System Monitor generates Program and Data Version files from these ASAP and Hex files and only accepts ASAP files from the ASAP project specified in the Customer Base. The Program and Data version files are placed in the Customer Base directory. A System Monitor Project links the Program and Data version files for an ECU with the Desktop that you use to configure the ECU. When Customer Bases are used, each project is associated with a Multi-Application Base that contains one or more Applications. The Project files are placed in the Multi-Application Base Directory. When Unit.Def support is used, the project file location is used as the base directory. ![Project Illustration](./assets/project-illustration.png) ## Creating a Project A project requires a Program Version for each Customer Base in the Multi-Application Base therefore Program Versions must be created before you can Create a Project. The ECU manufacturer may supply Program Versions as well as the ECU files, in which case it will not be necessary to create the Program Versions. --- # Customer Base ## Customer Base A Customer Base links an ASAP project to the file directories used by System Monitor. It ensures that only files associated with the specified ECU are used. - Each Customer Base defines an Application within a Project. - Customer Bases are associated with Projects via a Multi-Application Base. - Editable via the Select Active Multi-Application Base dialog. ### Adding or Editing a Customer Base Use the Enter Customer Base Details dialog: - **Customer Base**: Unique name. - **ASAP Project**: Must match the project name embedded in the ASAP file. - **Base Directory**: Must be a sub-directory of the Multi-Application Base directory. Overwriting or removing a Customer Base that is part of a Multi-Application Base will break access to its data. ## Multi-Application Base Controls access to files and prevents incorrect ECU programming. - Contains one or more Customer Bases. - Typically set up by the control system supplier. - Can be added or removed by Free Users. ### Selecting a Multi-Application Base 1. Go to `File > Select Base`. 2. Choose from available bases in the dialog. ### Prime Parameter Override File (.ppo) Used to configure prime parameters in logging setups. Format: `App id, parameter identifier, minimum frequency1,vCar,200 1,tTotalShiftTime,500` - No header line. - Minimum frequency of 0 removes the prime property. - Requires "Enforce Prime Parameter Requirements" to be enabled. ## Free User Status A Free User can create and remove Multi-Application Bases. If you have Free User status, you will see a Multi-Application Base with the name ‘Free User’ in the Select Active Multi-Application Base dialog box and the Add and Remove buttons in the Multi-Application Bases panel are active. Free Users can select a Multi-Application Base in the usual way. However, if you select ‘Free User’ as your active Multi-Application Base, you will be able to load any ASAP file without restriction. You will also have access to any directories, under the Free User Base Directory, to store any System Monitor files that you create. --- # Unit.Def Files Unit.Def files are XML-based configuration files that define project options. They replace the legacy Multi-Application and Customer Base structure. There can be one Unit.Def file per folder and it is applied toprojects opened in that folder. The absence of a Unit.Def file will result in System Monitor resortingto default options for any projects located in the folder. ## Configuration - One Unit.Def file per folder. - Applied to all projects in that folder. - Managed via `File > Manage Unit.Def Files…` (only when no project is loaded). ## Applications Applications can be added manually or auto-configured by scanning PGV files in subfolders. - Each application requires an ASAP definition name and default folder. - Applications can be enabled/disabled or deleted. - One application can be set as the master. ## Options - Define default locations for project files. - Override comms path settings. - Specify a Prime Parameter Override (.ppo) file. ## Saving Changes are saved when selecting a new Unit.Def file or exiting the dialog. ## Opening Projects Three methods: 1. `File > Open… Ctrl+O`: Opens from the most recent Unit.Def folder. 2. `File > Open… >`: Browse any folder. 3. Recent Projects list: Quick access to last 10 projects. System Monitor looks for a matching Unit.Def file in the selected folder. --- # Creating Versions Each ECU contains one or more controllers, which require programming before use. The ECU manufacturer usually programs the controllers before despatch, however System Monitor can reprogram the ECU. SystemMonitor also uses the program to recognize the configuration of the ECU. A controller used in this type of application is known as an embedded control system. A description of the embedded control system is supplied in the form of an ASAP file and a Hex File. The ASAP file contains the descriptions of the embedded control system and the Hex file contains the program code and program data. ## File Types - **Program Version (.pgv)**: Contains ASAP content, program code, and configuration. - **Data Version (.dtv)**: Contains editable parameters and memory layout. ## Creating Program Versions 1. Start System Monitor. 2. `File > New > Program Version`. 3. Select required files: - ASAP2 (required) - Hex (required) - Controllers (required) - Optional: Errors, Events, Sensors, etc. 4. Enter comments and notes. 5. Click OK to generate `.pgv` and `.dtv`. ![New Program Version](./assets/program-version.png) ## Command Line Creation Syntax: ``` SMv7U /newpgv /asap2 "pathasap2"/hex "pathhex " /controller"pathcontroller" /error"patherror" /event"pathevent" /sensor"pathsensor" /endis"pathendis" /injector"pathinjector" /prm"pathprm" [/customer basename] [/location folder] [/encrypt encryptionmode] ``` - **pathxxxx**: The full path name of the following files: - **pathasap2**: Asap2 File. Must come from the ASAP project specified in the Customer Base. - **pathhex**: Hex File. Must match the ASAP2 file and be suitable for the ECU. - **pathcontroller**: Controllers File. Must be suitable for the ECU. - **patherror**: Errors File. Required if you want to view Errors. - **pathevent**: Events File. Required if you want to view Events. - **pathsensor**: Sensors File. Required if a Sensor Injector file is loaded (See Sensors). - **pathendis**: Sensor Enable/Disable File. Required if you want to Enable and Disable sensors. - **pathinjector**: Sensor Injector File. Required if the Sensors file includes injectors. - **[/customer basename]**: Optional. The name of the Customer Base in which the Program Version is to be stored. - **[/location folder]**: Optional. The location Base in which the Program Version is to be stored. - **[/encrypt encryptionmode]**: Optional. The encryption mode should be either: - `0`: No encryption - `1`: Encrypt PGV files on creation. If this parameter is not specified, no encryption is used. ### Return Codes `0` success, `1` licence error, `2` invalid command, `3` command failed. --- # File Types System Monitor uses several different files in the following categories: - [ECU Files](#ecu-files) - Embedded software files distributed by the TAG customer support team or by the ECU manufacturer. Not part of System Monitor. - [Program and Data Version Files](#program-and-data-version-files) - The System Monitor working files. Generated by System Monitor from the ECU files. - [Project Files](./file-operations.md/#exporting-and-importing-files) - Files generated by System Monitor to store its own configurations. System Monitor stores project files in a structured directory layout under the Multi-Application Base. ## Directory Layout ``` \MultiAppName\ ├── CustBaseName\ │ └── version\ │ ├── version.pgv │ └── version000.dtv ├── Desktops\ │ └── ProjectName.dtp ├── LogCfgs\ ├── Sessions\ ├── Virtuals\ ├── Work\ ├── SenSets\ ├── HistReps\ └── ProjectName.prj ``` ### Description - **MultiAppName**: Top-level directory for the Multi-Application Base - **CustBaseName**: Sub-directory for each Application - **version**: Contains Program and Data Version files - **Desktops**: Stores Desktop files - **LogCfgs**: Default location for logging configurations - **Sessions**: Stores session logs from Live Logging - **Virtuals**: Stores virtual parameter files - **Work**: Stores event logs - **SenSets**: Stores sensor and actuator configurations - **HistReps**: Stores history report files - **ProjectName.prj**: Main project file Each new project creates its own `.prj` file and associated subdirectories. ## Program and Data Version Files The Program and Base Data Version files are generated by System Monitor when a New Program Version is created. ### Program Version File (.pgv) Encapsulates all program-related artifacts for a program version. - Typical contents (generated by System Monitor): - Full content of the ASAP2 (.a2l) file. - Program code for each ECU controller (from the HEX file). - The content of any additional files selected during creation. - Unique program version stamp (version). - Base directory and program version directory and filename. - ECU type and names of the original .hex and .a2l files. - Free-form comment/notes. ### Base Data Version File (.dtv) Contains the base set of calibration/data values and related metadata. - Typical contents: - Data for each controller forming the base parameter set (from HEX). - Data-area memory layout (from ASAP2). - Short and long comments/notes and the creating user name. - Unique stamp linking to the associated .pgv (program version). - Base directory, program version directory and filename. - Unique data version stamp and the default filename (version000.dtv). ## ECU Files ECU files are the embedded software files for the ECU, distributed by the customer support team or by the ECU manufacturer. Not part of System Monitor. - Required by System Monitor: - ASAP2 (.a2l) — ASAP2 project/description file required for mapping parameters and memory layout. - HEX (.hex) — compiled program + data; provides program code and controller data. - Controllers (.ini) — controller list and per-controller properties/IDs. - Optional (may be supplied by customer support or manufacturer): - Errors (.ini), Events (.ini), Adjustment Parameters (.ini) - Sensors (.ini), Sensor Injector (.ini) - Sensor Enable/Disable (.ini) - Sensor PRM (.prm) — paths/filenames for ASCII calibration data - Live Auto Tune (.ini) — tuning-related configuration (TBC) ### File Content - ASAP2 (.a2l) - Project name/security reference and ECU type. - Version string used for naming/version directory. - Definitions for measurement and editable parameters, groups, conversion formulas, axes, and memory location information for controllers and parameters. - HEX (.hex) - Program code for each ECU controller. - Controller configuration including revision/date/data-version stamps and data area sizes/locations. - Base data values used as the initial calibration set. - Controllers (.ini) - Defines the list of controllers (including possible external controllers), their names, IDs, ident numbers (for cross-reference to ASAP2), and controller properties. - Sensors and Related Files - Sensors (.ini) — lists available sensor types and selected sensors; defines sensor data memory for controllers. - Sensor Enable/Disable (.ini) — sensors that can be toggled on/off by ECU. - Sensor PRM (.prm) — links to ASCII calibration files for sensors. - Live Auto Tune (.ini) — placeholder for live autotune configuration details. --- # Application Management ## Applications Applications represent functional areas of the ECU system, such as: - **BIOS** – core control system - **Chassis** – vehicle chassis control - **Engine** – engine control Each application is linked to a Customer Base and can be set as Active or Read Only. ## Active Applications A Project contains one or more Applications. System Monitor controls which applications are affected by various operations, especially those involved in Developing Data Versions. The applications that are affected are called the Active Applications. The Active Applications are shown bold in the Parameter Explorer and the ECU Bar. ### Managing Active Status - Use `File > Active Application` or right-click in the Parameter Explorer. - Shortcut: `ALT+A` - In Parameter Explorer or ECU Bar: toggles active status. - In Main Window: cycles through applications if only one is active. !!! note Changes can only be made when the ECU is Off-Line and Live Updates are disabled. ## Read Only Applications Applications can be set to Read Only via the File Menu or Context Menu. An application that is both Active and Read Only cannot be edited but is included in operations like reprogramming. ## Application Properties Accessible via `File > Properties > Application`. Tabs include: - **Program Version** – shows file details and source files. - **Data Version** – shows current data file and metadata. - **Memory Layout** – displays memory usage. - **Unlock List Management** – controls access to parameters. ## Unlock List Management The Unlock List Management tab allows teams to Manage the Unlock Lists associated with the Application. Unlock lists are managed on a per Application basis, if the user is authorised for full access to that application. - Unlock lists are managed per team. - Parameters can be added or removed via the Select Parameters dialog. - Once locked, an application cannot be unlocked. Backups should be created beforehand. !!! note Before locking an application, you should ensure that a backup of the unlocked application is created, as you cannot unlock an application once it is locked. Due to the unlocked state of this application, you should store it in a secure location. --- # Data Version Development When a project is Created a Base Data Version is allocated to each Application in the project. The Base Data Version contains a full set of Editable Parameters with values that should enable the vehicle to operate, if not at its optimum. As the ECU is tuned to optimize the performance of the vehicle, the values of the Editable Parameters are changed. The changes are stored by System Monitor in the Edit Buffer. If the ECU is On-Line, the contents of the Edit Buffer can be sent to the ECU either immediately (using Live Tune) or on demand. **Features** - The Data Version in the ECU can be uploaded to the Edit Buffer. - Only Active Applications can be tuned. Ensure that the applications that you want to tune are set as active and are not read only. - The contents of the Edit Buffer can be saved at any time, to create a new Data Version File. If the contents of the Edit Buffer are different from the currently loaded Data Version, you will be prompted to save the new Data Version when the ECU is taken Off-Line or the project is closed. Data Versions are saved in the Customer Base directory of the Active Application. - A Backup of the Edit buffer is stored as: \MultiAppName\CustBaseName\version\version000xxx.dvb where: xxx is the user name of the person who is currently logged on. See Project Files for a description of the file structure. The location can be overridden by an entry in Tools > Advanced Settings. ## Using Data Version Files To load a different Data Version: 1. Select the Active Application. 2. `File > Open > Data Version (.dtv)` 3. Browse and select the file. Changes to the Edit Buffer prompt a save when the ECU is taken Off-Line or the project is closed. ## Saving Data Versions - File name format: `versionnnn.dtv` - `nnn` is a three-digit hex number incremented with each new version. - Comments and notes can be edited before saving. - Old versions are retained unless overwritten manually. !!! note - File names and location may be changed but the location can only be within the Multi-Application Base directory. - Comment and Notes in the Data Version panel can be edited before the file is saved. - Old Data Version files are not overwritten except to change the Comment and Notes. ## Export to MATLAB The MATLAB export creates one or more MATLAB script files (*.m) from the Data Version currently in the Edit Buffer. These files can be used for analysis in MATLAB or produced by MATLAB for re-import into System Monitor (directly or via Hex files). Export options: - Definition and Data Includes Property settings and the value(s) for each Editable Parameter. - Data only Includes only the value(s) for each Editable Parameter. How to export 1. Select the Active Application. 2. Choose: Tools > Export to MATLAB > Definition and Data or Data only. 3. In the Save As dialog, confirm the Save as type is "MATLAB Script Files (*.m)". A separate .m file is created for each Active Application. 4. Choose a folder and file name, then save. - Default file name: progversiondataversion.m - progversion: Program Version number used by the Application - dataversion: Data Version number used by the Application (including its sequential number) - The Save As dialog opens at the default path (the location of the currently loaded Data Version). You may store the file anywhere you have write access. !!! tip - Use "Definition and Data" when you need parameter metadata (units, conversions, properties). - Use "Data only" for a compact export of values for numerical analysis or scripting. - Confirm Active Applications are the ones you intend to export; only Active Applications produce files. - Ensure you have write permissions for the chosen export folder. ## Importing Hex Files System Monitor creates Data Versions from the information in Hex files (see Program and Data versions and ECU Files). This is normally done as part of the process of Creating a Program Version. However, it is possible to import a Hex file into an existing Project to create a new Data Version. To do this select: **File > Import Hex File** to browse for a Hex file. If the Hex file is compatible with the project, a new Data Version is created. This facility can be used to incorporate development data from other applications such as MATLAB into System Monitor. --- # Data Wizard Data Wizard compares the contents of the Edit Buffer with one or more previously created Data Versions, known as Candidate Data Versions. Using the Merge facility parameters in the current Data Version can be overwritten with those in the Candidate Data Versions. **Features** - Compare/Merge Tool - Candidate Version Window - Report View Window ### Compare/Merge Tool The Compare/Merge Tool compares Parameters from one or two Candidate Data Versions with the Merge Buffer, and allows values from a Candidate Version to be Merged into the Merge Buffer and then committed to the Edit Buffer. #### Merge Values 1. On the top-row drop-down list, select the Applications base to compare. 2. On the Dtv 1 row, browse for a Data Version or select one from the recently compared drop-down list. Note: Dtv1 and Dtv2 Data Versions must share the same Program Version. 3. On the Dtv 2 row, browse for or select a Data Version. 4. Select a display criterion for parameter values from the drop-down list (for example, Value, Delta Value). 5. The main window displays the differences between the Data Versions selected in Dtv 1 and Dtv 2 and the Merge Buffer. 6. To update a parameter value in the Merge Buffer with a value from one of the other Data Versions, click the `>` or `<` column in the appropriate cell. Click the same cell again to undo the change. !!! note Merge Buffer values can be edited directly in the Memory column. If the Merge Buffer differs from the Edit Buffer, an asterisk (*) is displayed in the Memory column header. #### Commit Changed Values 1. To commit all changes in the Merge Buffer to the Edit Buffer, click Commit. 2. The display is updated to show only those parameters that are still different. ### Map Compare Window The Map Compare Window displays values for Dtv 1, theCurrent Data Version and Dtv 2. The window mimics the Map Window for theParameter, displaying the relevant number of Breakpoints on all axes. To open a Map Compare Window, double-click a 1-Axisor 2-Axis Parameter in the Compare/Merge Tool Window. ### Minimum/Maximum Data Versions If a minimum and/of maximum Data Version file is available in the Program Version folder, it is accessed by System Monitor and used in the Compare Merge Tool to indicated if minimum or maximum values have been exceeded. Minimum and maximum Data Version files (.dtv) can be used as Dtv1 and Dtv2 Data Versions in the Compare/Merge Tool. To create a Minimum or Maximum Data Version: - Edit a Data Version and save it using the filenames - MIN.dtv for the minimum Data Version - Max.dtv for the maximum Data Version where pppp is the Program Version number. !!! note Save the min and max Data Versions in the same folder as the Program Version. ## Candidate Data Versions The Candidate Data Version is a previously created Data Version that you want to compare with the contents of the Edit Buffer. Features: - Individual Candidate Data Version Windows can be printed but are not included if you print a Page. - Candidate Data Version Windows are not stored as part of the Desktop and are not reloaded if a Project is closed and re-opened. - More than one Candidate Data Version Window may be opened at once. Merge operations apply to all the open Candidate Data Version Windows. - The display of Names and Descriptions reflects the setting in the Parameter Explorer. To open a Candidate Data Version Window: 1. Select: Data Wizard > Open Data Version and select a file *or* 2. Select a Data Version file from the Recent Files list on the Data Wizard Menu. *or* 3. Select File > Open to select the Data Version file. DO NOT open the file. Click Data Wizard. The Candidate Data Version Window appears on all pages. The full path name of the Data Version file is displayed on the Title bar. ## Report View The Report View Window shows the parameters in the Edit Buffer. These are compared to the same parameters in the current Data Version using the Different Value Mask and shows shows all changes made to application since data version was opened / last saved. The Report Window appears on all pages. The full path name of the Data Version file is displayed on the Title bar. To avoid confusion, it is advisable to use Data Wizard on a blank Page. To open a Report View Window: 1. Select: Data Wizard > Report View. A Window is opened for each Active Application. *or* 2. Select an Application in the Parameter Explorer, access the Context menu and select: Report View. !!! note Report View Windows are not updated automatically. If you change any parameters in the Edit Buffer and want to force an update, select: Data Wizard > Compare / Compare All. ## Parameter Masking Filters parameters by type of difference: - Absent - Different Definition - Different Size - Different Data Type - Different Conversion - Different Units - Different Value - Equal ## Merge Operations Merging attempts to overwrite parameters in the Edit Buffer, with the definition of the same parameter in a Candidate Data Version. If a parameter has a Different Definition, the parameter is not merged 1. Select: `Data Wizard > Merge` or `Merge All`. - **Merge** - Merges the current Data Version with the Candidate Data Version(s). The Selected Parameters and the Parameter Mask are not changed. Only the parameters displayed in the open Candidate Data Version Window(s) are merged. - **Merge All** - Merges the current Data Version with the Candidate Data Version(s). Before the merge operation is carried out, the Selected Parameters and the Parameter Mask are reset to their default values. All parameters are merged whether or not they are displayed in the open Candidate Data Version Window(s). 2. If the difference between the parameters is their value, conversion, or units, the raw bit value (the value before Conversion) is used for the merge. 3. If the difference between the parameters is their data type, the scaled value (the value after Conversion) is used for the merge. 4. If the differences cannot be resolved automatically, the **Resolve Parameter Merge** dialog box opens. 5. Check marks show which differences have been found for the parameter named at the top of the dialog box. 6. Select a method to merge the value by selecting from the available radio buttons. See dialog box. 7. If more than one Candidate Data Version is open, and the same parameter appears in more than one of them, the **Resolve Merge Conflict** dialog box opens. 8. The list shows the full path names of the candidate Data Version files that contain the parameter named at the top of the dialog box. 9. Select the file from which you want to merge the parameter and click OK. 10. If you select Cancel, the parameter is not merged and remains unchanged. ## Viewing Parameters Data Wizard allows temporary Windows to be opened to display the values of selected Parameters. ## Window Menu The Data Wizard Window Menu commands control the appearance of a Report View or a Candidate Data Version Window. --- # ECU Connection The Connections tab is used to view and modify the way System Monitor is connected to the ECU. To open the Connections tab select: **File > Properties > Connections** or use the ECU Menu. ## Connection Methods - **IP (Ethernet)** – TCP/IP connection - **HDLC (CA-Card or PCMCIA)** – Serial connection - **CAN (VECTOR card)** – Uses VECTOR XL Driver Library ### IP via Ethernet 1. Select `IP via Ethernet` in the Connections dialog. 2. Choose a Unit Definition for the main unit. 3. Choose a Unit Definition for the spare unit. 4. Edit Unit Definitions if required. 5. Select an ATLAS data server. 6. Click `Next` to continue to Desktop setup. ## Unit Definition Defines ECU identity and IP configuration. ![Connections Dialog](./assets/connection.png) ### Adding a Unit Definition 1. Click `Add`. 2. Enter a name. 3. Select ECU type. 4. Enter IP address(es). 5. Click `OK`. ### Editing or Removing - Select the Unit Definition from the list. - Modify name, type, or IP address. - Click `Remove` to delete. ## Spare Unit Typically, development and Tuning is done using one unit and it may be necessary to transfer the settings (Program and Data versions) to another unit. If the ECUs communicate over an Ethernet network, each unit has an unique IP Address so that the unit(s) designated as the spare will have different address(es) from the unit(s) designated as main. The Unit Definition for the Spare unit is defined for the Project in the New Project - Connections dialog box or the Project Properties dialog box, Connections tab. ### Programming the Spare Unit 1. `Tools > Program Spare Unit` 2. Select applications to program. 3. Enable merged programming sequences if needed. 4. Select components to download. 5. Click `OK`. ### Checking Unit Status - `Tools > Unit Status` opens the Unit Status dialog. - Allows upload/download of files and inspection of current ECU contents. --- # File Operations System Monitor uses various file types for ECU configuration, logging, and tuning. ## Create a Project 1. Select `File > New`. The File New dialog box opens. 2. In the Document list, select **Project** and click **OK**. The Project dialog box opens. 4. Enter a filename in the **Filename** text box, or browse for a path or an existing project filename (browsing is limited to the Multi‑Application Base directory). 5. Enter a **Comment** and **Notes**, if required. 6. Click **Next** to continue. The Connection dialog box opens — see *New Project - Connections* for further instructions. ## Opening Files Use `File > Open` to access: - Data Version (.dtv) - Program Version (.pgv) - Desktop (.dtp) - Logging Configurations - Sensor Sets - History Reports - Virtual Parameters - CAN Config - Project (.prj) If no project is open, only `.prj` files can be selected. ## Saving Files ### Save - Saves changes under the original name. - Overwrites the existing file without warning. ### Save As - Saves under a new name. - Allows editing of comments and notes. - For Data Versions, prompts for a new name unless only comments were changed. ### Save Copy As - Saves a copy under a new name. - Original file remains unchanged. ### Save All - Saves all modified files under their original names. ### Save All As - Saves the entire project and all associated files under a new name. ## File Type Selection Appears when using New, Open, Save, or Save As commands. Lists available file types for the selected operation. ## Exporting and importing files Use Export and Import to copy Project, Program and Data Version files (.pgv, .dtv, etc.) between Multi‑Application Bases. The export format is .smx, which can only be read by System Monitor’s Import facility. ### Exporting files 1. Open: File → Export → "Export Files from Customer Base" dialog. 2. The **Active Multi‑Application Base** panel shows read‑only info about the current Base. 3. In **Export to file**, enter or browse for the destination path and filename (use the .smx extension). 4. The **Files to export** list shows files that will be included. 5. Click **Add** to open a File Open dialog and select files from the Multi‑Application Base. If you add a project .pgv file, all files associated with that project are included. 6. To remove an item, select it and click **Remove**. 7. Click **Export** to create the .smx file and close the dialog. Note: Exported .smx files can only be read by System Monitor’s Import command. ### Export Project - Creates a snapshot of the current project containing all files in the project. - The file is named after the project with the .smx extension and is placed in the project’s directory. - No dialog is shown and there are no options. ### Importing files 1. Open: File → Import → "Import Files into Customer Base" dialog. 2. The **Active Multi‑Application Base** panel shows read‑only info about the current Base. 3. In **Import from file**, enter or browse to the .smx file to import. 4. The **Files to import** list displays the contents of the .smx file. Select files by checking their boxes, or highlight files and use **Include** / **Exclude**. 5. Click **Import** to extract the selected Program and Data Version files into the current Base directory and close the dialog. 6. Confirm or resolve any overwrite prompts as required by the application. --- # Overview The System Monitor is a data display tool designed to provide feedback during tuning by visualising Measurement Parameters. While it offers basic display capabilities, more advanced analysis is available through the [ATLAS visualiser](../../../../analyse/viewer/index.md). System Monitor presents data using various display windows, each tailored to different visualisation needs. These windows are configured to show live data from Measurement Parameters, and can be customised extensively via the Window Properties dialog. ## Display Types System Monitor supports multiple display formats to suit different data visualisation requirements. Each format uses a dedicated window type: ### [Numeric](numeric.md) Displays one or more Measurement Parameters in rows with descriptive text. ### [Big Number](numeric.md/#big-number-display) A variant of the Numeric display using a large font. Created automatically when a single parameter is selected via Parameter Explorer. ### [Bar Graph](bar.md) Displays parameters as vertical or horizontal bars. Bar height and colour reflect parameter values and warning limits. ### Dial Gauge Displays a single parameter on a dial-style gauge. ### [Oscilloscope](oscilloscope.md) Plots parameters against time, supporting multiple trace types (Linear, Digital, Logical). ### [Grid](grid.md) A spreadsheet-style layout allowing placement of text and parameter attributes in cells. Supports editable parameters and custom formatting. ### [Signal Sampling](signal.md) Provides functionality to sample or zero one or more input signal parameters. ## Adding Display Windows Display windows can be added to a System Monitor page via the menu: `Desktop > Window > [Display Type]` --- # Numeric The Numeric Display shows one or more Measurement Parameters in a tabular format. Each parameter is displayed in a row with descriptive text. ![Numeric](assets/numeric.png) ## Features - Each row includes: - **Name / Description**: Configurable via the Parameters Sheet in the Window Properties dialog. - **Value**: Display format set in the Parameters Sheet or System Menu. - **Units**: Engineering units symbol or text. - **Status Icon**: Indicates value change or error condition. - Column widths can be adjusted. - Parameters can be sorted. - If a parameter is not included in the Live Logging Configuration, its value is replaced by a dash. ## Big Number Display A Big Number Display is a specialised Numeric Display created when a single Measurement Parameter is selected via Parameter Explorer. ![Big Number](assets/big-number.png) ### Differences from Numeric Display - Uses the default large font. - Name column is hidden. - Parameter Name/Description shown in the title bar. - Column headings are hidden. --- # Bar The Bar Graph Display visualises one or more Measurement Parameters as bars. Each bar's height and colour reflect the parameter's value and status. ![Bar](assets/bar.png) ## Features - **Bar Height**: Proportional to parameter value. - **Bar Colour**: - **Normal**: Between warning limits (default grey). - **Low**: Below low warning limit (default blue). - **High**: Above high warning limit (default red). - **Orientation**: Bars are vertical by default; can be changed to horizontal via System Menu. - **Max/Min Values**: Shown beside each bar; editable via Display Limits dialog. - **Warning Limits**: Displayed and colour-coded. - **Name/Description**: Shown below each bar. - **Value**: Engineering value shown below name/description; matches bar colour. If a parameter is not in the Live Logging Configuration, “not logged” is displayed. ## Selecting Parameters - **Single**: Click bar or use cursor keys. - **Multiple**: - Mouse: CTRL + click. - Keyboard: CTRL + cursor keys + SPACE. - **All**: Use "Select All" from the Bar Graph Context Menu. ## Display Limits To change the display range for a parameter: 1. Open the Bar Graph Context Menu. 2. Select **Display Limits**. 3. Edit Low and High values. --- # Error The Error Display shows diagnostic error flags from the ECU. Each error is represented by a special Measurement Parameter. ![Error](assets/error.png) ## Features - Errors are defined by the Error File included in the Program Version. - Errors are uploaded live from the ECU (must be Online). - Errors are listed in order of occurrence. - Duplicate errors are not shown multiple times. - Errors can be: - **Current**: Actively occurring. - **Logged**: Stored in the ECU from past occurrences. - Logged errors persist until deleted. ## Creating an Error Window Select: `Desktop > Window > Error` Each error appears as a row with: - **Status Icon**: - Current error - Logged error - **Name**: From Error File. - **Description**: Cause or details. - **Status**: Current or Logged. ## Managing Errors - **Acknowledge**: `ECU > Acknowledge Errors` - Changes colour from Unacknowledged to Foreground. - **Delete**: `ECU > Delete Errors` - Removes logged errors from ECU and display. - **Masking**: - Use the Mask tab in Window Properties. - Mask entire groups or individual errors. ![Mask](assets/error-mask.png) ## Display Options - Errors appear on every page by default. - Window auto-restores if minimised during error occurrence. - Column widths are adjustable. - Appearance settings available in Display tab and System Menu. --- # Event Events report status changes in the control system and include a timestamp. They are defined by the Event File in the Program Version. ![Event](assets/event.png) ## Features - Events are displayed in order of receipt (most recent at bottom). - Maximum of 2000 events per window; older entries are deleted. - Events are automatically logged to: \work\Ev_.log - Events can be saved manually via `Save As`. ## Creating an Event Window Select: `Desktop > Window > Event` Each event appears as a row with: - **Priority Icon**: High, Medium, or Low. - **Date**: When the event occurred. - **Time**: Timestamp. - **Event Name**: From Event File. - **Description**: Diagnostic info (HEX or converted text). ## Display Options - Events appear on every page by default. - Column widths are adjustable. - Appearance settings available via Display tab and System Menu. ## System Menu – Events - **Clear All Event Windows**: Removes all listed events. - **Save As**: Save events to a file. ## Event Masking - Use the Mask tab in Window Properties. - Mask by: - **Group**: Entire application group. - **Individual Events** - **Priority Level**: Low, Medium, High, All ![Event Mask](assets/event-mask.png) --- # Grid The Grid Display is a flexible, spreadsheet-style layout that allows text and attributes of Measurement or Scalar Parameters to be placed in cells. It is commonly used to create customised views for specific operations. ![Grid](assets/grid.png) ## Features - Editable Parameters can be highlighted using a different background colour. - Values not included in the Live Logging Configuration are shown as a dash. - Appearance controls are available via the Window Properties dialog and System Menu. - Grid Displays can replicate the look and function of custom pages from System Monitor Version 6. ## Edit Mode Edit Mode allows modification of cell content and layout. - Enabled by default for new Grid Windows. - When enabled: - Column headers are shown. - Column widths can be changed. - Grid Cell Context Menu is available. - Text and cell properties can be edited. To disable Edit Mode, clear the checkbox in the Numeric Grid tab of the Window Properties dialog. ## Creating a Grid Window 1. Select: `Desktop > Window > Grid` 2. Configure: - **Rows and Columns**: Set initial size. - **General Panel**: - Edit Mode: Toggle cell editing. - Gridlines: Show/hide gridlines. ![Create Grid](assets/create-grid.png) Click **OK** to create the window. Cells must be configured individually. ## Selecting Cells - **Single**: Click or use cursor keys. - **Multiple**: - Mouse: SHIFT or CTRL + click. - Keyboard: SHIFT or CTRL + cursor keys + SPACE. !!! note Non-continuous blocks cannot be copied. ## Cell Properties To edit a cell: 1. Ensure Edit Mode is enabled. 2. Click the cell. 3. Press ESC to highlight. 4. Right-click > **Properties** ### Contents Tab - **Empty**: Clears cell. - **Text**: Enter free text. - **Parameter**: - Select a parameter. - Choose attribute to display (e.g. Engineering value). - Optional bit masking for logical display. ![Cell Contents](assets/cell-content.png) ### Formatting Tab - **Justification**: Set text alignment. - **Borders**: Add thick borders. - **Colours**: - Use default or custom background colour. ![Cell Formatting](assets/cell-formatting.png) ## Insert/Delete Rows or Columns - Use Grid Cell Context Menu: - **Insert**: Specify number and placement. - **Delete**: Remove selected rows/columns or cell contents. --- # Oscilloscope The Oscilloscope Display plots one or more Measurement Parameters against time. It consists of a Plot Area and a Parameter Info Box. ![Oscilloscope](assets/oscilloscope.png) ## Features - **Trace Types**: - **Linear**: Straight lines between values. - **Digital**: Horizontal and vertical steps. - **Logical**: TRUE (1.00) or FALSE (0.00); shown below the timebase. - **Y-Axis**: - Labelled Y-axis for selected parameters. - Option to show Y-axis for all parameters. - No Y-axis for Logical traces. - **Timebase Duration**: Configurable in Extras tab. - **Update Rate**: Set in Display tab. - **Grid Display**: Optional background grid. - **Band Clearing**: - **On**: Previous traces overwritten. - **Off**: Plot area cleared after each sweep. ![Trace Type](assets/types-of-trace.png) ## Parameter Info Box - Shows Name and Value of each parameter. - Can be hidden via Extras tab. - Text colour matches trace colour. - Width adjustable; height fixed relative to Plot Area. ## Selecting Parameters 1. Focus on the display. 2. Go to Desktop > Window > Properties 3. Add parameters in the Parameters tab: - **Single**: Click name or use cursor keys. - **Multiple**: - Mouse: SHIFT or CTRL + click. - Keyboard: SHIFT or CTRL + cursor keys + SPACE. - **All**: Use "Select All" from Context Menu. ## Trace Limits To adjust Y-axis limits: 1. Right click in the Plot Area. 2. Select **Trace Limits**. 3. Edit Low and High values. !!! note Trace Limits do not affect Logical traces. --- # Signal Sampling A signal sampling display can be added to a page by selecting **Desktop > Window > Signal Sampling**. ![Signal Display](assets/signal.png) The signal sampling tool allows one or more input signal parameters to be sampled either independently or simultaneously. Parameters can be added to the display by dragging them into the display from the parameter explorer. They can be deleted either by selecting the parameter in the display and pressing the ‘delete’ key or via the parameter explorer. Multiple signal sampling displays can be displayed on one page. Each input signal parameter is displayed as a row with the following columns: - **Name**: The name of the input signal parameter. This is set in the input signal configuration dialog. - **Raw**: The raw (un-scaled) value of the input signal. - **Units**: The units of the raw value of the input signal. - **Reference**: The scaled value the signal should read when sampled. This value is user editable. - **Scaled**: The current scaled value of the input signal including the offset. This is the value that will be logged. - **Units**: The units of the scaled value of the input signal. Input signal parameters can be sampled from the current value using one of the following options, all of which are available via both menus and keyboard shortcuts: - **Selected Parameter Only**: This will simultaneously sample the selected parameter(s) only. - **Sample Entire Display**: This will simultaneously sample all parameters within the currently selected input signal display. - **Input Signal Sample Entire Page**: This is selected via the Logging menu. This will simultaneously sample all parameters on all signal sampling displays on the current page. !!! note There must be an active link to the unit for the sampling functionality to be selectable. If not, then the sampling menu options will be disabled. If one or more parameters have been sampled, then the user will need to download the logging configuration to the unit so that the changes are applied. The ECU bar will indicate if the users’ current SM logging configuration file differs from the one on the unit. --- # Alarms The Alarm Display window shows active alarms triggered by defined conditions. It appears automatically when an alarm message is triggered, or can be opened manually via: Desktop > Window > Alarms ## Display Columns Each alarm is shown in a row with: - **Description**: The message text defined in the Actions tab of the Alarm Properties. - **Time**: The time the trigger condition became TRUE. ## Features - Rows are cleared when the trigger condition becomes FALSE. - The window remains open even if all rows are cleared. - Default location is at the top of the desktop, but it can be moved and resized. - Column widths are adjustable. - Alarm windows appear on every page by default. ## Alarm Setup Alarms consist of a **Trigger** and a set of **Actions**. Actions are executed immediately when the trigger condition becomes TRUE. ### Creating Alarms To open the Alarm Setup dialog: Desktop > Alarm Triggers Each alarm is listed with its message text. Alarms are active when their checkbox is selected. !!! note Alarms operate immediately upon creation. The setup dialog does not need to be closed for alarms to function. ## Alarm Setup Controls - **Add**: Creates a new alarm and opens the Alarm Properties dialog at the Trigger tab. - **Properties**: Opens the Alarm Properties dialog for the selected alarm. - **Remove**: Deletes the selected alarm. ![Setup](assets/alarm-setup.png) ## Alarm Properties The Alarm Properties dialog defines the behaviour of an alarm. It includes two tabs: - **Trigger**: Defines the condition that activates the alarm. - **Action**: Specifies what happens when the condition is met. ### Trigger Tab The trigger is a logical formula comparing one or more Measurement Parameters with each other or with a constant. - Use mathematical operators to build the condition. - Example: `RPM > 8000 AND Throttle > 90%` ![Trigger](assets/trigger.png) ### Action Tab Defines the response when the trigger condition becomes TRUE. Multiple actions can be combined. ![Action](assets/alarm-properties.png) **Available Actions** - **Display a Message** (default: on) - **Jump to a Page** (default: off) - **Emit a Sound** (default: off) **Message Panel** - **Display warning message**: Shows message while condition is TRUE. - **Use alarm condition for message text**: Uses trigger formula as message. - **Message Text**: Custom message if above option is disabled. **Goto Page Panel** - **Goto page**: Jumps to selected page when condition is TRUE. - **Page name**: Select from dropdown list. **Sound Panel** - **Sound alarm**: Plays audible alert. - **Test**: Play/stop alarm sound. - **Default**: Use system default tone. - **Custom**: Use selected `.wav` or `.mp3` file. - **Repeat**: Sound repeats until condition clears. - **Duration**: Sound plays for set time (in seconds). - **Reset**: Delay before alarm can restart while still active. --- # Overview System Monitor is a configuration and monitoring tool used in TAG systems to manage how Electronic Control Units (ECUs) and other data logging devices acquire and store data. It supports both live and offline data acquisition workflows and integrates with ATLAS for analysis. ## Key Functional Areas System Monitor provides tools grouped into two main categories: - **Logging**: Configure how data is acquired and stored by the ECU, including live streaming, memory-based logging, and telemetry. - **Sensors**: Set up and calibrate sensors connected to the ECU, including both software-calibrated and externally configured sensors. ## Logging Methods System Monitor supports three primary logging methods: - **Live Logging**: Real-time data acquisition via wirelink. - **Remote Logging**: Data stored in DATALab memory for later upload. - **Burst Logging**: High-speed data stored in Burst memory, often used with Wide Band Telemetry. Each method has its own configuration tools and memory management strategies. ## Telemetry Support TAG systems support two types of telemetry: - **Narrow Band (Continuous)**: Low data rate, not configured via System Monitor. - **Wide Band (Burst)**: High data rate, configured via Burst Logging tools. --- # Overview System Monitor provides comprehensive tools to configure how data is logged by an ECU or other data logging device. Logging is the process of acquiring data from measurement parameters and storing it for analysis, either live or post-session. ## Logging Categories System Monitor supports three main logging methods: - [**Live Logging**](live-logging.md): Sends data directly to System Monitor via wirelink, without storing it in the ECU. - [**Remote Logging**](remote-logging.md): Stores data in DATALab memory for later upload. - [**Burst Logging**](burst-logging.md): Stores high-frequency data in Burst memory, also used for Wide Band Telemetry. Each method has its own configuration tools and session management features. ## Memory Allocation TAG systems use two types of memory for data logging: ### Burst Memory - Designed for high-speed logging of engine parameters. - Triggering is simple and data is collected at the highest rate supported by the hardware. - Configured via the Burst Logging configuration. ### DATALab Memory - Used for chassis parameters that require lower sampling rates. - Offers versatile triggering and configurable sampling rates. - Configured via the Remote Logging configuration. ## Session Generation Sessions represent the data collected during a test or race. They can be generated in three ways: ### Live If the ECU is On-Line (in a testbed situation for example), a Session file can be generated which is suitable for analysis by ATLAS. The characteristics of the Session file are set using the Live Logging configuration. !!! note The Session data generated by System Monitor Version 7 is a single file in the format used by ATLAS Version 7. It is not compatible with ATLAS Version 6, which uses a different format. ### Upload If Data Logging is used, System Monitor can configure the ECU to choose the parameters to be included in the session as well as the trigger and sample rate characteristics. System Monitor does not have any direct involvement in generating the Session files. They are created by uploading the data from the ECU into ATLAS. ### Telemetry Data is transmitted from a moving vehicle during the test session. System Monitor does not have any direct involvement in generating the Session files. ATLAS receives the data and analyses it live. ATLAS generates a Session file when Telemetry stops. TAGtronic systems support two types of telemetry: - **Narrow Band (Continuous)**: Low data rate telemetry that sends data continuously using low frequency radio. System Monitor is not used to configure this type of telemetry. - **Wide Band (Burst)**: High data rate telemetry that sends data in bursts using high frequency (Microwave) radio. Data is logged into the Burst Memory, the contents of which are transmitted in each burst. System Monitor is used to configure this type of telemetry using the Burst Logging configuration. --- # Live Logging Live Logging is the process by which System Monitor obtains data from a connected ECU. The Live Logging Configuration serves two purposes in System Monitor: - Obtains data for display in the current Desktop so it must include all the parameters that have been selected for display. If there is no Live Logging Configuration File in the Project, all the parameters are available for display. - Selects the parameters to be included in a Session. The live logging functionality in System Monitor captures slow row data. Therefore, to view it in ATLAS, ATLAS must be configured to display slow row parameters. To configure Live Logging select `Logging > Live Configuration`. ### Session Tab Use the Session tab of the Configure Live Logging dialog box to edit the characteristics of the Session that is created by System Monitor. - **Files Generated Panel**: - **Next Session Filename**: Displays the name and path of the file to be created for the Session. The default filename format is `SESSnnn.SSN` (where `nnn` starts at `000` and increments with each recorded session). Files are stored in the `\Sessions\` subdirectory under the Customer Base directory, which is automatically created when you create a new project. - **Multiple Shot Sessions**: When enabled, records the number of sessions specified in the **Number of Shots** field. Each session is recorded automatically when trigger conditions are met, with unique filenames generated by incrementing the last three digits. When disabled, only one session is recorded when trigger conditions are met. This option has no effect if triggering is set to Manual on the Trigger Timing tab. ![Session](./assets/session-tab.png) ### Trigger Timing Tab Use the Trigger Timing tab of the Configure Live Logging dialog box to set the size of the Session and the point in time that the trigger occurs in the data. #### Logging Time - The **Logging Time** is the length of time taken to log the session and is shown as Hours:Minutes:Seconds. - Default Logging Time: 10:00 (ten minutes). #### Mode panel Select where the trigger appears in the recorded session data: - **Manual** — The trigger is not used. - **Start Logging** — The session starts when the Trigger Condition is TRUE and ends when the Logging Time has elapsed. - **Stop Logging** — The session ends when the Trigger Condition is TRUE; the session length is defined by the Logging Time. - **Pre and Post** — Partition the Logging Time into a Pre-trigger period and a Post-trigger period: - Use the **Pre trigger time** and **Post trigger time** text boxes to set the durations, or adjust the slider between them to automatically partition the Logging Time. - If editing the text boxes directly, ensure Pre + Post = Logging Time. - The slider provides a quick visual partition; the text boxes allow precise control. #### Disk Space in bytes panel The panel displays read-only information about storage for the session: - **Required** — Amount of disk space required to store the session, calculated from Logging Time, enabled parameters and their sampling intervals. - **Free** — Amount of free space available at the session storage location. !!! note To capture data before the trigger condition is met, System Monitor continuously acquires data into a buffer sized to hold all data for the entire Logging Time so that pre-trigger data can be written when the trigger occurs. ![Trigger](./assets/trigger-tab.png) ### Trigger Condition Tab Use the Trigger Condition tab of the Configure Live Logging dialog box to define the condition that acts as a trigger for the recording of the Session. A session is recorded according to the settings in the Trigger Timing tab. The trigger condition is a logical formula created from one or more Measurement Parameters combined by Mathematical Operators. ![Condition](./assets/condition-tab.png) Live Logging can record a Session in two ways: - Manual — Manual is selected as the Mode in the Trigger Timing tab. - Automatic — A Trigger is selected as the Mode in the Trigger Timing tab. The Status Bar indicates the status of recording as follows: - **Recording OFF** — The Session is not being recorded. - **Recording ON** — The Session is being recorded. - **Trigger Pending** — Waiting for the trigger condition to become TRUE. Make an Automatic Recording: 1. Select a Trigger as the Mode in the `Trigger Timing` tab. 2. Close the Live Logging Configuration dialog box. 3. Select `Logging > Trigger Enable`. 4. Select `ECU > On Line` to put the ECU On-Line. 5. Recording starts and stops automatically according to the settings in the `Trigger Timing` tab. Recording can also be stopped by selecting `Logging > Recording` or by taking the ECU Off-Line. --- # Remote Logging Remote Logging is the process by which an ECU, stores data in DATALab Memory while not connected to System Monitor. System Monitor can be used to edit the logging configuration in the ECU. To do this it is necessary to download the logging configuration from System Monitor. Similarly, to modify an existing logging configuration in the ECU, that configuration must first be uploaded to System Monitor. Remote Logging Configuration is done in the Configure Remote Logging dialog box. To open the dialog box, select: `Logging > Remote Configuration`. ### Configure Remote Logging Dialog Use the Configure Remote Logging dialog box to define the characteristics of the channels used to store data in the ECU. Click the information icons for context-sensitive help on the various elements of the dialog. ![Configure Remote Logging Dialog](./assets/remote-logging.png) #### Memory end strategy panel The Memory end strategy applies to all channels. Two strategies are available: - **Stop when full** — Data is logged from the start trigger until either the stop trigger occurs or the ECU memory is full. - **Wrap** — Data is logged from the start trigger until the stop trigger occurs. If the available memory is exceeded, new data overwrites the oldest data in memory so the memory always contains the most recent logged data. Use this strategy to analyse data leading up to a specific event. #### Driver push button panel Driver push buttons (dash or steering-wheel mounted) can be configured as either a trigger condition or to clear the ECU data logging memory. Options: - **Button on clears logged data** — Pressing the driver push button clears the ECU data logging memory. If this option is used on the Channel Trigger dialog for any channel, that portion of the trigger condition will not operate. - **Button on used as a trigger condition** — The driver push button is available as a trigger input for Channel Trigger conditions. - **Button disabled** — The driver push button has no effect. If this option is used on the Channel Trigger dialog for any channel, that portion of the trigger condition will not operate. #### Logging duration and rate - The remote logging session duration is displayed on the Logging Duration panel as Hours:Minutes:Seconds and as a number of laps. The calculation is based on the sample rates of all enabled parameters across all channels and the settings in the Logging Duration dialog. Click `Setup` to open the Logging Duration dialog. - The Logging and Telemetry displays indicate the required update rate for all channels. These values are calculated from the number of enabled parameters across all channels and their individual update frequencies. #### Channel properties and configuration - Remote Logging Configuration targets the DATALab logging system. DATALab provides eight channels that can be configured independently. - Add parameters using `Parameters` to open the Select Parameters dialog. Maximum parameters allowed: 512. The number currently loaded is shown by the Logging slots used value. - Each parameter is available on all eight channels. When a parameter is added it is disabled by default; to log it you must set a sampling frequency. - The same parameter can be logged at different frequencies on different channels so it can be sampled at different rates under different conditions (each channel may have a different trigger). In ATLAS, parameter data from different channels can be combined to achieve better resolution. | Column | Description | |--------|-------------| | **Parameter** | The parameter name. | | **Description** | The parameter description. | | **Application** | The application associated with the parameter. | | **Bytes** | Size of the parameter in bytes. | | **Prime** | Prime parameters are defined in the ASAP file. Prime parameters are logged continually in Channel 1 and cannot be disabled in that channel. An asterisk in the Prime column indicates a Prime parameter. Note: The Parameter, Description, Application and Prime columns can be toggled on and off using context menu commands. | | **Channels** | Sampling frequency for the parameter on that channel. `---` indicates logging is disabled for the parameter in that channel. Note: If a cell background is red, the selected frequency is higher than the maximum refresh rate of the parameter. | #### Context menu commands With one or more cells selected, these context menu commands are available: - **Frequency** — Sub-menu to select a logging frequency for the selected parameter cells. - **Cycle** — Log the selected parameter once per engine cycle. Cells affected display `1 Cycle`. - **Engine Segment** — Opens the Synchro Pattern dialog to select which engine segments to log the selected parameter. Cells affected display ``. #### Controls Control buttons apply to the currently selected cell(s). With multiple cells selected you can apply a command to: - **Row** — e.g., enable a parameter in all 8 channels. - **Column** — e.g., set the same frequency for all parameters in a channel. - **Block** — e.g., enable parameters in a subset of channels. Selection: - Select multiple cells by click-and-drag or with keyboard arrow keys. - To select all channels for a parameter, click the parameter name in the Parameter column. | Button | Behavior | |-----------------|--------------------------------------------------------------------------------------------------------------------| | **Enable** | Enables the selected parameters in the selected channels. If a frequency was previously set it is reinstated; if the parameter was previously disabled the frequency is set to 100 Hz. | | **Disable** | Disables the selected parameters in the selected channels. The DELETE key has the same effect. | | **Set freq** | Opens the Set Uniform Frequency dialog. Set the sampling frequency for the selected parameters in the selected channels. | | **Freq +/-** | Increases/decreases the frequency of selected parameters in the selected channels to the next valid frequency. If the parameter is disabled in that channel the previous value is recalled and increased/decreased. If no prior value was set, the default of 100 Hz is assumed and adjusted. Keyboard shortcuts: `+` and `-`. | | **Set Trigger** | Opens the Channel Trigger dialog where start and stop triggers for the selected channel are defined. | | **Channel** | Opens the Channel dialog for the selected channel. Use this to edit the channel name and select whether the channel is used for Telemetry, logging to the unit, or both. | | **Arrange** | Opens the Arrange Channels dialog. Use this to select which channels are logged and change the channel order in the grid. | | **Row example** | Enable a parameter in all 8 channels. | | **Column example** | Set the same frequency for all parameters in one channel. | | **Block example** | Enable parameters in a contiguous subset of channels. | ### Remote Logging - Duration System Monitor calculates the duration of the Remote Logging session and displays it on the Channel Overview tab in hours:minutes:seconds and number of laps. The calculation is based on the sample rates of all enabled parameters in all channels and information set in the Logging Duration dialog. ![Logging Duration Dialog](./assets/remote-duration.png) To open the Logging Duration dialog, click `Setup` in the Configure Remote Logging dialog. !!! note These settings are only used for calculating the Logging Duration display; they do not affect the logging configuration. #### Logging Duration Settings - **Logging Memory** — Size of the memory available for logging. - **Out Lap time** — Estimated duration of the out lap in seconds. One out lap is included in the Duration display. The right-hand value shows the volume of memory required to log data for that time period. - **In lap time** — Estimated duration of the in lap in seconds. One in lap is included in the Duration display. The right-hand value shows the volume of memory required to log data for that time period. - **Typical lap time** — Estimated typical duration of a flying lap in seconds. This value is used to calculate the number of laps shown in the Duration display. - **Row data freq** — Average sample rate of Row data. The right-hand value shows the volume of memory required to log Row data for the entire session. This value is subtracted from the memory available for configured logging. - **Average RPM** — Estimated average RPM. The Duration calculation can allow for channels in which sampling is not continuous. ### Remote Logging - Channel Trigger Use the Channel Trigger dialog to define the start and stop triggers for a channel. To open the dialog, click `Set Trigger` on the Channel Overview tab of the Configure Remote Logging dialog. ![Channel Trigger Dialog](./assets/remote-trigger.png) #### Start and Stop Trigger Panels The Start and Stop triggers both have the same format, consisting of three conditions with the following relationship: ``` Condition 1 OR (Condition 2 AND Condition 3) ``` #### Condition Options Each condition is defined by a row. The left-hand drop-down list has the following condition options: - **On data** — When selected, the Select Trigger Parameter dialog opens to enable parameter selection. Only parameters included in the current Remote Logging Configuration are available for selection. Selecting On data enables the other fields in the row. - **Driver push button** — The condition is TRUE when the driver presses the button. For this condition to work, the Driver push button option on the Remote Logging - Channel Overview tab must be set to **Button on used as a trigger condition**. - **Ignition on** — The condition is TRUE when the engine ignition is switched on. - **Lap trigger** — The condition is TRUE when the ECU receives a signal from the Lap Trigger. - **No condition** — Fixed at FALSE for Condition 1 and TRUE for Condition 2 and Condition 3. This ensures that selecting No Condition does not override the other conditions. #### On Data Settings When **On Data** is selected as a condition, all fields are enabled: - **Name** — The name of the selected parameter. - **Comparison Operator** — Select the operator from the drop-down list: - `=` — The condition is TRUE when the parameter value exactly equals the test value. - `<` — The condition is TRUE when the parameter value is less than the test value. - `>` — The condition is TRUE when the parameter value is greater than the test value. - **Test Value** — Enter the test value. Parameter units are shown next to this field where appropriate. - **Condition repeat** — This field is enabled for all condition types. Enter the number of times the condition must occur before it is considered TRUE. The default is 1. #### Start/Stop Post Trigger The effect of the trigger is delayed by the time (in milliseconds) set in the Start/Stop Post Trigger text boxes. The default is 0. Entering a negative value enables a pre-trigger delay. #### Settings Panel - **Trigger re-arms after stop trigger** — When enabled, a set of data will be logged every time the trigger conditions are met. When disabled, only one set of data will be logged. - **Multiple triggers** — Calculates the Duration shown on the Channel Overview tab. This setting does not affect the logging configuration. When enabled, the **Triggers/lap** and **Trigger duration** settings are used to estimate the time required to log the data. When disabled, the duration is calculated assuming continuous parameter logging. - **Triggers/lap** — Estimate of the expected number of times the trigger will be activated in each lap. - **Trigger duration** — Estimate of the average length of time that data will be logged each time the trigger is activated. ### Arrange Channels The Arrange Channels dialog controls which channels are used for logging and the order in which they are displayed in the Configure Remote Logging dialog. ![Arrange Channels Dialog](./assets/arrange-channels.png) To disable a logging channel, select the check box adjacent to the channel name and click `OK`. The channel column is removed from the Configure Remote Logging dialog. To change the order in which channels are displayed, select a channel and click `Up` or `Down` to move it up or down the channel list. Click `OK`. The channel columns are rearranged in the Configure Remote Logging dialog accordingly. ### Synchro Pattern The Synchro Pattern dialog determines which engine segments a parameter is logged in. ![Synchro Pattern Dialog](./assets/synchro-pattern.png) To open the Synchro Pattern dialog, select `Engine Segment` from the Configure Remote Logging dialog. The selections apply to the parameters selected in the Configure Remote Logging dialog. Select which segments to log by selecting the appropriate check boxes. --- # Burst Logging Burst Logging is the process by which an ECU stores data in Burst Memory while not connected to System Monitor. In some types of ECU, the content of the Burst Memory can be transmitted at intervals using Wide Band Telemetry. Logging menu options enable Burst Logging Configuration in the ECU. In order to change the logging configuration in the ECU, it is necessary to download it from System Monitor. Similarly, to modify an existing logging configuration in the ECU, that configuration must first be uploaded to System Monitor. When ATLAS reads the data supplied by the ECU, it needs information about the Burst Logging Configuration. This information is supplied by System Monitor in a special export format that can be read by ATLAS. To open the Burst Logging Configuration select: **Logging > Burst Configuration**. !!! note If this menu item is not available, a new Burst Logging Configuration File must be created. Burst Logging Configuration is only available if the relevant Customer Base has permission, as set in **Advanced Settings**. The dialog box consists of the following tabs: - **Parameters** - Select the parameters to be included in the Session that is created when the contents of the Burst Memory are uploaded to ATLAS. - **Triggers** - Set the trigger conditions to start and stop data logging. - **Session** - Edit the characteristics of the Session that is created when the contents of the Burst Memory are uploaded to ATLAS. ## Export When ATLAS reads the data supplied by the ECU, it needs information about the Burst Logging Configuration. This information is supplied by System Monitor in a special export format that can be read by ATLAS. A Burst Configuration Export file is created automatically every time a Burst Logging Configuration is downloaded to an ECU. The file name is generated from the timestamp in the ECU. The file name is used by ATLAS to link the session to the configuration file so it must not be changed. The path is defined in the **Advanced Settings** dialog box - select: **Tools > Advanced Settings**. To create a Burst Configuration Export file at any time select: **Logging > Burst Export**. ## Parameters Use the Parameters tab of the Configure Burst Logging Dialog to edit the characteristics of the Session that is created when the contents of the Burst Memory are uploaded to ATLAS. ![Parameters Tab](./assets/burst-parameters.png) **Display Control** The Display drop-down list determines how the Parameter List is displayed. The options are: - **Parameters** - Shows all the selected parameters with a row for each one. - **A Blocks** - Shows all the available A blocks with a row for each one. - **B Blocks** - Shows all the available B blocks with a row for each one. **Size and Logging Time Display** The following information is displayed: - **Size** - The size of the Logging Memory in the ECU. Taken from the ASAP file. - **Logging Time** - The duration (in hours:minutes:seconds) of logging that will fill the memory. Calculated from the Logging Intervals and the Size of the memory. - When **Parameters** is selected in the Display drop-down list, logging time is calculated from the Parameter intervals. - When **A Blocks** or **B Blocks** is selected in the Display drop-down list, logging time is calculated from the block intervals even if the blocks do not have any parameters assigned to them. **Parameters List** | Column | Description | |---|---| | Parameter | Parameter name. Only shown when "Parameters" is selected in the Display drop-down list. This column can be hidden using the context menu. | | Description | Parameter description. Only shown when "Parameters" is selected in the Display drop-down list. This column can be hidden. | | Application | The Application associated with the Parameter. Only shown when "Parameters" is selected in the Display drop-down list. This column can be hidden. | | Block | The block to which the Parameter has been assigned. '---' indicates the Parameter is not assigned to a block. | | Interval | The sampling interval for the Parameter. When a Parameter is assigned to an empty block, the interval is set to the lowest available value. '---' indicates logging is disabled for the Parameter. | | Bytes | The size of the Parameter in bytes. Only shown when "Parameters" is selected in the Display drop-down list. | | Bytes Available | The space available in bytes for assigning Parameters. Only shown when "A Blocks" or "B Blocks" is selected in the Display drop-down list. | **Adding Parameters** Click **Parameters** to open the **Select Multiple Parameters** dialog box. Use this to add and remove Parameters. The number of Parameters that you can add depends on their type and how they are assigned to blocks. If the list is empty when parameters are added, use **Auto Assign** to place all the Parameters in B blocks. If the list is not empty, the new Parameters are added using the following rules of precedence as the available space is filled up: 1. Same block as currently selected Parameter 2. New block of the same type (A or B) at the same interval 3. Other type of block at the same interval 4. Same type of block at higher interval 5. Other type of block at higher interval 6. Not assigned If **A Blocks** or **B Blocks** is selected in the Display drop-down list, and one or more blocks are selected, the Parameters are assigned to the selected blocks. **Set Block** Opens the **Select Burst Block** dialog box. Only available if **Parameters** is selected in the Display drop-down list. The drop-down list shows all the blocks which have sufficient space to accept the selected Parameter(s). Click OK to assign the selected Parameters to the selected block. If “---“ is selected, the Parameter assignments are deleted and the list shows in the Block column. The Parameter is not logged. **Auto Assign** Opens the **Assign Burst Blocks** dialog box. Only available if **Parameters** is selected in the Display drop-down list. The selected Parameters are assigned (in the order that they appear in the list) to the selected type of block, in numerical order of blocks. **Set Interval** Features: - The interval applies to a complete Block. The intervals of Parameters that share a block with the selected Parameters are also changed. - The interval can be set to “---“, which prevents the Parameter from being logged while retaining its block assignment. - Set Interval only operates on Parameters that are assigned to blocks. - If **A Blocks** or **B Blocks** is selected in the Display drop-down list, the interval is applied to the selected block or blocks. The blocks do not have to have any parameters assigned to them. - If **A Blocks** or **B Blocks** is selected in the Display drop-down list, the interval is applied to the selected block or blocks. If multiple blocks are selected, the intervals are increased sequentially in each block in the list. The blocks do not have to have any Parameters assigned to them. **Interval +/-** Increases/decreases the interval of the selected Parameters to the next valid interval. This feature only operates on Parameters that are assigned to blocks and do not have an interval set to “---“. You can also use the plus and minus keyboard equivalent keys. ## Triggers tab Use the Triggers tab to set the trigger conditions to start and stop data logging. ![Triggers Tab](./assets/burst-triggers.png) **Start Panel** - **Driver pushes button** - When selected, logging starts when the driver presses the control button that is configured for use with data logging. The default is selected. When **Driver pushes button** checkbox is cleared, the following options are available. Select one of the options. - **Ignition on** - When selected, logging starts when the ignition is switched on. A delay of up to 120 seconds can be entered, to a resolution of 0.1 seconds. The default is 0. - **Engine started** - When selected, logging starts when the engine is started. A delay of up to 120 seconds can be entered, to a resolution of 0.1 seconds. The default is 0. - **Engine speed** - When selected, logging starts when both Engine Speed and Throttle Position have exceeded the specified thresholds. This is the default. Enter Speed as an integer, between 0 to 20000 rpm, and Throttle position as a percentage to a resolution of 0.1%. The default for both of these is 0. **Stop Panel** - **Speed below** - When selected logging will stop when the engine speed has fallen below the specified threshold. Enter Speed as an integer from 0 to 20000 rpm. The default is 0. - **Memory end strategy** - Two strategies are available: - **Stop when full** - Data is logged from the start trigger until either the stop trigger occurs or the ECU memory is full. This is the default. - **Wrap** - Data is logged from the start trigger until the stop trigger occurs. If the available memory is exceeded, new data overwrites the oldest data in the memory, so the memory always contains the most recent logged data. Use this strategy if you want to analyze data leading up to a specific event. ## Session tab Use the Session tab to edit the characteristics of the Session that is created when the contents of the Burst Memory are uploaded or transmitted to ATLAS. For ECUs that utilize Burst memory for Wide Band Telemetry, the following controls are available: - **Car ID** - The car is given an ID number to distinguish it from other cars (For example, cars in the same team using the same telemetry system). The number entered here must match the ID of the car. The value is an integer from 1 to 4. - **Transmitter** - When selected, the transmission of data via telemetry is enabled. - **Frequency** - The value in this text box is used to set the radio frequency of the transmitter if the transmitter is software configurable. Ensure that the value is within the operating range of the transmitter. - **Multi-Burst File** - Some versions of ATLAS can configure the retransmission of data in multi-burst telemetry. If your version of ATLAS has this facility, it will generate a Multi-Burst Coverage File (*.mbc). The coverage data is downloaded to the ECU with the Logging Configuration. If you do not have a Multi-Burst Coverage File this field is empty. ## Blocks The Burst Logging blocks are set up in the Parameters tab of the Burst Logging dialog box. The hardware design imposes restrictions on the way the Logging Memory is used and on the speed of logging. System Monitor is configured to automatically apply rules regarding such restrictions. Blocks are automatically filled and any attempts to select more than the maximum number of parameters are prohibited. The following describes the way calculations are performed. There are two types of Parameter: - **WORD** - Single WORD parameter: The parameter value is coded as one WORD. - **DWORD** - Double WORD parameter: The parameter value is coded as two WORDs. A DWORD parameter uses exactly twice as much memory as a WORD parameter, so that a given amount of memory can only accept half as many DWORD parameters as WORD parameters. Memory is arranged in Blocks. There are two sizes of Block: - **A Blocks** - Each A Block can contain 4 WORD size parameters, 2 DWORD parameters or a mixture of both. - **B Blocks** - Each B Block can contain 8 WORD size parameters, 4 DWORD parameters or a mixture of both. The number of Blocks is restricted as follows: | Max No. of Blocks | Max No. of WORD Parameters | |--------------------|----------------------------| | A Blocks | 64 | 256 | | B Blocks | 64 | 512 | Each DWORD Parameter takes up two WORD parameter places. The memory is ordered on data type. It is filled first with DWORD parameters then WORD parameters. For example, if 7 DWORD parameters and 5 WORD parameters are assigned to B blocks, the blocks would be filled as follows: | DWORDS | WORDS | Empty | Total WORD Equivalent | |--------|-------|-------|-----------------------| | 1st Block | 4 | - | - | 8 | | 2nd B Block | 3 | 2 | - | 8 | | 3rd B Block | - | 3 | 5 | 8 | | Total | 7 | 5 | | | ## Intervals The Logging Interval for each Burst Logging block is set up in the Parameters tab of the Burst Logging dialog box. The hardware design imposes restrictions on the way the Logging Memory is used and on the speed of logging. System Monitor is configured to automatically apply rules regarding such restrictions and automatically calculates the minimum interval. There is a minimum logging interval (fastest logging rate) for each Block. Parameters can only be logged in complete blocks even if a block is not full. Logging is ECU time synchronous. Each block takes 1ms to log therefore: **Minimum interval (ms) = number of blocks with parameters assigned** A blocks and B blocks are independent and have different minimum intervals. If higher rate sampling is required (reduced interval) for selected parameters, the minimum interval can be increased by doing one or more of the following: - Increase the interval of some other parameters - Move parameters from A blocks to B blocks or from B blocks to A blocks - Set the interval of some other parameters to “---“ - Set the block assignment of some other parameters to “---“ - Remove some other parameters --- # Remote Configuration Compare Remote Configuration Compare is the process that allows the comparison of the existing project remote logging configuration to an additional existing configuration file. The configuration details can be compared and merged as required. ## Remote Configuration Compare Dialog Remote Logging Compare is done in the Logging Configuration Compare dialog box. To open the dialog box, select: **Logging > Remote Configuration Compare**. On opening, the existing project logging configuration file is displayed. To compare with an additional configuration, click the **Load** button and select the required configuration. Both files will then be displayed for comparison and merging. ![Remote Configuration Compare Dialog](./assets/remote-config-compare-dialog.png) ### Options - **Load**: Loads a configuration to enable comparison and merging. - **Unload**: Unloads the compare configuration. - **Logging Slots**: Displays the logging slots used in both the current configuration and, if loaded, the compare configuration. - **Show parameters with differences only**: When selected, only differences between the configurations are displayed. - **Commit**: When selected, any outstanding merging will be committed into the existing project remote logging configuration. - **Reset**: When selected, any outstanding merging will be undone. Once committed, a merge cannot be undone. ## Config Tab The config tab displays the differences between the parameter logging rates for the two configurations, with a color scheme used to highlight any differing rates. ![Remote Configuration Compare - Config Tab](./assets/remote-config-compare-config.png) The grid consists of the following columns: - **Parameter**: The parameter name. (Can be toggled on and off via the context menu.) - **Description**: The parameter description. (Can be toggled on and off via the context menu.) - **Application**: The application associated with the parameter. (Can be toggled on and off via the context menu.) - **Bytes**: The size of the parameter in bytes. (Can be toggled on and off via the context menu.) - **Prime**: A parameter defined as a Prime Parameter in the ASAP file. Prime Parameters are logged continually in Channel 1 and cannot be disabled in that channel. An asterisk in the Prime column indicates a Prime Parameter. (Can be toggled on and off via the context menu.) - **Channels**: The sampling frequency for the parameters on that channel. The header displays the channel title and type for each of the configurations. ### Logging Rate Indicators - Logging rate in secondary configuration higher than in master. - Logging rate in secondary configuration lower than in master. - Logging rate in secondary configuration same as in master. - Logging rate exists in master only. - Logging rate exists in secondary only. - Logging rate merged from secondary into master. To merge a value from the secondary configuration, select the appropriate cell or cells and right-click to display the merge option. The merge option is only available if there are differences between the values selected. If items within the selection are already merged, an undo option will be available. ### Merge / Undo Selections - **Channel Title Merge**: Right-click in the first two header rows and from the menu select 'Merge Channel title into Master'. This will set the master channel title to the same as the secondary. - **Channel Type Merge**: Right-click in the lower two header rows and from the menu select 'Merge Channel type into Master'. This will set the master channel type to the same as the secondary. - **Row Merge**: Right-click in any header row and from the menu select 'Merge into Master'. Set the logging rates of all channels for the parameter selected to the same as the secondary. - **Channel Merge**: Right-click in any header row and from the menu select 'Merge Channel into Master'. Set the logging rates of all parameters in the channel to the same as the secondary. - **Cell Merge**: Select one or more cells in the grid and from the menu select 'Merge into Master'. Set the logging rates of all cells selected to the same as the secondary. ## Triggers Tab The triggers tab displays the differences between the triggers for the two configurations. ![Remote Configuration Compare - Triggers Tab](./assets/remote-config-compare-trigger.png) ### Merging Triggers - **<<**: Selecting the '<<' option will merge the secondary trigger into the master. Can only be selected if a difference between the triggers exists. - **X**: Selecting the 'X' option will undo any merge of the trigger. Can only be selected if the trigger has been merged. - **Next**: Selecting 'Next' will cycle to the next trigger. - **Prev**: Selecting 'Prev' will cycle to the previous trigger. --- # Logging Configuration ## Session Details Session Details identify the Session and include information to assist in the analysis of session data. Session details are required for Live Logging, Remote Logging, and Burst Logging. To open the Session Details dialog box, click **Edit** on the Sessions tab of the Live Logging dialog box. The details are the same on the Remote Logging and Burst Logging dialog boxes. ### Read-only Session Details - **Session** - The name of the Session File. For Live Logging this is set automatically. For Remote and Burst logging the file name is generated by ATLAS. - **Serial Number** - Taken from the ECU hardware. - **Session Start** - Taken from the clock of the system creating the Session file. - **Session End** - Taken from the clock of the system creating the Session file. ### Editable Session Details - Car - Driver - Circuit - Race/Test - Weather - Notes ### Edit Session Details 1. Select an editable item with mouse or keyboard. 2. Click in the text box at the bottom of the tab or press **TAB** and enter any text in the box. 3. To accept the text: press **ENTER** or **TAB** or select another item with the mouse. Session details are shown in ATLAS to help you identify the Session. ## Channels To enhance the versatility of the logging process, parameters are grouped into Channels (sometimes known as Sets). Because channels correspond to physical storage locations in the Data Logging hardware, they are subject to various restrictions. System Monitor recognizes the unit type from its Program Version and only offers the appropriate channel type. Individual units may have further restrictions (such as memory size limitations) which are recognized by System Monitor and used to further restrict available options and calculate sampling rates. ### Channel Types **Raw Data** Used in Live Logging to acquire data from an ECU. Parameters are acquired synchronously. The schedule is based on the PC timer with a minimum interval of 5ms. This time interval determines the rate at which parameter values are logged to disk. Since the ECU sends Parameter values at an unknown and variable rate, some values may be recorded into the session file more often than others. The Parameters logged as Raw data are set by the embedded code independently of logging configurations. Any Parameter logged by a command from the embedded code instead of by a logging configuration is called a Fixed Parameter. Fixed Parameters are typically those whose value does not change often, such as Battery Voltage, but any, and in some cases all, Parameters may be set as Fixed Parameters. Fixed Parameters are generally logged at a much slower rate than the Parameters in a logging configuration and are sometimes called Slow Raw data. **Request Channel** Used in Live Logging to acquire data from the CA-Card via CAN or direct connection. Parameter values are acquired asynchronously. The minimum acquisition interval is calculated based on the time to acquire each Parameter and the number of Parameters. You can set any time interval on or above this minimum. **Non-ECU Request Channel** A restricted form of the Request Channel. Used in Live Logging for Virtual Parameters. **TAGtronic Channel** Used in Burst Logging to acquire data from the Burst memory in an ECU. **DATALab Channel** Used in Remote Logging to acquire data from the DATALab memory in an ECU. Four independently triggered channels are available. A different sampling rate can be assigned to each Parameter. ## Logging Configuration Files When a Logging Configuration is edited, the changes made are stored in a Logging Configuration file. Whenever a New Project is created, a file directory is automatically generated for each logging configuration type - Live, Remote, and Burst. These directories are initially empty. A new Logging Configuration file must be created before a Logging Configuration can be edited, or an existing Logging Configuration file can be opened. To open a new Logging Configuration File, select: **File > New** to open the File New dialog box. A Logging Configuration file can be saved. If you change a logging configuration, you are prompted to save the modified configuration when you close the project. To keep an old configuration, use the **Save As** command to save the modified configuration under a different name before closing the project. Only one instance of each logging configuration type can be loaded into a project. If you have modified a configuration, you are prompted to save it when you create a New or Open an existing file. Remote and Burst logging configurations can also be Uploaded from an ECU. ## Loading Configurations The Remote logging, Burst logging, and External Inputs configurations are stored in and used by an ECU when it is working independently of System Monitor. To change a configuration in the ECU, it must be downloaded from System Monitor. Similarly, to modify an existing configuration in the ECU, that configuration must first be uploaded to System Monitor. This is not automatic and must be done manually using menu commands. Separate commands are provided for Remote logging and Burst logging configurations on the Logging Menu and for External Inputs configurations on the Sensors Menu. Note that some hardware may have both types of logging configuration. ### Download Transfers the configuration from System Monitor to the ECU. Download only takes place if System Monitor detects an ECU that matches the Program Version used by the current project. The old configuration is overwritten. ### Upload Transfers the configuration from the ECU to System Monitor. Uploading overwrites the current configuration in System Monitor. If you have modified a configuration in System Monitor, you are prompted to save it when you select Upload. Any parameter that is not defined by the current system description is discarded from the configuration. A warning will appear listing the parameters that have not been included in the upload. --- # Overview System Monitor includes tools to configure Controller Area Network (CAN) messages for data logging. These messages can be transmitted to external units or received and logged internally using CAN buffers. ## CAN FD Support System Monitor supports **CAN FD (Controller Area Network with Flexible Data-rate)**, enabling higher-bandwidth CAN communication with data payloads up to **64 bytes** per message (compared to the classic CAN limit of 8 bytes). CAN FD is available on hardware units that advertise CAN FD capability in their controller descriptions. For configuration steps, see [CAN FD](configuration.md#can-fd). ## Configuration Dialog Box The CAN Configuration dialog box is used to configure: - Transmit CAN messages for logging by an external unit - Receive CAN messages for logging internally using the CAN buffers To open the dialog box, select: **Logging > CAN configuration**. !!! note - If the CAN Configuration menu option is not enabled, a new CAN Configuration file must be created or an existing CAN Configuration file opened. See **Creating New Files**, **Opening Files**. - When a new CAN Configuration file is added to the project, there are no CAN messages configured initially. Messages can be configured manually or imported from existing files. See **Import/Export Buffers and Messages**. - Creating or adding a CAN Configuration file to the project adds a **CAN Parameters** folder to the System Monitor Explorer. This folder contains CAN Parameters configured in the ECU, which can be associated with CAN Receive Messages. See **CAN Parameters**. ![CAN Configuration Dialog](./assets/can-config.png) The CAN configuration dialog supports the use of up to 8 configurations. These can be loaded and unloaded as required and will be merged to create a single configuration. - A main configuration must exist in the project, but any additional configurations can be loaded, deleted, or set active in the dialog. - Any errors from merging the selected configurations are displayed below the file list. Depending on the project, the FIA can sanction the use of an FIA supplied CAN configuration. In this case, a list of radio buttons allows the FIA configuration to be selected. This results in the defined CAN configuration being included in the logging configuration as a separate entity. ## CAN Configuration Explorer The CAN Configuration explorer shows the CAN Buses configured for the connected unit: - The ECU type is shown at the top with the available CAN Buses listed below it. - The nodes available are defined within the ASAP file. - With a CAN bus node selected, the configurable options available affect all Buffers and their associated CAN messages. --- # CAN Configurations ## CAN Bus With Customer Bus selected in the explorer, the configurable options available affect all Buffers and their associated CAN messages. Expanding the Bus node displays the Buffers and CAN messages configured for the Bus. ![CAN Bus Configuration](./assets/bus.png) **Customer Bus Panel** | Option | Description | |--------|-------------| | **Buffer Count** | The number of Buffers available for configuration on the selected Bus. | | **Global Mask** | The receive mask for all Buffers that do not have a configurable Receive Mask. *Note: Not all control units support a configurable global mask.* | | **Link Speed** | Select a speed in kbits for the selected Bus from the drop-down list. This sets the arbitration-phase bit rate. | | **CAN-FD Supported** | An indicator that appears when the connected controller supports CAN FD. This label is only visible on CAN-FD-capable hardware. | | **Data Speed** | Select the CAN FD data-phase bit rate from the drop-down list. This is separate from the Link Speed (arbitration-phase bit rate). Available speeds are determined by the hardware controller. Only visible when the controller supports CAN FD. | !!! note The **Data Speed** dropdown and **CAN-FD Supported** label are only visible and enabled when the hardware unit reports CAN FD capability. On classic-CAN-only hardware, these controls are hidden. **Parameter Block Panel** Received Measurement Parameters for the CAN bus are stored in a single contiguous block of memory. | Parameter | Description | |-----------|-------------| | **Ctrl Id** | The Control Id of the memory block. | | **Parameter Address** | The start address of the memory block. | | **Block Size** | The size of the memory block in number of blocks. | ## Buffers With a Buffer selected, the configurable options available apply to that Buffer only. ![CAN Buffer Configuration](./assets/buffer.png) !!! note The CAN messages associated with this buffer are listed in the Messages node for this CAN Bus. **Add a CAN Message to an Empty Buffer** 1. Select a Buffer 2. Click **Add** 3. In the Add MessageId dialog box, select a Message Id from the drop-down list and click **OK** The CAN message is assigned to the selected Buffer. When a specific Message is selected in the explorer, the Buffer to which it is assigned is shown in the Message Details. A CAN message can only be assigned to one Buffer. !!! note Buffers can be configured individually as described above, or existing Buffer configurations can be Imported. See [Import/Export Buffers and Messages](#importexport-buffers-and-messages). **Mode Panel** Define the mode of the CAN messages configured for this buffer. | Mode | Description | |------|-------------| | **Not Configured** | Any messages configured for this buffer will not be included in the logging. | | **Receive** | CAN messages are in receive mode. | | **Transmit** | CAN messages are in transmit mode. | **Buffer Panel** | Option | Description | |--------|-------------| | **Receive Mask** | Any bits that are zero in the Receive Mask are ignored when comparing a received dataframe's ID against the acceptance ID of the buffer. For example, if the receive mask is set to `0xFFFFFFF0` and ID of the first message is `0x210`, then the buffer will receive all messages in the range `0x210` to `0x21F`. *Note: Not all buffers have a configurable receive mask.* | | **Rate** | The time between each message being transmitted in ms. | | **Message Count** | The number of CAN messages configured for the selected Buffer. | **Messages Panel** The Messages Window lists the IDs of CAN messages currently configured for this buffer. | Action | Description | |--------|-------------| | **Add** | Click to add a CAN message to the selected buffer. | | **Delete** | Click to delete the selected Message Id. | ![CAN Messages](./assets/messages.png) ## CAN Messages With the Messages node selected, the **New Message** button is displayed to allow new CAN messages to be added to the Configuration. To add a CAN message, click **New Message** and enter a Message ID in the Add Message ID dialog box. The new message is added to the list. !!! note CAN messages can be added and configured individually as described above, or existing CAN Message configurations can be Imported. See [Import/Export Buffers and Messages](#importexport-buffers-and-messages). To view the CAN messages configured for the CAN Bus, expand the Messages Node. With an individual CAN message selected, the configurable options apply to that message only. Use these commands to configure the CAN message and to add new Parameters as required. **Message Panel** | Option | Description | |--------|-------------| | **Message Id** | The unique CAN identifier for the selected CAN message. | | **Description** | Enter a short description of the CAN message. | | **CAN FD** | Enable this checkbox to mark the message as a CAN FD frame. Only available when the parent CAN bus supports CAN FD. | | **Message Length** | Enter the length of the CAN message. For classic CAN messages the maximum is 8 bytes. When CAN FD is enabled, the maximum increases to 64 bytes and the control enforces valid CAN FD data lengths (0–8, 12, 16, 20, 24, 32, 48, 64). For CAN messages with multiplexed data, the length must be the same for all multiplexer values. | | **Buffer** | The Buffer to which the selected CAN message is assigned. (Read only) | | **Parameter Count** | The number of Parameters associated with the selected CAN message. | | **Mode** | TRANSMIT / RECEIVE / NOT CONFIGURED - The Mode of the selected CAN message (the same for all messages in that Buffer). | | **New Parameter** | Opens the Add Parameter dialog box. | When CAN FD is enabled on a message: - The message length range extends to 64 bytes. - Stepping through the length field automatically snaps to the next valid CAN FD data length (e.g., incrementing from 8 jumps to 12, not 9). - Messages are displayed with an **"FD"** label in the tree view (e.g., `Message FD - 0x1A3` instead of `Message - 0x1A3`). **Multiplexed Panel** Multiplexed CAN messages allow more measurement parameters to be received by a CAN buffer than can be packed into a single CAN message. | Option | Description | |--------|-------------| | **Multiplexed** | Select this checkbox to multiplex some or all of the CAN message data. | | **ByteOrder** | Select the byte order of the multiplexer: MSB First or MSB Last. | | **StartBit** | Select the position within the CAN message data where the multiplexer value starts. | | **BitLength** | The length, in bits, of the multiplexer value. Maximum length is 16. | | **MultiplexerValues** | When transmitting this message, multiplexed messages in the range of 0 to Multiplexer Values - 1 will be sent. | ## Parameter Details With a specific Parameter selected, the configurable options available apply to that Parameter only. ![CAN Parameter Configuration](./assets/parameter.png) **Parameter Panel** | Option | Description | |--------|-------------| | **Parameter** | The name of the Parameter. For received Parameters, this is the name displayed by System Monitor and ATLAS. For transmitted Parameters, this is the name of the parameter to be transmitted. | | **Properties** | Open the Parameter Properties dialog box to edit the Parameter Properties. | | **Start Bit** | Select the position within the CAN message data where the Measurement Parameter starts. For classic CAN messages the range is 0–63; for CAN FD messages the range extends to **0–511**. *Note: When MSB Last is selected as the Byte Order, the default Start Bit shows the normalised value. For a parameter at the start of a CAN Message, the Start Bit is shown as 0, although its underlying value is 7.* | | **Bit Length** | Select the length in bits of the Measurement Parameter. Maximum length is 32. | | **Value** | Select whether the Measurement Parameter is Signed, Unsigned, or Floating Point. If Floating Point is selected, the Bit Length must be set at 32. | **Scaling Panel** | Option | Description | |--------|-------------| | **Gain** | Set the gain for converting a scaled value into a raw value. | | **Offset** | Set the offset for converting a scaled value into a raw value. | **Scaling Formula:** ``` Raw Value = GAIN × Scaled Value + OFFSET ``` **Byte Order** Select the byte order of the Measurement Parameter: - **MSB First** - **MSB Last** ### Multiplexer Panel **ID** If the message is multiplexed, the multiplexer ID defines which multiplexed messages the parameter will be in. A Measurement Parameter can be defined in more than one multiplexed message, since the multiplexer ID is a combined value and mask: - `0`, `1`, or `X` can be used for each bit in the multiplexer ID, where `X` means that bit does not matter - For example, if a measurement parameter is in all messages with an even multiplexer value, the multiplexer ID should be set to `XXXXXXXXXX0` **Value** The Parameter is transmitted/received if the value equals the multiplexer value for the message currently being processed, after any bits in the mask are ignored. **Mask** Calculated from the entered multiplexer ID and defines which bits of the multiplexer value should be ignored when determining whether to include this parameter in the message. ## CAN Bit Definitions For each Measurement Parameter and multiplexer value, the start bit, length, and byte order must be defined. Bits and bytes within a CAN message are numbered sequentially from left to right. ### MSB First Format Values When a Measurement Parameter or multiplexer value is defined as **MSB_FIRST**, the start bit defines the location of the most significant bit. The progression of bits and bytes is to the right. ![MSB First Format](./assets/msb-first.png) ### MSB Last Format Values When a Measurement Parameter or multiplexer value is defined as **MSB_LAST**, the start bit defines the location of the least significant bit. The progression of bits is to the left and the progression of bytes is to the right. ![MSB Last Format](./assets/msb-last.png) !!! note - In an MSB Last word, the Start Bit is the last bit of the first byte. Whilst the Start Bit for the word at the start of a CAN message is bit 0, the Start Bit in terms of MSB is bit 7. When a Parameter is added to the CAN Message, the CAN Configure dialog automatically displays normalised values for the Start Bit. For example, if the Parameter appears at the start of a CAN Message, the value 0 is shown, although its underlying value is 7. The Start Bit of a 2nd Parameter is shown as 8, although its underlying value is 15, and so on. - When a CAN message is exported, the saved Start Bit values are the actual values and not the normalised values shown in the dialog box. These can be seen in the Start Bit column of the .csv Export files. ## CAN Message Parameter Layout The **Parameter Layout** dialog displays the byte grid for a CAN message, allowing you to visualise parameter placement within the data frame. - Classic CAN messages display the existing 8-row (byte) grid. - CAN FD messages display a **64-row** grid, covering the full 64-byte payload. The dialog supports vertical scrolling to navigate the larger layout. Use the **Parameter Layout** button on the CAN Message configuration page to open the layout grid and verify parameter placement. ## CAN Parameters When a CAN Configuration file is added to the project, a CAN Parameters folder is added to the System Monitor Explorer. This folder contains CAN Parameters configured in the ECU, which can be associated with CAN Receive messages. !!! note Parameters associated with Transmit CAN messages are not listed in the CAN Parameters folder. However, for a Parameter to be associated with a CAN Transmit message, it must exist in the ECU Measurement Parameters folder. **Add a CAN Parameter** To associate a new Parameter with a CAN message: 1. Select a message and click **New Parameter** to open the Add Parameter dialog box 2. **Name** - Enter the name of a Parameter to associate with the selected CAN message - *Note: For a Parameter to be selected for a Transmit message, it must be listed in the ECU Measurement Parameters folder in the System Monitor Explorer.* - Click to browse for a Parameter in the Select Parameter dialog box 3. Click **OK** to associate the Parameter with the selected CAN message ## Import/Export Buffers and Messages Existing Buffer Configuration (.csv) files and Message Configuration files can be imported into the CAN configuration. Buffer and Message configurations can be exported from the CAN Configuration as .csv files. !!! note When importing Buffer Configuration files or Message Configuration files, all Buffer/Message configurations are overwritten by the imported file. ### Import/Export Buffer Configuration Files 1. Select **Logging > CAN Configuration** to open the CAN Configuration dialog box 2. Expand the CAN Bus and select **Buffers** 3. Right-click and select **Import Buffer** or **Export Buffer** ### Import/Export Message Configuration Files 1. Select **Logging > CAN Configuration** to open the CAN Configuration dialog box 2. Expand the CAN Bus and select **Messages** 3. Right-click and select **Import Messages** or **Export Messages** ### CAN FD Import/Export Notes - **CSV/Text Export** — CAN FD messages are exported with an `fd` suffix on the message ID (e.g., `0x1A3fd` rather than `0x1A3`). Extended-frame identifiers remain compatible (e.g., `0x1A3fdx`). - **CSV/Text Import** — The import parser recognises the `fd` suffix on message IDs and automatically creates CAN FD messages. - **A2L / ASAP2 Import** — The ASAP2 parser now recognises the `TES_SUPPORTS_CANFD` and `TES_DATA_SPEEDS` keywords, allowing CAN FD bus descriptions to be imported from A2L files. - **DBC Export** — DBC export includes the CAN FD flag, ensuring that exported DBC files correctly represent CAN FD message properties. ## CSV File Formats Buffer and CAN message CSV import files must be formatted as described below. **Buffer CSV File Format** | Col | Name | Description | |-----|------|-------------| | 1 | Buffer | The number of the buffers in the CAN Bus. | | 2 | Direction | Mode of the CAN messages configured for this buffer. TX = Transmit, RX = Receive, Blank = not configured. | | 3 | RX Mask | The Receive Mask value, if one is configured for the buffer. Blank = not configured. | | 4 | Rate[ms] | The time (ms) between each CAN message being transmitted. | | 5 | Messages | The CAN message ID configured for the buffer. If more than one CAN message is configured, they must be comma-separated. | **CAN Message CSV File Format** Following a change in the representation of the Parameter Start Bit, v2 of the CAN Message CSV format is now in use. When parameters are set as MSB Last, the value in the .csv is now represented by the logical bit position. v2 files are identified by the text `v2` as the first value in the header row. All exported files now conform to the new format; both formats can be imported. | Col | Name | Description | |-----|------|-------------| | 1 | CAN ID | The CAN message ID. 11-bit IDs: 0 to 0x7FF. 29-bit IDs: 0 to 0x1FFFFFFF. CAN FD messages include an `fd` suffix (e.g., `0x1A3fd`); extended-frame CAN FD identifiers use `fdx` (e.g., `0x1A3fdx`). | | 2 | Message Length | The CAN message length. | | 3 | Start Bit | Multiplexer messages - the position within the CAN message data where the multiplexer value starts. | | 4 | Length | Multiplexer messages - the length, in bits, of the multiplexer value. Maximum length is 16. | | 5 | Byte Order | Multiplexer messages - the byte order of the multiplexer: MSB First or MSB Last. *Note: If the Message is not multiplexed, columns 3, 4, and 5 are blank.* | | 6 | Parameter Name | The name of the Parameter associated with the CAN message. If more than one, subsequent parameters are listed on a new line. | | 7 | Start Bit | The position within the CAN message data where the Measurement Parameter starts. | | 8 | Length | The length in bits of the Measurement Parameter. Maximum length is 32. | | 9 | Multiplexer ID | The multiplexer ID defines which multiplexed messages the parameter will be in. | | 10 | Signed | Whether the Measurement Parameter is signed: YES or NO. If Floating Point, this column must be NO. | | 11 | Floating Point | Whether the Measurement Parameter is Floating Point. | | 12 | Byte Order | The byte order of the Measurement Parameter: MSB First or MSB Last. | | 13 | Scaling Gain | Parameter property. The Gain set for converting a scaled value into a raw value. | | 14 | Scaling Offset | Parameter property. The Offset set for converting a scaled value into a raw value. | | 15 | Description | Parameter property - Description. | | 16 | Display Format | Display format. | | 17 | Units | Display units. | | 18 | Lower Limit | Lower value limit. | | 19 | Upper Limit | Upper value limit. | | 20 | Identifier | CAN Parameter unique identifier. | | 21 | Display Limit Low | Lower display limit. | | 22 | Display Limit High | Higher display limit. | ## CAN Configuration Compare CAN Configuration Compare :material-information-outline:{ title="Minimum Version: 8.86.2.4" } lets you inspect two `.clc` CAN configurations side‑by‑side and optionally merge any differences back into the active document before downloading the result to an ECU. 1. Launch System Monitor - Ensure the project you want to update has a CAN configuration document open in System Monitor (this becomes the **Main** side of the dialog). 2. Choose **Logging > CAN Configuration Compare** from the menu or press **Ctrl+F9**. System Monitor briefly locks communications and runs offline while the dialog is open to prevent background threads from modifying writable CAN parameters. - The dialog stays on top until you press **Save** (applies changes) or **Cancel** (closes without modifying the active CAN configuration). ![CAN Configuration Compare](./assets/can-config-compare-menu.png){ width="200" } !!! info "Reading the comparison" - The dialog is divided into two mirrored panes. The **Main** pane (left) is read-only and always reflects the currently active CAN configuration; its text box shows the full path of that document. - The **Compare** pane (right) starts empty. Click the **`...`** button to open any `.clc` file. You can only open documents that use the CAN configuration template, and only the first CAN configuration inside that file is used for comparison. - The **Compare** text box and label update with the loaded file name. Press **Unload** to drop the comparison document, empty the right tree, and clear the copy buttons so you can start over with another file. - Check **Show Differences Only** to hide every node that matches (green items), leaving only the red nodes that carry mismatches. Unchecking the box reruns the comparison to reveal the full structure again. - Both trees use color‑coding: **green** nodes match, while **red** nodes differ or are unmatched in the opposite tree. The parent branches also turn red when any child is different. - Clicking a node on either side automatically selects its best match on the other tree and opens the detail panel below it. Each side hosts a dedicated detail mini-dialog, so you can inspect either bus-level, buffer-level, message-level, or parameter-level properties without leaving the window. - **Bus detail panel** shows name, buffer count, global mask, link speed, controller ID, parameter address, and block size. - **Buffer detail panel** shows the buffer name, receive mask, rate, number of messages, and the list order of those messages. - **Message detail panel** lists the description, standard/frame flag, buffer assignment, message length, parameter count, multiplexing attributes, byte order, start bit, bit length, and the number of values. - **Parameter detail panel** surfaces the parameter name, start bit, bit length, signed/unsigned, byte order, scaling gain/offset, multiplexer ID/value, and mask. - The detail panels highlight only the fields that differ. - Right-clicking on the Compare tree pops up a context menu that mirrors the `<`/`<<` buttons and shows tooltip text (e.g., “Merge 'MainBus' details”). ![CAN Configuration Compare](./assets/can-config-compare-layout.png){ width="500" } 3. Merging differences - The `<` button (Copy Selected) copies the currently highlighted Compare node into the Main tree. It works for buses, buffers, messages, and parameters. The command overwrites the selected node, and—when appropriate—adds missing messages or parameters to the Main configuration and preserves existing parameter objects. - The `<<` button (Copy All) is designed for whole branches. When you select a bus or message and press `<<`, the dialog copies that node plus every child buffer/message/parameter so the entire subtree matches the Compare side. - Both buttons remain disabled until you select a **red** node in the Compare tree, and the selection also enables the detail panels for a quick sanity check. - After any merge, the dialog reruns the comparison, collapses/expands the tree to refresh the view, and re-evaluates the `Show Differences Only` filter. 4. When you are happy with the merged result, click **Save** (default button) to close the dialog with `IDOK`. - If System Monitor is connected to an ECU and the ECU is not running in read-only mode, you will see the prompt: _“Do you want to download a new logging configuration to the ECU?”_ Confirming that prompt pushes the merged configuration down to the ECU. - If the ECU connection is not available or CAN-only behavior is enabled, System Monitor exports the merged CAN messages instead. - Press **Cancel** to close the dialog without persisting any merges. The temporary configuration copy is discarded, and the active CAN configuration remains untouched. !!! tip - Always load the correct base CAN configuration before opening the compare dialog—the left tree is read-only and cannot be swapped inside the dialog. - Unload a compare file before selecting a different one to avoid keeping multiple documents open at the same time. - You can toggle **Show Differences Only** at any time to keep the trees focused on mismatches; unchecking the box reruns the comparison and shows every node again. - The Compare tree tries to match nodes by name or ID, so renamed buses/messages may still appear as differences even when the parameters inside line up. - Right-click on the Compare side to see context-aware merge descriptions and to invoke the same `<`/`<<` actions without moving your mouse to the buttons. - After closing the dialog with Save, use the standard System Monitor workflow to save the project and to revisit any downstream logging or download steps. ## CAN FD ??? "What Is CAN FD?" CAN FD (Controller Area Network with Flexible Data-rate) is an extension of the classic CAN protocol that provides: - **Larger payloads** — up to 64 bytes per message frame (classic CAN is limited to 8 bytes). - **Higher data-phase bit rates** — a separate, faster bit rate for the data portion of the frame while keeping the arbitration phase at the standard speed. - **Defined frame sizes** — CAN FD uses discrete data length values: 0–8, 12, 16, 20, 24, 32, 48, and 64 bytes. ### How to Configure CAN FD **Prerequisites** - A hardware controller that supports CAN FD (the controller's PGV/description file must declare CAN FD support and available data speeds). **Steps** 1. **Connect** to a CAN-FD-capable unit. The CAN Bus configuration page will show the **"CAN-FD Supported"** label and the **Data Speed** dropdown. ![CANFD](assets/can-config-fd.png){ width="200" } 2. **Set the Data Speed** — Select the desired data-phase bit rate from the **Data Speed** dropdown on the CAN Bus configuration page. The **Link Speed** (arbitration speed) is configured as before. 3. **Create or edit a CAN message** — On the CAN Message configuration page, tick the **CAN FD** checkbox to mark the message as a CAN FD frame. ![CANFD](assets/can-config-fd-msg.png){ width="200" } 4. **Set the message length** — Use the length spinner to choose a valid CAN FD data length (up to 64 bytes). The control will snap to the nearest valid CAN FD length. 5. **Add parameters** — Configure parameters as usual. Start bit values can now range from 0 to 511 for CAN FD messages. 6. **Review the layout** — Use the **Parameter Layout** button to open the layout grid and verify parameter placement across the full CAN FD frame. 7. **Download the configuration** — Download to the hardware unit as usual. The system automatically selects the correct parameter structure format (V1 for classic CAN, V2 for CAN FD) based on the controller capability. ### Valid CAN FD Data Lengths CAN FD does not support arbitrary data lengths above 8 bytes. The following data lengths are valid: | Bytes | Bits | |-------|------| | 0–8 | 0–64 | | 12 | 96 | | 16 | 128 | | 20 | 160 | | 24 | 192 | | 32 | 256 | | 48 | 384 | | 64 | 512 | The message length control in System Monitor enforces these values automatically. --- # Overview Configure and calibrate sensors in TAGtronic systems to convert physical properties into electrical signals for logging and analysis. System Monitor has the following facilities to configure the way that Sensors are used: - Sensors & Actuators - Used to apply Calibration data to sensors, enable/disable individual sensors and enter values for sensor constants. Used on sensors and actuators defined in the Program Version. - External Inputs Configuration - Used to configure sensor inputs and apply Calibration data to sensors connected to the ECU. Used on sensors that are not defined in the Program Version. Also used to configure systems using communication links such as TAG Sensor Bus or CAN. - Sensor Calibration Tool - Used to view and edit sensor calibration curves. ## Configuration Tools ### [Sensors & Actuators](./sensors-and-actuators.md) For Program Version defined sensors: - Apply calibrations - Enable/disable sensors - Assign serial numbers - Set sensor constants !!! note Requires associated Sensor Set file ### [External Inputs Configuration](./external.md) For non-Program Version sensors: - Set up input sources and buses - Configure and assign sensors - Apply calibrations - Supports drag-and-drop ## [Software Calibration](./software-calibrated-sensors.md) TAGtronic systems use lookup tables for software calibration: - `.dat` files contain calibration data - Convert to `.bcf` using `Sbinary.exe` - Store in Multi-Application Base root !!! important Sbinary must be configured in Advanced Settings --- # External Configuration The External Inputs Configuration Tool allows sensor inputs to be configured and calibration data to be applied to sensors connected to the ECU. External Inputs Configuration operates on sensors that are not defined in the Program Version. Use Sensors & Actuators Tool for sensors and actuators that are defined in the Program Version. It is also used to configure systems using communication links such as TAG Sensor Bus or CAN. To open the External Inputs Configuration Tool, select: **Sensors > External Inputs Configuration**. !!! note If this menu item is not available, select: **File > New** and then select **External Inputs Config** from the list of documents in the File New dialog box. External Inputs Configuration is only available if the relevant Customer Base has permission, as set in Customer Base Advanced Settings. ## Prerequisites To change the External Inputs configuration in the ECU, you must download it from System Monitor. To modify an existing External Inputs configuration in the ECU, that configuration must first be uploaded to System Monitor. The External Inputs parameters are shown in the Parameter Explorer. The **Parameter Properties - Sensor** tab shows information about the external inputs configuration for the selected sensor. ## Configuration Steps To configure a DATALab system repeat these steps as required until the system is fully configured. The configuration can then be sent to the ECU and/or saved as a file. ### Select An Input Source The left-hand side of the dialog box displays the sensor inputs to the unit. The type of unit is shown at the top with the input sources listed below it. System Monitor only shows the input sources that are available for the unit. ![Inputs](./assets/input-source.png) TAGtronic data loggers and ECUs can accept input from the following sources: - **TAG Sensor Bus** - The communication link in the DATALab system. - **CAN Link** - A serial data communication protocol. - **Unit Sensors** - Sensors which are connected directly to the unit. ### Select a Bus To select a bus: 1. Expand the **TAG Sensor Bus** item by clicking the expand icon. 2. Click on the **Bus** (or Loom) to configure. 3. Select the type of Sensor Node to configure by selecting **None**, **SN-6**, or **SN-32**. 4. Click **Apply** to accept the configuration. 5. The tree structure expands to allow you to select a Node. ![Bus Selection](./assets/bus.png) ### Select a Node After you have selected a Bus, the tree structure expands to allow a Node to be selected. ![Node Selection](./assets/node.png) To select a node: 1. Select the **Node** that you want to configure in the left-hand pane. 2. Enter the address of the Node in the **Serial Number** text box in the Node panel. The address is set by hardware and is printed on a label on the case of the sensor node. > **Caution:** An address of 0 means that the node is not connected and cannot be configured. If you change the address of a configured node to 0, all its settings are lost without warning. 3. Select the type of Sensor Node from the **Type** drop-down list. There is only one type of SN-6. 4. Click **Apply** to accept the configuration. The node in the tree structure expands and the **Timer Resolution** boxes appear. 5. Select the **Timer Resolution** from the drop-down lists. Timer resolution is used for pulsed speed sensors. Two timer resolutions are available. The range is from 400 nSecs to 51.2 µSecs, doubling with each step. The default is 400 nSecs. ### Add a Sensor After you have selected a Node, sensors can be added to it. The tree structure shows the configuration of sensor inputs to the unit. ![Sensor Addition](./assets/sensor.png) To add a sensor to the system: 1. Expand the **Node** where you want to add the sensor by clicking the expand icon. If you want to connect directly to the unit, expand **Unit Sensors**. 2. Select the sensor location by clicking on it. 3. Select the **Type** of sensor from the **Type** drop-down list. The list is taken from the Sensors.ini file. Some locations are only suitable for particular types of sensor. The list only shows those sensors which are suitable for the location. The **Harness** text box shows the pin and socket to which the sensor must be connected. > **Note:** Speed sensors (Hall effect or Inductive) require further scaling. For these sensors, the **Advanced** button is enabled. Click this to open the **Revs Scaling** dialog box. 4. Select the serial number of the sensor from the **Serial No.** list. For Software Calibrated Sensors, this associates the appropriate calibration data with the sensor. The calibration data is shown graphically. For other sensors, select zero. If no serial numbers are available, the box is grey and a serial number of zero is assumed. > **Note:** A serial number can also be assigned to a sensor by Sensors & Actuators. This is not linked to the External Inputs Configuration, so you will not be prevented from using the serial number more than once. 5. Enter the **Name** to be used for the Measurement Parameter generated by the sensor in the **Parameter** text box. The **Identifier** is fixed by the location of the sensor in the DATALab configuration and cannot be edited. 6. Enter the **Description** to be used for the Measurement Parameter generated by the sensor in the **Description** text box. 7. Click **Properties** to open the **Parameter Properties** dialog box. The Parameter Properties are the same as for a Measurement Parameter but more attributes can be edited. 8. To add another sensor, click on the appropriate location on the tree diagram and repeat from step 3. > **Note:** To select a different Node or Bus to configure, click on the appropriate location on the tree diagram. 9. When you have finished adding and configuring sensors, click **OK**. The changes made are accepted and a Measurement Parameter is added to the External Inputs Parameters Group for each newly added sensor (the Group is created if it doesn't already exist). A message box opens asking whether to download the configuration. If you select **No**, the configuration is retained in the project and may be saved or downloaded at another time using the **External Inputs Download** command on the Logging Menu. ## Speed Sensor Scaling The **Revs Scaling** dialog box is opened for Inductive and Hall Effect sensor types by clicking **Advanced** on the **External Inputs Configuration** dialog box when a Sensor is being added. Speed sensors provide a pulse signal when they sense the movement of a piece of ferrous metal. This signal is usually caused by the rotation of a toothed wheel, and the output of the sensor is converted to a road speed or revolution value. Use the **Revs Scaling** dialog box to configure the conversion of the sensor output to give the desired parameter. ![Revs Scaling](./assets/speed-sensor.png) ### Configuration Options **Output Units** - Select the units that you want the sensor output scaled to: - **MPH** - Road speed in miles per hour (default) - **KPH** - Road speed in kilometres per hour - **RPM** - Engine speed in revolutions per minute - **Hz** - Frequency in hertz (cycles per second) - **µSecs** - Time interval in microseconds **Wheel Radius/Circumference** - For MPH and KPH scaling only. Enter the size of the wheel in metres. Entering a value in one field automatically recalculates the other. For example, the circumference value is updated if a value for a radius is entered. The Radius value must be non-zero. **No. of Teeth** - Enter the number of teeth on the wheel. Automatically set to 1 if the Output Units are Hz and 0 if the units are µSecs. For MPH, KPH, or RPM, the value must be non-zero. **Edge Trigger** - Choose which edge of the sensor signal will trigger the timing pulse: - **Positive Edge** - The trigger detects a rising voltage (default) - **Negative Edge** - The trigger detects a falling voltage **Timer** - Select the Timer from the drop-down list. The default is Timer 0. The Timer Resolution is set when you [Select a Node](#). **Max. Speed** - A nominal maximum speed can be entered. This is optional. **Max. Speed Accuracy / Low Speed Cut Off** - If a Max. Speed has been entered, System Monitor calculates the Max. Speed Accuracy and Low Speed Cut Off from the Timer Resolution and the Number of Teeth. These are read-only items. ## Input Signal Configuration Tool To open the Input Signal Configuration Tool, select: **Logging > Input Signal Configuration**. ![Input Signal Configuration](./assets/input-signal.png) The Input Signal Configuration Tool provides the ability to configure analogue and digital input signals from MESL units. This tool enables you to: - Manually configure analogue and digital input signals - Sample the signal/sensor's transfer function - Export an input configuration to file - Import an input configuration from file - Recalculate the offset - Obtain an overview of how all inputs are configured - Manage the Input Scaling regime - Manage the Hardware and Filter configuration **Scaling Methods** The tool supports the following scaling methods: - Interpolation Table - Look-up Table - Gain and Offset - High and Low Sample Points - Extrapolation - Rotational - Wheel Speed (Metric) - Wheel Speed (US) When you click the **OK** button in the dialog, all modifications are accepted and written to the data version file, which must be downloaded to the unit for the modifications to be applied. If you have re-sampled one or more signals, the logging configuration file containing these modifications must also be downloaded to the unit. The status of both files (data version and remote logging configuration) can be viewed in the ECU Bar Display. Within the System Monitor Parameter Explorer, all scaled measurement parameters are grouped in a new top-level menu called **Input Signals**. These parameters also appear in an **Input Signals** group within ATLAS logged data. **Context Menu** By right-clicking the explorer tree on either a group node or an individual input signal, you are given the following options: - **Import Signal** - Exports the selected signal properties to a CSV file. - **Export Signal** - Imports the selected signal properties from a CSV file. - **Clear Signal** - Clears the selected signal, resetting all values to 0. If you right-click the group header node (for example, **AN01-AN12(400kSPS)**), the same context menu appears; however, the selected action will be performed for all signals in that group. --- # Software Calibrated Sensors The output of many sensors is inherently non-linear (the output is not proportional to the input). A linear output is often achieved by limiting the sensor range and adding complex compensation circuitry. This reduces sensor usefulness and increases cost. An alternative is to accept the non-linear output and use the ECU's computing power to compensate by referring to a lookup table. This is known as Software Calibration and is supported by all TAGtronic control systems and sensors. This method lets you use simple, robust, and relatively low-cost sensors while obtaining the best possible accuracy. ## Calibration Files When the project is created, the Sensor PRM file is read and the latest ASCII (.dat) Sensor Calibration data is converted to a binary calibration file by Sbinary.exe. Sbinary is a conversion utility that runs in DOS. The name and location of the Sensor.prm file is defined in the Advanced Settings dialog box. **Features** - The binary calibration file is stored in the Multi-Application Base root directory and has the same name as the .prm file but with the .bcf extension. - When data is downloaded to the ECU, a sensor configuration created from the .bcf file is also downloaded. - If the .prm file or any .dat files are newer than the most recent .bcf file, Sbinary runs to update the .bcf file when Sensors & Actuators is opened. - **Note:** You can also run Sbinary.exe using the Force Prm Check command for the Sensors & Actuators Tool. - Sbinary is supplied with sensor calibration data obtained from McLaren Applied. To install Sbinary, run Install.bat from the calibration data disk. - **Note:** Ensure the Location of sensor calibration files in Advanced Settings matches the Sbinary installation location. - Sbinary requires the following file structure: - Sbinary.exe must be in the same directory as the Sensor PRM file. - You must have write access to the directory containing Sbinary.exe to allow it to create temporary files. - The .dat calibration files are placed in subdirectories below the directory containing Sbinary.exe and the Sensor PRM file. - The names of .dat files and their containing directories must conform to DOS 8-character naming rules. Long names must not be used. ## Sensor Calibration Tool The Sensor Calibration tool allows sensor calibration curves to be viewed and edited. To open the Sensor Calibration dialog, select: **Sensors > Live Calibration**. If the ECU is online, the Select Live Calibration Parameter dialog appears first. If not required, click **Cancel** to open the Sensor Calibration dialog. The dialog box has three panes: - **Left pane:** Contains a list of sensors from the .prm file. Only sensors with calibration data are shown. Click `+` to show serial numbers of individual sensor types. If `+` is unavailable, the default calibration is used. The selected Sensor Type and Sensor Number also appear in the text boxes at the bottom. - **Centre pane:** Shows the calibration data for the selected sensor from the .dat file: - **Voltage:** The sensor output in volts. Only 0-5 volt output sensors are supported; other sensors require an interface amplifier. - **Value:** The quantity in engineering units associated with the sensor output. For example, a pressure sensor with a 0-10 bar range may have a value of 10 at 5 volts, 5 at 2.5 volts, and 0 at 0 volts. - **Right pane:** Displays calibration data as a graph. The graph also appears for selected sensors in External Inputs - Configuration. ## Live Calibration Live Calibration allows you to calibrate a sensor using real input. 1. Set up a connection to the sensor using External Inputs Configuration. Ensure the configuration is downloaded to the ECU. 2. Prepare to apply known engineering loads to the sensor. 3. Put the ECU online. 4. Select: **Sensors > Live Calibration** to open the Select Live Calibration Parameter dialog. 5. Select the parameter associated with the sensor you want to calibrate. 6. Click **OK** to open the Sensor Calibration dialog. 7. On the Sensor Calibration dialog, click **Live Cal** to start the calibration wizard. The current voltage output from the sensor appears in the Voltage text box and under Breakpoint in the centre pane. 8. In the Engineering text box, enter the value in engineering units corresponding to the current load on the sensor. Click **Next**. 9. Change the sensor load so the Voltage value increases and repeat step 8. 10. Continue until you have entered all calibration points. Click **Finish** to stop the calibration. 11. Click **OK** to accept the calibration. The .dat file associated with the sensor is updated. --- # Sensors and Actuators Sensors & Actuators allows you to apply calibration data to sensors, enable and disable individual sensors, and enter values for sensor constants. It operates on sensors and actuators defined in the Program Version. !!! note - Sensors & Actuators is available only if the relevant Customer Base has permission (set in Customer Base Advanced Settings). - Use External Inputs Configuration for sensors that are not defined in the Program Version. ## Sensor Sets Sensor sets are files containing the sensor and actuator data. They are stored in the `\SenSets\` sub-directory of the Multi-Application Base. When a Sensor Set is opened (File > Open) the Sensors & Actuators dialog box for the Sensor set is opened and the set is associated with the Project. ### Creating new Sensor Sets - To create a new sensor set select: File > New. The Sensor Set Upload command automatically creates a new sensor set file if there is not already one associated with the Project. !!! note The new file only contains sensor data for the Active Applications. Ensure that you make the appropriate applications active before creating a new sensor set file. Default name for a new sensor set: - `XXXXXnnn.set`, where: - `XXXXX` — first five characters of the Application Name, or `Calib` if more than one application is active. - `nnn` — a three-digit hex number, starting with `001`, incremented whenever a new sensor set file is saved. ### Saving Sensor Sets - If changes are made to the sensor data (including uploading from the ECU), you will be prompted to save the sensor set when you close the Sensors & Actuators dialog box. - To save under a different name: File > Save As. !!! note To see which Sensor Set files are in use and to upload or download files, use the Unit Status command in the Tools menu. ## Sensors & Actuators dialog box The dialog box is opened by one of the following actions: - Opening an existing Sensor Set (File > Open or Sensors > Open Sensor Set). - Uploading an existing Sensor Set (Sensors > Sensor Set Upload). - Creating a new Sensor.Set file (File > New). The dialog box is opened for a single Sensor set. The Sensor.Set file name is shown in the title bar. A Sensor set links each sensor and actuator connected to an ECU to its calibration data. The `Sensors.ini` file lists the sensors and actuators connected to the ECU and defines on which tab they are placed in the Sensors & Actuators dialog box. Tab allocation is arbitrary and is not linked to the Groups described in Names, Descriptions & Groups. Each tab has the following columns: - Name — allocated to the sensor by the `Sensors.ini` file. Note: This name is not linked to the Name or Description of the Measurement Parameter that the sensor generates. - Type — allocated to the sensor by the `Sensors.ini` file. More than one sensor can be of the same Type. Note: This Type is not the same as the Type described in Names, Descriptions & Groups. - Serial — each TAGtronic sensor is unique and individually calibrated. To ensure correct calibration data is used, the serial number in this column must match the serial number of the device. - If the serial number is shown as `0`, a default set of calibration data is used. - If the serial number is shown as ``, System Monitor was unable to find a default set of calibration data. If there are also no numbers in the drop-down list, the `.dat` file was most likely not found. - Date — the date that the sensor type was calibrated as defined in the `.dat` file. - Status — general comment about the sensor (for example, “Passed Inspection”). ![Dialog Box](./assets/dialog-box.png) !!! tip - The Notes text area allows up to 238 characters to be entered, which are stored in the ECU. (If Unicode ASCII format is used, only 119 characters may be entered.) - To print the information in the dialog box: File > Print while the box has focus. ## Sensors & Actuators — Serial Numbers Assign calibration data to sensors or actuators: 1. Open the Sensors & Actuators dialog box. 2. Select the tab on which the device is listed. 3. Select the row for the device. If you select more than one row (using CTRL or SHIFT), you can automatically assign sequential serial numbers. 4. Click the cell in the Serial column for that row to open a text box. To open the text box with the keyboard, type any number. To assign sequential serial numbers to multiple devices, type any number to open a text box for the top row of the selection. 5. Select a serial number from the drop-down list. It is recommended to use the drop-down list rather than entering a serial number manually, as it only includes valid numbers for calibrated sensors. Uncalibrated sensors may be given any serial number. - When the text box is open, use the UP and DOWN cursor keys to select a number. 6. Accept the selected serial number by pressing ENTER or double-clicking the number. To leave the row without changing the value, press ESC or click another row. - If more than one row is selected: the first item is assigned the chosen serial number and each subsequent item receives a number one higher (if available). If a serial number is not available, `0` is assigned to that device. If `0` is entered for the first device, all selected devices are assigned `0`. - A serial number can only be assigned to one sensor of a given Type. If you choose a serial number that is already assigned to another sensor, the following warning is displayed: - `image\S_A_Serial_Warning.gif` - Select YES to assign the serial number to the newly selected sensor; the previously assigned sensor will be assigned a serial number of `0`. - Select NO to assign a serial number of `0` to the newly selected sensor; the previously assigned sensor retains its number. > **Note:** > - A serial number can also be assigned by the External Inputs Configuration. This is not linked to Sensors & Actuators, so you will not be prevented from using the serial number more than once. > - Injectors have a further Class characteristic. The Class is associated with the serial number. System Monitor ensures that injectors of the same Type also have the same Class. 7. Repeat from step 1 until all allocations are complete. 8. Save the file. 9. Use Download on the Sensors & Actuators sub-menu of the Tools menu to send the changes to the ECU. ## Sensor and Actuator Constants Some sensor types require one or more constant values to be entered to complete their calibration. Constants can be placed on any tab in the Sensors & Actuators dialog box and may be mixed with sensor descriptions. For clarity, constants are usually placed on a separate tab. Assign a value to a constant: 1. Select the row for the constant. 2. Enter the value. A text box opens automatically. The Edit Context Menu becomes available. 3. To accept the value, click another row or press ENTER. To leave the row without changing the value, press ESC. The data on the Constants tab is part of the Sensor set data. It is sent to, and obtained from, the ECU with the sensor set data. ## Enabling and Disabling Sensors and Actuators The Enable/Disable Sensors dialog box lists all Sensors and Actuators which can be enabled and disabled by System Monitor. The `Sensor Enable/Disable.ini` file defines these sensors and actuators; they do not have to be the same as those listed in the Sensors file. ![Enable](./assets/enable-disable.png) To open the dialog box: Sensors > Enable/Disable. - The list can be sorted by Name or Group. Names and Groups are taken from the Enable/Disable file and are not the same as the names and groups in the Sensors & Actuators dialog box. - Enable/Disable Sensors only operates on a connected ECU and its settings are not saved in the Sensor Set files. When an ECU is connected, the State column shows whether each sensor or actuator is enabled or disabled. Change the state of a sensor: 1. Select the row for the device by clicking or by using the UP and DOWN cursor keys. To select multiple rows, use CTRL or SHIFT in conjunction with the mouse or the Space Bar. 2. Right-click and select Enable or Disable from the Context menu. - A message is sent to the ECU to enable or disable the sensor. When the change has been made, the State column updates to confirm the new status. !!! note Pressing ENTER toggles the status of the selected sensors between Enable and Disable without using the Context menu. - Disabling a sensor makes a change in the ECU memory that prevents the output of the sensor from being used in any way. The logging of a sensor can be switched off without disabling it if required. --- # Obtaining Sensor Calibration Data Sensors and actuators supplied by Motion Applied are tested at multiple points in their operating range to produce a calibration table. This table is saved as an ASCII format file with the `.dat` extension and is shipped with the sensor. !!! warning - Do not amend the sensor calibration database while System Monitor is running - Do not edit the contents of `.dat` files You can also obtain calibration information by connecting to the Motion Applied network using the Get Sensor Data Utility. ## Get Sensor Data Utility ### Access Requirements To obtain sensor calibration data from the Motion Applied network: 1. Apply for a remote login account 2. You'll receive setup instructions (WPD.TWPD0176) for Dial-up Networking via modem 3. You'll be allocated a username and initial password (which you can change) ### Network Access You'll have access to: ``` TAGSVR1\SYS:\HOME\username\ ``` This path maps to the `F:` drive (if available). Access is restricted to the network supervisor, whether within or outside Motion Applied. The `gs` (Get Sensor Data) utility retrieves TAG sensor calibration data on a read-only basis and cannot modify database master files. ## Using the Get Sensor Utility The `gs` utility is located at: ``` TAGSVR1\SYS:\HOME\username\FROMTAG\SENSOR ``` 1. Run `gs` to open a DOS window with the prompt: ``` Enter Order Code [TYPE QUIT TO EXIT]: ``` 2. Enter the full order code (include leading `o` and all spaces; case-insensitive): ``` o 030 330 001 001 ``` 3. Confirm the sensor type when prompted: ``` The order code you have entered corresponds to: ––– 250 bar Pressure Sensor ––– is this correct? [Y/N] ``` Enter `Y` to accept. 4. Enter the quantity of sensors needed. Precede with `R` for automatic sequential serial numbering. 5. Enter the serial number for each sensor (e.g., `160`). 6. The utility retrieves data and appends it to `SENSOR.DAT` in the current directory. For amplified pressure sensors, a second file `SENSOR2.DAT` is created. 7. Repeat steps 4-6 for each sensor type required. 8. Type `QUIT` at the prompt to exit. The filename and size are displayed. 9. Transfer the `OUTPUT` subdirectory contents to your computer and copy to CD-ROM. !!! note Exit at any time by entering `quit` or `QUIT`. If an order code or serial number isn't recognized, you'll be prompted to re-enter it. --- # Overview Tuning is the process of optimising vehicle performance by adjusting the values of **Editable Parameters** within the ECU (Engine Control Unit). These adjustments can be made in several modes depending on the setup and workflow. ## Tuning Modes ### Off-line Tuning - Changes are saved in a **New Data Version**. - The ECU does **not** need to be connected during editing. - Changes are downloaded to the ECU at a later time. ### On-line Tuning - The ECU is connected to **System Monitor** during editing. - Changes can be sent immediately using **Live Tune** or on demand. ### Automatic Tuning - System Monitor can operate with an automatic **Testbed**. - Supports the **AVL PUMA** protocol for unattended engine tuning. ## [Adjustment Board](../hardware/index.md/#adjustment-board) To simplify value adjustments, an **Adjustment Board** can be connected to System Monitor. This allows physical control over parameter changes using potentiometers. --- # Editable Parameters Editable Parameters are values within the ECU that can be modified to optimise engine performance. These parameters are defined in the ASAP file by the ECU manufacturer and may be edited depending on permissions. ## Scalar Parameters A **Scalar** parameter holds a single constant value. ![Scalar Window](./assets/scalar.png) **Scalar Edit Window** - Displays multiple Scalar parameters in rows. - Columns include: - **Name/Description**: Configurable via Window Properties. - **Value**: Editable; colour changes on acceptance (default red/blue). - **Trend Arrow**: Indicates increase or decrease. - **Units**: Shown if Engineering format is selected. - **Address**: Shown if Raw Hex or Binary format is selected. !!! note Scalar parameters can also be edited in a Grid window. ## 1-Axis Maps A **1-Axis Map** is a one-dimensional lookup table. The output value depends on the selected **Breakpoint** on the axis. **Features** - Active cell outlined in black when ECU is online. - Colour changes on value acceptance. - Optional **String Bar** displays associated string values. - **Edit Bar** shows Breakpoint and Output values. - Display options: Engineering, Raw Hex, or Address. - Graphical format available. - Orientation can be rotated via System Menu. ### Multiple Parameters - A 1-Axis Map window can include multiple parameters sharing a common axis. - Parameters not sharing the axis are excluded with a warning. - Orientation can be toggled to display parameters in columns. ![1 Axis Map with Multiple Parameters](./assets/1-axis-multiple.png) ## 2-Axis Maps A **2-Axis Map** is a two-dimensional lookup table. Output depends on selected Breakpoints on both axes. ![2 Axis Map](./assets/2-axis-map.png) **Features** - Output and Breakpoint values are editable; colour changes on acceptance (default red/blue). - Active cell outlined in black when ECU is online. - Grid structure includes: - X-axis identifier name row - X-axis Breakpoint values row - Y-axis rows with Output values and Y-axis identifier/Breakpoints - Optional **String Bar** displays associated string values. - **Edit Bar** shows X-axis Breakpoint, Y-axis Breakpoint, and selected cell values; non-editable boxes display Index Parameters and Output value when online. - Window and axis titles can display Name, Description, or both (configured in Window Properties). - Display options: Engineering, Raw Hex, or Address format. - Graphical format available. - Live Tune Overview minimizes empty cells for better visibility. - Orientation can be exchanged via System Menu. - Appearance controls available in Window Properties Display tab. ## Arrays An **Array** is a one-dimensional ordered group of values selected by an index. ![Array Window](./assets/array.png) **Features** - Only one Array parameter per window. - Editable values with colour change on acceptance. - Grid layout includes: - Output values (rows/columns configurable) - Index row - Optional **String Bar** and **Edit Bar** - Window title can show Name, Description, or both ## Strings A **String** parameter outputs text rather than numerical values. ![String Window](./assets/string.png) **Features** - Multiple String parameters can be displayed. - Editable within defined length (set in ASAP file). - Columns include: - **Name/Description** - **Value** - Column widths and sorting are configurable. - Can be associated with Maps or Arrays. !!! note - Editable Parameters must be part of an **Active Application** and not marked as **Read Only**. - Editing permissions are controlled by the ASAP file. --- # Changing Values Each Parameter is defined in the ASAP file by the ECU manufacturer. The ASAP file either permits or prevents System Monitor from changing the values of an Editable Parameter. If the Editable Parameter values are prevented from being changed, the edit boxes are disabled. You can only change values of parameters in an Application that is Active and not Read Only. When a value is changed and accepted, its colour changes to the Greater than / Less than colour (default red/blue) as appropriate. These colours are defined in the Display tab of the Window Properties dialog box. The Scalar Display uses arrow icons to show the direction of value changes. ## Editing options - Edit commands are available on the Edit Context Menu. - Use the Maths Bar to carry out mathematical operations on the selected value or values. - The Breakpoint Values of 1-Axis and 2-Axis Maps can be edited if required. - To restore values to their original values, select: Edit > Restore or use the Windows Context menu. ## Parameter Edit Mode A single value of an Editable Parameter can be displayed with Edit Mode either On or Off. Entering a new value turns Edit Mode on and the value is overwritten. Edit Mode is turned on when a single cell is selected using the mouse. ![Edit Bar](./assets/edit-bar.png) **Edit Mode On** - Only the text is highlighted. - Only one cell can be selected. - The Edit Context Menu is available. - If Copy is selected, only the value is sent to the clipboard. Paste and Paste Special can only paste the value as text. To turn Edit Mode Off, press ESC or ENTER. **Edit Mode Off** - The whole cell (or row in Scalar or String Windows) is highlighted. - Blocks of cells can be selected. - The Windows Context Menu is available. - If Copy is selected, the value plus a reference to the original parameter is copied to the clipboard. All Paste and Paste Special options can be used. Pressing a cursor key while the cell is in Edit mode selects the next cell in the direction of the arrow with Edit Mode off. If multiple values are selected, Edit mode is always off. Editing a number overwrites the first (top left) of the multiple values. **Changing Parameter Values by Mouse** Click on the value to be changed and enter a new number to overwrite the existing value. Alternatively, select the appropriate text box in the Edit Bar and enter a new value. To select more than one value do one of the following: - Click and Drag to select a continuous rectangle of values. (If the value is in Edit Mode press ESC before dragging) - Click on a corner of the rectangle of values you want to select. Hold down the SHIFT key and click on the opposite corner - To select non-continuous values, hold down the CTRL key while clicking on values. (Non-continuous blocks of values cannot be copied.) If multiple values are selected, entering a value overwrites the first (top left) of the multiple values. The Maths Bar operates on all the selected values. !!! note It is also possible to change values by dragging with a mouse in a Graphical Map. **Changing Parameter Values by Keyboard** Select the value to be changed using the cursor keys and enter a new number to overwrite the existing value. Alternatively, use the TAB key to cycle through the edit boxes in the Edit Bar. 2 axis Map Edit Bar To select more than one value: - Using the cursor keys, select a cell at a corner of the rectangle of values you want to select. Hold down the SHIFT key and select a cell at the opposite corner. - To select non-continuous values, hold down the CTRL key while using the cursor keys, press the SPACE bar to include values. (Non-continuous blocks of values cannot be copied.) If multiple values are selected, entering a new value overwrites the first (top left) of the multiple values. The Maths Bar operates on all the selected values. ## Value Limit Warning Each Parameter has an upper and lower limit that is set in the ASAP file. The limits can be viewed in the Conversion tab of the Parameter Properties dialog box. ![Limit Warning](./assets/limit-warning.png) If a value is changed so that it is outside the limits, the Limit Warning dialog box opens. The Upper and Lower limits are shown in the Limits panel. If the Parameter Properties can be edited, the Upper and Lower limits text boxes are editable and the limits can be changed. In the Value panel: - Entered — Shows the out-of-range number that was entered. If you edit this box to a number that is within range, the new number is accepted when you click OK. - Adjusted — System Monitor automatically suggests a new number, which is the value of the limit nearest to the out-of-range number that was entered. If you click OK without editing a number in the Entered text box, the Adjusted number is accepted. Buttons: - OK — Replaces the selected value with the value in the Entered text box provided it is within limits. - If the value in the Entered text box is not within limits, the selected value is replaced by the value in the Adjusted text box. - If multiple values are being changed, steps through the values one at a time. - OK to All — Available for multiple values only. As OK but applies the same change to all the values. - Cancel — The change is ignored and the original value is restored. ## Changing Breakpoint Values ### Mouse operation To select an X-axis Breakpoint click on the column header. To select a Y-axis Breakpoint click on the row. When a cell is selected, it changes to Edit Mode. Enter a new value to overwrite the existing value. !!! note If Orientation is selected in the Window System Menu, the axes are rotated. Alternatively, click on the appropriate edit box in the Edit Bar. Select the required breakpoint edit box and overwrite the value. ### Keyboard operation Use the cursor keys to select a cell that uses the breakpoint that you want to change. Use the TAB key to cycle through the edit boxes in the Edit Bar. The TAB key then returns to the value in the map table. Select the breakpoint that you want to change and enter a new value to overwrite the existing value. ## Breakpoint Warning Changing Breakpoint values affects the axis wherever it is used, not just in the current Window. If the axis is used by other Parameters, System Monitor displays a Breakpoint Warning. ![Breakpoint Warning](./assets/breakpoint-warning.png) To prevent this box from being displayed again, select: Don’t show this Warning again for this Axis. Click List to display a list of all the Parameters that use the axis that you are editing to help you to determine the effect of the change. ## Breakpoint Monotony All Breakpoint values must either increase or decrease along the axis. This restriction is called Monotony. The direction of monotony is set in the ASAP file and can be viewed in General tab of the Axis Properties dialog box. This means that a breakpoint value cannot be higher than the next highest value or lower than the next lowest. If you enter a value that breaks this rule, a warning is shown and the value is not accepted. Click OK to close the warning. The new value that was entered is replaced by the original value. Only values that are within the surrounding breakpoint values can be accepted. --- # Graphical Maps 1-axis and 2-axis maps can be displayed in several graphical formats. Select the view format from the System menu for the Window or step through the options with the View Mode button on the Map Bar. All Graphical Views can be zoomed. 2‑axis Graphical Views can also be rotated. Breakpoint spacing can be Even or Proportional. Values can be changed in all Graphical Views except Difference View. Values can only be changed in active applications. ## Available Views ### 1-axis Map - Graphical View – 1 axis A two‑dimensional graph of Parameter value plotted against Index Value. The display includes a plot for each Parameter in the Window. The name of the currently selected Parameter is displayed at the top of the Window. The trace of the currently selected Parameter is shown bold in the colour selected for Active Map in the Display tab of the Window Properties (Display sheet). Remaining traces use the colour selected for Inactive Map. - Axis upper and lower values are set in the relevant Conversion dialog boxes (see Parameter Properties – Conversion sheet and Axis Properties – Conversion). Axis subdivisions and labels are generated automatically. - Breakpoint Spacing can be Even or Proportional. - You can select all values of the selected parameter with the Select All button on the Map Bar. - Values can be changed in this view. ![1-axis Graphical View](./assets/1-axis.png) ### 2-axis Map - Graphical View – 2 axis A three‑dimensional graph of Parameter value plotted against two Index Values (X and Y). - Axis upper and lower values are set in the relevant Conversion dialog boxes (see Parameter Properties – Conversion sheet and Axis Properties – Conversion). Axis subdivisions and labels are generated automatically. - Breakpoint Spacing can be Even or Proportional. - Hidden lines are removed. To see values below the surface, use End View or Slice View. - The 3D view can be rotated and zoomed. Values can be changed in Graphical View. ![2-axis Graphical View](./assets/2-axis.png) - Difference View (1‑axis and 2‑axis) The Difference value is the amount by which the current value in the Edit Buffer differs from the value in the Data Version. Difference View plots the Difference value against the Index Value(s). Values cannot be changed in Difference View. - Unchanged values are set to zero. - 1‑axis Difference View: two‑dimensional graph of Difference value plotted against Index Value. Only one Parameter is shown; the name of the currently selected Parameter is shown at the top. The largest difference (positive or negative) is used to set the axis upper and lower values. - 2‑axis Difference View: three‑dimensional graph of Difference value plotted against two Index Values (X and Y). The upper axis value equals that used in Graphical View – 2 axis; the lower axis value is the same magnitude but negative. The 2‑axis Difference View can be rotated. - Axis subdivisions and labels are generated automatically. Breakpoint Spacing can be Even or Proportional. - Both Difference Views can be zoomed. ![Difference View](./assets/difference-view.png) - End View (2‑axis only) A two‑dimensional graph of Parameter value plotted against the X Index Value for each Y Index Value. The display includes a plot for each Y Index Value. - The trace of the currently selected Y Index Value is shown bold in the colour selected for Active Map; other traces use the colour selected for Inactive Map. - Axis upper and lower values are set in the relevant Conversion dialog boxes. Axis subdivisions and labels are generated automatically. - Breakpoint Spacing can be Even or Proportional. - Values can be edited in this view. ![End View](./assets/end-view.png) - Slice View (2‑axis only, available in Graphical View) Selected by buttons on the Map Bar. - Slice X: two‑dimensional graph of Parameter value plotted against the X Index Value for the Y Index breakpoint that includes the selected value. The trace for the selected breakpoint is shown bold in the Active Map colour; traces for higher Y Index values (behind) are shown in the Inactive Map colour. - Slice Y: two‑dimensional graph of Parameter value plotted against the Y Index Value for the X Index breakpoint that includes the selected value. The trace for the selected breakpoint is shown bold in the Active Map colour; traces for higher X Index values (behind) are shown in the Inactive Map colour. - Axis upper and lower values are set in the relevant Conversion dialog boxes. Axis subdivisions and labels are generated automatically. - Breakpoint Spacing can be Even or Proportional. - These views can be zoomed and their values are editable.# ![Slice View](./assets/slice-view.png) ## Zooming a Graphical view All Graphical Views can be zoomed using buttons on the Map Bar: - Zoom in — increases the size of the view by 50%. - Zoom out — cancels each Zoom In setting sequentially. When any level of Zoom In is selected, the view is centred on the currently selected value. As soon as a value is selected or moved, the display is redrawn to place the value at the centre (the “landscape” appears to move under the point). ![Zoom Buttons](./assets/zoom-view.png) To select a value: - On a 3D display - Click on an intersection on the surface; or - Use the cursor keys to move around the surface. - On a 2D display - Click on the trace where it is intersected by a breakpoint; or - Use the left/right cursor keys to move along the trace and the up/down keys to select a trace. - For either display - Switch to Engineering View; select a value using the mouse or keyboard, then return to Graphical View. !!! tip Use Alt+G to toggle between Graphical and Engineering views. The Zoom setting is retained for a Window even if its View mode is changed. For 2D displays, the vertical axis can be scaled using AutoZoom so that the traces fit the plot area. ## Breakpoint Spacing All Graphical Views can be shown with Breakpoint Spacing either Even or Proportional using the button on the Map Bar. - Even Breakpoint Spacing - Proportional Breakpoint Spacing ![Breakpoint Spacing Buttons](./assets/breakpoint-spacing.png) ## Rotating a Graphical view The three‑dimensional Graphical View for 2‑axis Maps can be rotated using the Mouse or toolbar buttons: - Toolbar buttons — four buttons rotate the display left/right or up/down in 10° increments. - Mouse — click anywhere on the Map window (take care not to select a value) and drag the cursor; the front of the display follows the cursor movement. ## Changing Values in a Graphical view Parameter values can be changed in any Graphical View except Difference View. Values can only be changed in active applications. !!! note In a 1‑axis Graphical View you can select all values of the selected parameter with the Select All button on the Map Bar. In all other Graphical Views you can only select one value at a time. To change multiple values. Steps to change a value in a Graphical View: 1. Select a value: - On a 3D display: click on an intersection on the surface or use the cursor keys to move around the surface. - On a 2D display: click on the trace where it is intersected by a breakpoint or use the left/right cursor keys to move along the trace and the up/down keys to select a trace. - Or switch to Engineering View, select a value using the mouse or keyboard, then return to Graphical View. (Tip: Use Alt+G to toggle between Graphical and Engineering views.) - The selected value is identified by a black spot. The Breakpoints and Value of the selected point are shown in the Edit Bar. 2. Change the value: - By mouse — drag the value up or down. If the view is Zoomed In, the display continually redraws to place the value at the centre (the landscape appears to move under the point). - By keyboard — use the TAB key to cycle through the boxes in the Edit Bar. 3. Enter a new value. 4. Press ENTER to accept the value and redraw the Map. !!! note The Maths Bar can be used to change values selected in a Graphical View. --- # Live Tune When Live Tune is on, the Edit Buffer and the ECU RAM are kept synchronised — any changes in the Edit Buffer are downloaded to the ECU as soon as they occur. ![Live Tune](./assets/live-tune.png) ## Toggling Live Tune To toggle Live Tune on and off select: ECU > Live Updates. The status of Live Tune is shown on the Comms Bar. !!! note The data changes only affect the ECU RAM. To apply the changes to the vehicle systems, the ECU must be operating from RAM. The type of memory being used by the ECU is selected from the Run from ROM and Run from RAM commands on the ECU Menu. ## If the Data Version Does Not Match If the Data Version in the Edit Buffer does not match the one in the ECU, the following message is opened for each Application when you start Live Updates. The message presents these options: - **Sync SM to Unit** Synchronises the files in System Monitor to match those in the Unit. The specified files are opened in System Monitor, replacing those that are already open. If a file cannot be found, a new file is created using the filename and modification information taken from the unit. Uploads data and places it in the Edit Buffer. Files and data which are the same as already in System Monitor are not uploaded. - **Reprogram** Reprograms the ECU then downloads the data changes in the Edit Buffer to the ECU. - **Force Complete Reprogram** Forces a Complete Reprogram then downloads the data changes in the Edit Buffer to the ECU. - **Ignore** Closes the message box. No changes are made. The download process continues with the next Application. - **Cancel** Closes the message box. No changes are made. # Live Auto Tune Live Auto Tune assists with manual testbed tuning. It is used in conjunction with Live Tune and the Adjustment Board. ## Definition File and Parameters The Live Auto Tune Definition File defines a set of Live Auto Tune Parameters. A Live Auto Tune Parameter defines a group of 2‑axis maps and an adjustment parameter. The maps in a Live Auto Tune Parameter share the same axes (for example, throttle position vs engine speed). Typically, Live Auto Tune Parameters are provided for: - ignition, - injection quantity, - injection end angle, and - nominal lambda. Turbocharged engines may have additional parameters for boost pressure and waste gate control. The Parameter Properties - Live Auto Tune Sheet gives information about the selected Live Auto Tune Parameter. ## Tune Groups and Z Axis The Live Auto Tune Parameters all have the same number of maps in a set order (e.g. 1, 2, 3). The group of maps at the same level in all the Live Auto Tune Parameters is called a Tune Group. Typically, the Tune Groups are selected for different operating conditions such as “Optimum Power”, “Minimum Consumption” or “Wet Racing”. The active Tune Group can be selected by: - the result of a calculation in the ECU, - the value of a measurement parameter, or - the driver using a cockpit‑mounted switch. The index that selects the Tune Group is called the Z axis. ## Operation and Adjustments As a test is run, the various indexes and controls select a value on one of the maps in each Live Auto Tune Parameter. The Adjustment Parameter is then applied to this value. If the Adjustment Parameter is associated with a control on the Adjustment Board, turning the control applies the adjustment. The way in which the adjustment is applied is defined by the Correction type in the Parameter Properties. To ensure that a value is selected properly, the Live Auto Tune Parameter defines breakpoint tolerances which override the breakpoint tolerances set for each map (see Parameter Properties - 1‑Axis Map Sheet). ## Saving Adjustments When you are satisfied with the changes, update the data version in the ECU by selecting: ECU > Save Auto Tune Adjustment. --- # Overview The System Monitor can be integrated with an AVL PUMA testbed system to enable unattended tuning. Communication between the PUMA system and System Monitor is established via a serial port using the ASAP3-compliant protocol. ## Capabilities Using this setup, PUMA can: - Read and modify editable parameters - Download editable parameters to the ECU - Acquire measurement parameters - Select parameter description and binary data files !!! note Only one project (description) and one embedded data (binary) file can be active at a time. --- # Testbed Operations - **Configure** — Opens the Testbed Properties dialog box (Shortcut CTRL+SHIFT+F5) - **Enable Link** — Switches the Testbed Link on and off (Shortcut CTRL+SHIFT+F6) - **Message Window** — Opens the Testbed Message window (Shortcut CTRL+SHIFT+F7) ## Link Selecting/deselecting the **Enable Link** command (Tools menu) switches the link between System Monitor and the testbed on and off. - When the link is enabled, the ECU is put On-Line. - Note: If Full Operation mode is selected on the Testbed Properties → Link tab, Live Tune is also enabled. - If the ECU is taken Off-line or Live Tune is disabled, the Testbed link is automatically disabled. - When the link is enabled, `PUMA enabled` is displayed on the Status Bar. System Monitor windows can be used to monitor parameter values while the link is enabled. The status of the link, and the information passing across it, are shown on the Testbed Message Window. The parameters that System Monitor acquires from the ECU are defined in the Live Logging Configuration. Ensure that Auto Configuration is On so that the correct data is available to send to the testbed and to display on the screen. ## Message To open the Testbed Message window select Tools > Testbed > Message Window. ![Testbed Message Window](./assets/message-window.png) The window shows information about the communication between System Monitor and the testbed. It consists of the following areas. ### Parameters Bar PUMA defines a single block of parameters, and declares the rate at which it will request those parameters. The Parameters Bar displays the following: - **Params** — the number of parameters in the block - **Rate** — the rate at which these parameters will be requested ### Message Window Shows the information which is being transferred between System Monitor and the testbed as follows: - **Commands** — Mnemonics received from the testbed, for example: ‘Select Parameters for Acquisition’. - **Arguments** — Information associated with a Command, for example: `Rate: 1s Params: Speed, Gear, Revs..` - **Messages** — Mnemonics sent in reply to a request for parameter values, for example: ‘Reply to Acquire Parameters’ - **Parameter** — Identifiers and values (in engineering units) of parameters associated with a Message. - **Debug** — When Debug is on requests repetition of the last message from PUMA to System Monitor or System Monitor to PUMA. Also requests acknowledgements from System Monitor to PUMA that the last message was received. - **Watchdog** — When Watchdog is on watchdog time-outs are shown in the following form: - `Byte timeout `, where `bto` is the byte timeout value. - `Message timeout `, where `mto` is the message timeout value. - **Errors** — possible values are: - Command not executed - Error during execution ``, where `` gives details of the command arguments in error, or the external error that occurred - Command not available ### Status Bar - **Link** — Indicates the link status: - **enabled** — Link is selected, but no bytes have been received - **disabled** — Link is not selected - **active** — Link is selected, and bytes have been received - **inactive** — Link is selected, but a message timeout has occurred - **Bytes** — the number of bytes since (and including) the last Init command. - **Message** — Indicates the current message status: - **waiting** — Waiting for a command - **received** — Received a command, not yet sent a confirmation or definitive reply - **confirmed** — Sent a confirmation, not yet sent a definitive reply - **replied** — Sent a definitive reply, waiting for handshake - **repeat** — Waiting for PUMA to repeat a command after a byte timeout - **Last Error** — Shows the last error that occurred. This is displayed even if the link has since recovered from the error. --- # Testbed Properties System Monitor must be configured to operate with the testbed. To open the Testbed Properties dialog box select: Tools > Testbed > Configure. ## Link Configure the link between PUMA and System Monitor on the **Link** tab. ![Link](./assets/link.png) The tab has the following controls: - **Mode** - **Measurement** — PUMA can take measurements from System Monitor. System Monitor only responds to the Initialisation and Data Acquisition commands sent by PUMA. - **Full Operation** — System Monitor responds to all the commands sent by PUMA. - **Comms Setup** — Opens the Serial Comm Setup dialog box. The default settings are chosen to suit most applications. If the setup is not working, check that it conforms to the requirements of your testbed. The setup is saved as part of the Project settings, so it may need to be edited for each project on which it is used. ## Watchdog Configure the watchdogs on the link between PUMA and System Monitor on the **Watchdog** tab. ![Watchdog](./assets/watchdog.png) **Message Timeout panel** - **Time Between** — Enter the timeout period. Range: 0.1 to 36,000 seconds. Resolution: 0.1 seconds. Default: 300 seconds. - **Enabled?** — When checked, the link is terminated if the time between messages exceeds the value in the Time Between field. The link is re‑established by the PUMA testbed. **Byte Timeout** - **Time Between** — System Monitor sends a request to the PUMA testbed to repeat the last command if the time between bytes of the same message is greater than the value in this field. Range: 0.1 to 100 seconds. Resolution: 0.1 seconds. Default: 1 second. This feature is always enabled. !!! note The Watchdog properties cannot be changed while the Testbed Link is Enabled. ## Debug Use the **Debug** tab to configure the Debug file. ![Debug](./assets/debug.png) - **Debug Enabled?** — When checked, the text that appears in the Testbed Message window is copied to the Debug file. If the file already exists, text is appended to it. - **Debug file** — Enter or browse for the name and path of the file where you want to save the debug information. There is no default path but the default file name is `tb_debug.txt`. The debug filename cannot be changed if the Testbed Link is Enabled, but debug may be switched on or off. ## Commands Use the **Commands** tab to configure the way System Monitor responds to [PUMA commands](puma-commands.md). ![Commands](./assets/commands.png) - **Send command confirmation to PUMA?** — When PUMA sends System Monitor a command, it expects a reply within a certain time. If System Monitor cannot execute the command and reply within that time, a command confirmation message may be sent in place of the reply. This gives System Monitor longer to execute the command and allows the definitive reply to be sent later. - When a command check box is selected, that command will be confirmed before the definitive reply is sent. - **View as** — PUMA uses different command names to System Monitor. Commands can be viewed either as the PUMA command names or as their System Monitor equivalents. ## Translate Use the **Translate** tab to configure the correspondence between System Monitor and PUMA parameter identifiers. ![Translate](./assets/translate.png) - **Command translation enabled?** — When selected, the parameter identifiers used by PUMA are translated to those used by System Monitor and vice‑versa. - **Translation file** — Enter or browse for the name and path of the file where System Monitor can find the equivalents list to translate the parameter identifiers. --- # PUMA Commands System Monitor supports a subset of commands from the AVL PUMA protocol V2.0. Commands from V1.0 or its sub-versions are not supported. All values exchanged use engineering units for: - Measurement parameters - Editable parameters - Map breakpoints - Plausible physical limits - Minimum increments ## Initialisation Commands ### `Init` Initialises System Monitor for a PUMA tuning session. When received: - Starts the Watchdog (if enabled) - Opens the Debug File (if enabled) ### `Identify` Sent by PUMA to declare protocol version and identify itself. - If version is V2.0 or later: System Monitor replies with its version (V7.0) - If version is V1.0 or sub-version: System Monitor replies with `command not available` error and rejects all subsequent commands ## Configuration Commands ### `Select Project and Embedded Data` Must follow an `Init` command. PUMA must provide: - Project file name - Embedded data file name System Monitor checks program stamps for compatibility. If valid: - Opens project and embedded data file - Returns LUN = 1 ### `Copy Embedded Data` Copies data from edit buffer to file. PUMA must specify: - Source = 3 (edit buffer) - Destination = 2 (file) - LUN = 1 System Monitor writes: - New data stamp (if filename changed) - Comment (includes changed parameters and PUMA name) - User name and date - Editable parameter values ### `Change Embedded Data Filename` Changes the filename for future saves. - Does not save data - If filename is invalid, returns `error during execution` ## Map Manipulation Commands ### `Select Map` PUMA nominates a map to read or modify. System Monitor returns: - Map number - Number of breakpoints (x and y axes) If invalid or not a map, returns `error during execution`. ### `Get Map from System Monitor` Returns: - Breakpoint values - Map site values - Minimum increment - Plausible physical limits ### `Put Map to System Monitor` Sets map values as specified by PUMA. - Ignores increment and limits - Caps values outside limits without error - Returns error if breakpoints are out of sequence ### `Get Map Value` Returns value of a single map site. PUMA must specify: - Map number - x and y index ### `Set Map` Sets all map sites in a grid range to a single value. Returns error if: - Grid range is off-map - Values are capped if outside limits (no error sent) ### `Increase Map` Increases map site values in a grid range. Returns error if: - Grid range is off-map - Values are capped if outside limits (no error sent) ## Scalar Manipulation Commands ### `Get Scalar from System Monitor` PUMA must send: - LUN = 1 - Parameter identifier System Monitor returns: - Scalar value - Plausible physical limits - Minimum increment > Cannot be used for measurement parameters. ### `Set Scalar on System Monitor` PUMA must send: - LUN = 1 - Parameter identifier - Value Returns error if: - Parameter is not editable - Value is capped if outside limits (no error sent) ## Data Acquisition Commands ### `Select Parameters for Acquisition` Specifies parameters to be acquired. PUMA must send: - LUN = 1 - Sampling rate - List of parameter identifiers - Multiple commands append to the list - Empty list clears all selections - Updates Live Logging Configuration Returns error if parameter is not in project description. ### `Switch Offline/Online` Switches ECU mode. #### Online Mode: - Downloads edit buffer changes immediately - Generates new data stamp - Accepts `Acquire Parameters` commands #### Offline Mode: - Changes affect edit buffer only - No data is downloaded - `Acquire Parameters` commands are rejected !!! warning Switching modes without saving may result in data loss. ### `Acquire Parameters` Requests current values for selected parameters. Returns: - List of parameter values (in order) If offline, returns `command not executed`. ## Special Parameters ### `APS_Errors` #### `Get Scalar` Returns: - Number of errors since link enabled - Min = 0, Max = defined limit, Increment = 1 #### `Set Scalar` Returns `error during execution` #### `Select Parameters for Acquisition` Includes error count in response to `Acquire Parameters` ### `APS_LinkOK` #### `Get Scalar` Returns: - 1 if link is active - 0 if link is inactive #### `Set Scalar` Returns `error during execution` #### `Select Parameters for Acquisition` Includes link status (1 = active, 0 = inactive) ## Unsupported Commands System Monitor does **not** support the following PUMA commands: ### Initialisation - `Emergency` ### Recording - `Define Recording Parameters` - `Define Trigger Parameters` - `Activate Recorder` - `Get Recorder Status` - `Get Recorder Result Header` - `Get Recorder File` - `Save Recorder File` - `Load Recorder File` ### Miscellaneous - `Set Graphic Mode` - `Reset Device` - `Set Format` ## Error Handling System Monitor may return the following errors: - **`command not executed`** Incorrect command sequence - **`error during execution`** Invalid arguments or external error (e.g. disk full) - **`command not available`** Command not supported in PUMA protocol V2.0 > All errors include an error code and comment. > Errors are visible in the Testbed Message Window. ## PUMA Errors Although PUMA is designed for unattended tuning, commands may occasionally fail. System Monitor can return the following error messages: ### `command not executed` - The command could not be executed due to an incorrect sequence. - Example: A configuration command sent before initialisation. ### `error during execution` - The command started but could not complete. - Causes may include: - Invalid command arguments (e.g. non-existent project file) - External issues (e.g. disk full) ### `command not available` - The command is not supported in PUMA protocol V2.0. - Typically occurs when using commands from V1.0 or unsupported groups. ### Error Details - Each error includes: - An error code - A comment generated by System Monitor ### Viewing Errors - Errors are displayed in the **Testbed Message Window** - The **Last Error** field in the status bar shows the most recent error, even if the link has recovered !!! reminder System Monitor only supports one project at a time. The LUN parameter must always be set to `1`. Any other value will trigger an `error during execution`. --- # Overview The ECU Control System Monitor allows programming and monitoring of the ECU via the ECU Menu commands. The program is loaded into System Monitor during the creation of a Program Version. Security features in the Customer Base ensure compatibility between the program and the ECU. ## Key Features - **ECU Bar**: Displays the status of the ECU. - **Unit Status**: Shows data versions and configuration files in System Monitor and the ECU units. Supports upload/download of data. - **ECU Search**: Provides detailed information about the ECU. - **Kill Engine**: Stops or prevents engine start via the Tools menu. - **Set ECU Time and Date**: Synchronises ECU time with the PC. - **Read RAM**: Accesses detailed ECU status information. --- # ECU Operations ## Selecting Applications An Application in System Monitor corresponds to a set of ECU Memory locations. Most ECU Control commands operate on the Active Applications. Commands for Setting Time and Date, ECU Properties, and ECU Search can only operate on one Application at a time. When these commands are selected, the **Select an Application** dialog box is opened. 1. Select one Application and click **OK** to continue. Commands for Reprogramming the ECU, Force Complete Reprogram, Program Spare Unit, and Upload Sensor Sets can operate on more than one Application. When these commands are selected, the **Select Applications** dialog box is opened. 1. Select one or more Applications by selecting the appropriate check boxes and click **OK**. By default, all the Active Applications are selected. 2. Click **OK** to continue. ## Setting Time and Date The ECU time and date settings are used to timestamp Events and Logged Data. ![Select an Application Dialog](./assets/set-ecu-dt.png) **Set the Time and Date:** 1. Select: **ECU > Set System Time and Date.** 2. Select an Application and click **OK.** 3. Configure the **Set System Time and Date** dialog box as follows: - **New PC:** Enter a new time and date for the computer that you are using. A Calendar is provided for easy date selection. - **Set PC:** Sets the Time and Date in the computer you are using to the values entered in New PC. - **Current PC:** Read only. Displays the Time and Date as set on the computer you are using. - **Set ECU:** Sets the Time and Date in the ECU to the values in the Current PC boxes. - **ECU:** Read only - Displays the Time and Date set on the ECU. ## ECU On-Line The ECU must be On-Line to: - Display or Log Measurement Parameters as defined by the Live Logging Configuration. - Respond to changes in Editable Parameters during Live Tune. - Operate with a Testbed. The ECU can only be On-Line if the link to System Monitor is OK. The Link status is displayed in the Comms Bar. ![](./assets/comms-bar.png) Select: **ECU > On-line** to toggle the ECU between on-line and off-line modes. This command can be selected even if the Link is not OK. In that case, the ECU will be put on-line as soon as the link is detected. The time during which System Monitor will try to send a command message is set in the Advanced Settings dialog (Timeout before link lost). !!! note System Monitor can be configured to be On-Line whenever a Project is opened by setting **Go online when opening project** to “Y” in Advanced Settings. When the ECU is On-Line: - The Comms Bar shows “Monitor ON”. - The program stamp is read from the ROM of each of the controllers for each Application. If the program stamp does not match the corresponding one in the current Program Version for each controller, the ECU OnLine message box opens with the options: - **Reprogram:** Reprograms the ECU with the current Program Version then switches the ECU On-Line. - **Ignore:** Leaves the Program Version in the ECU unchanged and switches the ECU On-Line. System Monitor continues with a mismatch between the Program Versions. - Closes the message box and the ECU remains Off-line. !!! note If **Warn if Program Versions do not match** is set to N in Advanced Settings, the ECU OnLine message box does not appear and Ignore is assumed. Use this setting to allow the ECU to come on-line without user intervention. When the ECU is Off-Line: - The Comms Bar shows “Monitor OFF”. - Acquisition of data is stopped and the most recent values are frozen in the Measurement windows. - Recording of data to disk is stopped - The Testbed Link is disabled. ## Kill Engine The Kill Engine command stops the engine or prevents it from starting. Select: **Tools > Kill Engine** (Shortcut ALT+K). While Kill Engine is active, System Monitor sends a message to the ECU to stop the engine. The **Sending Engine Kill** message box is displayed and the System Monitor user interface is disabled. However, measurement windows, events, etc. still update. The engine kill message is sent to all the controllers whose application includes the system constant **ENGINE_KILL_CONTROLLER**. We recommend that you only define the constant in one application, although System Monitor does support sending the message to more than one controller if required. If the constant is not defined in any application, the message is sent, by default, to controller AC in a TAG-300 unit. The engine kill message is 0x01a. It is sent continuously at an interval of 200ms. If the message is acknowledged, the indicator in the message box is red; if an acknowledgement is not received for 2s, the engine kill indicator changes to yellow. To close Kill Engine and allow the engine to start, click **Enable**, or press **ENTER** or **ESC**. ## Cursor Control and Trace In a 1-axis or 2-axis Map, the active cell is selected by the values of the Breakpoints. The Cursor Control and Trace functions on the ECU Menu use the active cell to assist you with Tuning. They are only available when the ECU is On-Line. - **Cursor Control:** When checked, the active cell is automatically selected for editing. - **Trace:** When checked, each cell that has been active since Trace was switched on is outlined in grey. ## Single Message The Single ECU Message dialog box allows a single message to be sent to the ECU and displays the response. To open the dialog box select: **Tools > Single Message**. ![Single ECU Message Dialog](./assets/single-message.png) **Comms Path:** Select the Comms Path from the drop-down list. This is only necessary if there is more than one ECU in the vehicle sharing an Ethernet network. **Message Panel:** - **Content:** Enter the message that you want to send as a sequence of word-length hexadecimal values. Only valid hexadecimal values are accepted. - **Words:** The number of words in the message calculated from the message content. Read only. - **Timeout:** The timeout interval in milliseconds. - **Send:** Sends the message. **Repeat Message Panel:** - **Repeat enabled:** When selected, the message is sent repeatedly when Send is clicked, according to the Interval and Number settings. - **Interval:** The time, in milliseconds, between repetitions of the message. Only available when Repeat enabled is selected. - **Number:** The number of times the message is repeated. Only available if Repeat enabled is selected. **Response panel:** - **Content:** A read-only hexadecimal display of the response sent by the ECU. - **Words:** A read-only integer displaying the number of words in the response. - **Status:** A read-only display of the current response status. - **OK:** indicates that a response was received from the ECU. - **Link NOK:** indicates that the ECU is not connected. - **Timeout:** indicates that a Timeout occurred while waiting for a response. - **Display all responses:** When selected, all response messages are displayed. When cleared, only responses intended for System Monitor are displayed. --- # ECU Memory Electronic Control Units (ECUs) used in motorsport contain one or more controllers with access to two types of memory **RAM (Random Access Memory)** - Volatile: requires power to maintain contents. - Fast, byte/word level modification possible. - System Monitor can modify RAM while the ECU is On‑Line and running a test. - The RAM that System Monitor can modify is called Application RAM; it contains the Data Version. RAM can be changed by: - Copying ROM to RAM - Resetting ROM - Downloading data to the ECU - Live Tune **ROM (Read Only Memory)** - Implemented by programmable non‑volatile devices (EEPROM, Flash‑RAM). - Programmable only when the ECU is Off‑Line. - For each controller, ROM is divided into areas: Program, Data and Configurations. - A whole memory area must be reprogrammed at once; individual memory locations cannot be modified. ROM can only be changed by reprogramming the ECU. Most ECUs can operate from either RAM or ROM. To choose, use the Run from ROM or Run from RAM commands. The ECU Bar indicates how each application is running. ## Reprogramming the ECU The ROM in ECU memory contains: - Program — programs for all controllers, built from the project Program Version file. System Monitor validates program versions against the Customer Base. - Data — values of Editable Parameters held in a Data Version file. Systems are tuned by loading different Data Versions. - Configurations — logging, sensor/actuator calibration, external inputs, etc. System Monitor can program various configurations. To reprogram the ECU: 1. Select: `ECU > Reprogram Unit`. - The Select Applications dialog opens. The ROM contents in the ECU are compared with the currently open project; any changed item is downloaded. - The command is available only if the relevant Customer Base has permission. ![Select Applications Dialog](./assets/reprogram-unit.png) 2. Optionally select the **Merged program sequences** check box. - Note: when programming selected applications you may merge programming sequences. When programming sensor sets that cover more than one application, sequences are always merged. 3. Click **OK**. - A progress message window appears. ![Reprogram Progress Window](./assets/reprogram-unit-status.png) !!!warning Clicking **Cancel** stops reprogramming but does not reset items already reprogrammed. This may leave the ECU in an inconsistent state and could prevent correct operation. If the reprogram sequence does not complete, proper function cannot be guaranteed. The `ECU > Reprogram All` option reprograms all code and data areas for all applications and downloads the logging configuration — a one‑stop reprogram facility. !!! note - Controllers are placed in Boot mode (Zone 0) for programming; successful reprogramming exits Boot mode automatically. - Logging configurations and other non‑vehicle‑critical data can be programmed in Zone 1 while some ECU functions continue. - Controller status is indicated on the Comms Bar. - `Force Complete Reprogram` downloads all program code, data and configurations regardless of change (useful for embedded developers). - Transfer information to a different unit using Spare Unit procedures. - Update the Data Version only. `Tools > Force Complete Reprogram` downloads all program code, data and configurations whether they changed or not. Intended as a diagnostic tool for embedded code developers; for normal updates use the Reprogramming the ECU procedure. ## Run from ROM / Run from RAM Most ECUs default to running from RAM when connected to System Monitor. - Select `ECU > Run from ROM` to switch active applications to ROM. The command acts only on Active Applications. - Select `ECU > Run from RAM` to switch active applications to RAM. The command acts only on Active Applications. The ECU Bar shows which memory type (RAM or ROM) each application is using. ## Copy ROM to RAM `ECU > Copy ROM to RAM` sends a signal to copy the Data Version from ROM to RAM. The command acts only on Active Applications. The ECU Bar shows software versions in each application. ## Min/Max Statistics - Measurement Parameters are typically sensor outputs or ECU calculations. - The ECU may hold minimum and maximum values of parameters which can be read as Measurement Parameters. - `ECU > Delete Min/Max` sends a signal to set Minimum and Maximum Measurement Parameters to zero. This command is available only if the relevant Customer Base has permission - The command applies to Active Applications only. ## Read RAM The Read RAM window displays RAM contents for a particular controller in the attached ECU. Open it with `Tools > Read RAM`. Multiple Read RAM windows may be opened on the same Desktop. ![Read RAM Window](./assets/read-ram.png) The Read RAM Properties dialog (Window System Menu icon or Alt + Enter) controls content and layout. Tabs: - Read RAM — source and format of the data (specific to Read RAM windows). - Display — appearance controls (common to all windows). **Status bar** Shows information about the control unit and its connection: - Status: - DataOK — ECU transmitting requested data - TimeOut — last request did not get a response - NoData — no data being received - Controller — controller name as set in Read RAM Properties - Logging — indicates if data is being logged to a file (set in Read RAM Properties) **Display memory panel** - Comms Path — select from drop‑down (needed only if more than one ECU shares an Ethernet network). - Controller — list of controllers in the connected ECU; select the controller to display RAM contents. - The Applications dialog Memory Layout tab shows which controllers and locations an Application uses. - The ECU Properties Controllers tab allows viewing and enabling/disabling controllers. - Start address — hex address where display starts; must be a word boundary (even). - Addressing Size — start address of the last block of data to display. - Format — choose display format: - Hexadecimal — 4 characters (default) - Decimal — 5 characters - Binary — 16 characters - Floating Point — 14 characters Memory addresses are always shown in hexadecimal. - Swap Bytes — when selected, each word’s byte order is reversed. - Data Only — hide the addresses column when selected. **Logging panel** - Log to file — when selected, RAM contents are logged to a file updated at the same rate as the display. The display update rate is set in the Display tab. An indicator shows when logging is active. Default: unchecked. - Filename — enter or browse for the file name (default: `test1.rrl`). No default path; the most recently opened path is used. --- # Programming Data The changes made during Tuning are stored as a Data Version in the Edit Buffer. Only the Active Applications can be Tuned. ## Live Tune modes The Data Version may be transferred to the ECU in one of two ways: - **Live Tune On** As soon as an Editable Parameter is changed, the new value is loaded into RAM. - **Live Tune Off** !!! note The data changes only affect the ECU RAM. To apply the changes to the vehicle systems, the ECU must be operating from RAM. ## Downloading Data to the ECU The difference between the data in the Edit Buffer and the RAM are detected and loaded into RAM to make the contents of the RAM match the Edit Buffer when the **Download Data Changes** command is selected. The command only acts on the Active Applications. ### Download Data Changes Select: **ECU > Download Data Changes.** ![Download Data Changes Dialog](./assets/download-data-changes.png) If either the Program Version or the Data Version in the Edit Buffer do not match those in the ECU, a message is displayed with the following options: - **Sync SM to Unit** Synchronises the files in System Monitor to match those in the Unit. The specified files are opened in System Monitor, replacing those that are already open. If a file cannot be found, a new file is created using the filename and modification information taken from the unit. Uploads data and places it in the Edit Buffer. Files and data which are the same as already in System Monitor are not uploaded. - **Reprogram** Reprograms the ECU then downloads the data changes in the Edit Buffer to the ECU. - **Force Complete Reprogram** Forces a Complete Reprogram then downloads the data changes in the Edit Buffer to the ECU. - **Ignore** Closes the message box. No changes are made. The download process continues with the next Application. - **Cancel** Closes the message box. No changes are made. ## Uploading Data from the ECU Select: **ECU > Upload Data** to load the Data Version in the ECU into the System Monitor Edit Buffer. You are prompted to save the contents of the Edit Buffer as a new Data Version File. The command only acts on the Active Applications. ## Undo Data Changes Data changes can be undone using the following ECU menu options: - **Undo Data Changes in the Unit** Undo data changes that have been downloaded to the ECU but do not undo the changes in the Edit Buffer. - **Undo Data Changes in the Unit and Edit Buffer** Undo data changes downloaded to the ECU and undo changes in the Edit Buffer. - **Undo Data Changes in the Edit Buffer** Undo data changes in the Edit Buffer. Changes downloaded to the ECU will not be undone. !!! note The Undo Data Changes commands can only undo changes made since the Data Version was last saved. ## Reset ROM The **Reset Data to ROM** command in the ECU menu does the following: 1. Overwrites the contents of the Edit Buffer with the Data Version that was used to open the Project. This is equivalent to restoring all values. 2. Reprograms the ECU with the contents of the Edit Buffer. 3. Sends a signal to the ECU to copy the Data Version from the ROM to the RAM. The command only acts on the Active Applications. --- # Diagnostics Status This section covers tools and views that help users inspect ECU status, synchronize files, and verify program and data versions. ## Unit Status The Unit Status dialog box indicates which files have been loaded into the Edit Buffer and the unit, and is used to synchronize the two. The dialog box is resizable and the column widths are editable. To open the dialog box, select: **Tools > Unit Status**. To select more than one cell, use the SHIFT and CTRL keys. ![Unit Status Dialog](./assets/ecu-status.png) ### Columns | Column | Description | |----------|-----------------------------------------------------------------------------| | **Project** | Information about the files currently loaded into System Monitor. | | **ECU** | Information about the files currently loaded into the Main unit. | | **Spare** | Information about the files currently loaded into the Spare unit. This column is only shown if the Spare check box is selected. | ### Rows | Row | Description | |--------------------|-----------------------------------------------------------------------------------------------------------------------| | **Data Versions** | For each Application. | | **Sensor Sets** | Allows you to apply calibration data to sensors, enable and disable individual sensors, and enter values for sensor constants. | | **Configurations** | The Remote logging, Burst logging, and External Inputs configurations are stored in and used by an ECU when it is working independently of System Monitor. | | **History** | During its operation, the ECU generates various statistics about the operation of the vehicle that it is controlling. | ### Cell Colours The background colours of the cells indicate the following: - **White** - Unit contains the same information as in System Monitor. - **Red** - Unit contains a different program or data version from the one in System Monitor. - **Orange** - Unit contains the same program and data version but different Live Tunes. - **Blue** - Unit contains a different configuration (sensor set, remote logging, burst logging, external inputs, history report). - **Yellow** - Selected cell. A summary of the file history is displayed at the bottom of the dialog box for the selected cell. ### Other Symbols The following symbols may also appear: - `(*)` - The data has been modified and not yet saved. - `?` - System Monitor is unable to obtain the information from the ECU. - `(L)` - DTV has been locked with standard RDA. - `(L2)` - DTV has been locked with enhanced RDA2. ### Controls With a cell or cells selected, the following commands are available: - **Refresh** - Updates the display. - **<>** - Download the information for cells selected in the Project column to the main unit. Only downloads data that is not the same as already in System Monitor. For Data Versions, runs Force Complete Reprogram before downloading data. - **Sync Unit>>** - Synchronize the information in the main unit with the data in the Edit Buffer. This command only applies to cells selected in the Project column. If the Program or Data versions are different, runs Reprogram Unit then Download Data Changes. Configuration files that are the same as already in System Monitor are not downloaded. - **All Sync Unit>>** - Synchronize all files in the main unit with the data in the Edit Buffer. This command is the same as selecting all cells in the Project column and clicking Sync Unit. - **Spare>>** - As Download>> but sends information to the Spare unit. Only available when the Spare column is displayed. - **Sync Spare>>** - As Sync Unit>> but sends information to the Spare unit. Only available when the Spare column is displayed. - **All Sync Spare>>** - As All Sync Unit>> but sends information to the Spare unit. - **Spare** - When this check box is selected, the Spare column is displayed. Only available if a Spare Unit has been configured. ## ECU Properties - Controllers The Controllers tab of the ECU Properties dialog box allows controllers within the ECU to be viewed and enabled/disabled. ![ECU Properties - Controllers Tab](./assets/ecu-properties.png) All the Applications are shown in a list. Click the expand icon to see the controllers connected to an Application. The check box beside the controller indicates its status. When selected, the controller is connected to the application and responds to reprogramming, tuning, and monitoring. When cleared, the controller is disconnected from the application and does not respond. Controllers are connected or disconnected by selecting or clearing the check box for the Application. If an Application is connected to some, but not all, of the controllers, its check box is greyed out. ## ECU Properties – Devices The Devices tab of the ECU Properties dialog box displays information about the ECU hardware. ![ECU Properties - Devices Tab](./assets/ecu-devices.png) Each unit is shown in the following format: ``` aaaaa (Detected Name: bbbbb) ``` Where `aaaaa` is the type of ECU as specified in the Program Version, and `bbbbb` is the type of ECU as detected by interrogating the unit (should be the same as `aaaaa`). Click the expand icon to see the following: - **Serial Number** - The serial number of the ECU as detected by interrogating the unit. - **Address** - The IP Address of the ECU as defined in Unit Definition. ## ECU Search The ECU Bar displays Data Version information for all Applications. ECU Search provides more detail about the Program and Data Version Files loaded into the ECU. To initiate an ECU search, select: **ECU > Search for Versions**. Following Application selection in the **Select an Application** dialog box, the ECU Search dialog box opens. The ECU Search dialog box consists of the following tabs: ### Program Version on ECU The Program Version tab of the ECU Search dialog box displays the Filename, Saved By, Saved On, Comment, and Notes information for the Program version on the ECU. The files used to create the Program version are shown in the Properties list. The file name of the original Data version, generated when the Program Version was created, is shown as **Base Data**. ![Program Version on ECU Tab](./assets/ecu-program-version.png) All the information on this tab is read-only. ### Data Version in Flash-RAM This tab shows information about the Data Version being used by the ECU. ![Data Version in Flash-RAM Tab](./assets/ecu-data-version.png) When an ECU is connected, System Monitor locates the files that were used to program it. The ECU does not hold the file name but stores an ID which System Monitor can use to identify the file. System Monitor searches for files when: - A new project is loaded. - The ECU memory is changed. - The ECU is reconnected after the link has been lost. A search can also be forced by the **Query Versions** command on the ECU Menu. The Program Version ID is taken from the Reference Controller. If that controller is faulty or not connected, System Monitor cannot locate the Program Version and `???` is shown in the ECU Bar. Ensure that the Reference Controller is enabled in **ECU Properties - Controllers**. Data Version file locations are searched in the following order: 1. The directory that contains the data version currently loaded into the project. 2. The directory and sub-directories that were created to contain the same program version as detected in the ECU. 3. The entire Customer Base directory and sub-directories. Once the Data Version file is located, its contents are compared with the settings of all the controllers in turn. If the data in any controller is different from the version in the file, `(*)` is shown in the ECU Bar. The status of the search is indicated by the background colour of the ECU Bar. The search can take some time. To speed up the response of the ECU Bar, System Monitor can be prevented from searching the entire Customer Base directory. To do this, set **Search base for versions** to `N` in **Advanced Settings**. --- # Hardware ## Adjustment board The Adjustment Board is a box with four high‑precision rotary switches. System Monitor can map each switch to a Measurement Parameter (used as an Index Parameter in a 1‑axis or 2‑axis map) or to a scalar editable parameter. This lets operators adjust control values by feel while watching System Monitor and other instruments. All controls operate the same; the knobs differ in size to aid identification. Turning clockwise increases the assigned parameter; turning counter‑clockwise decreases it. ### Configuration To configure the Adjustment Board: 1. Open the **Pot Board Configuration** dialog: - Navigate to `Tools > Pot Board > Configure` - Shortcut: `CTRL + SHIFT + F1` 2. Assign parameters to controls: - Drag parameters from the **Parameter Explorer** to the corresponding row in the configuration window. - Each row corresponds to a labelled control on the board. 3. Remove assignments: - Right-click a parameter in the configuration window and select **Remove**. 4. Assign a control to the **Maths Bar**: - Right-click a row and select **Maths Bar**. - Turning the assigned control adjusts the selected editable parameter. ### Adjustment Groups Parameters are assigned in groups of four, matching the four physical controls on the board. These groups are defined in the `Pot.ini` file used during program version creation. - The active group is selected in the **Adjustment Board Configuration** dialog. - Group names and assignments can be edited in the **Adjustment Board Groups** dialog. ### Supported Parameter Types - **Scalar Editable Parameters**: Turning a knob is equivalent to changing the parameter value manually. - Each parameter includes an **Adjustment Properties** tab to define: - **Step size** - **Zero value** ### Keyboard Operation The keyboard can replicate Adjustment Board functionality: - **Increase value**: - Hold `CTRL` and press a number key (1–4) - **Decrease value**: - Hold `CTRL + SHIFT` and press a number key (1–4) > Use the number keys on the main keyboard, not the number pad. Step size is defined in the **Adjustment Sheet** of the Parameter Properties. ### Pot Board Properties To configure the connection between System Monitor and the Adjustment Board: 1. Open the **Window Properties** dialog: - Shortcut: `ALT + Enter` 2. Navigate to the **Pot Board** tab. 3. Select connection type: - **CA Card** - **Serial Port** (COM1 or COM2) > The serial port is configured automatically to suit the board, regardless of Windows settings. ### Adjustment Board Menu Commands Located under `Tools > Pot Board`: - **Configure**: Opens the configuration dialog (`CTRL + SHIFT + F1`) - **Enable**: Activates the controls (`CTRL + SHIFT + F2`) - **Reset Adjustments**: Function depends on parameter type (`CTRL + SHIFT + F3`) - **Recent Files**: Lists recently opened configuration files ### Available types #### VE‑80 - Passive device using dedicated connections on the CA‑card. - Connection details are in the CA‑card product specification. - No separate power supply required. - In the Adjustment Board Configuration dialog, select **CA Card** on the Link panel. #### VE‑81 - Active device connected via an RS‑232 serial port. - Connect the VE‑81 to an RS‑232 port on the PC using the supplied cable. - Connect the provided power supply to the VE‑81 and a suitable AC outlet. The rear LED flashes once per second when the unit is powered and transmitting. - In the Adjustment Board Configuration dialog, select **Serial Port** on the Link panel and choose the COM port (e.g., COM1 or COM2). !!! note The Adjustment Board is often called a "Pot" or "Potentiometer Board." This is not strictly accurate: a potentiometer gives an absolute output for a given position, whereas the rotary switches on the Adjustment Board produce a pulse train and direction. Switch position has no absolute value. ## CAN link CAN (Controller Area Network) is a serial data protocol originally developed for automotive use and now common in industrial applications. The protocol is defined in ISO 11898. Key characteristics: - Access is controlled by non‑destructive, bit‑wise arbitration. - Messages are compact (up to eight data bytes) and include a checksum. - Messages carry numeric identifiers rather than explicit addresses; the identifier determines priority and can indicate message content. - Robust error handling retransmits messages if necessary. - Faults are isolated and faulty nodes are removed from the bus. System Monitor can be used to configure an ECU or data logger to interpret sensor data on a CAN link. --- # Developer Functions System Monitor includes various features available for experienced operators. These come under the following categories: - **Diagnostics** — Provides information for embedded software developers and can be useful for troubleshooting the operation of the ECU and of System Monitor itself. - **System Log** — Files of errors and events that occur during the operation of System Monitor. - **Productivity Tools** — Allows repetitive processes to be automated and provides links with other applications. - **Settings** — Changes various System Monitor configuration settings. - **FIA Upload** — A utility for verifying that the code in the ECU meets FIA requirements. ## Diagnostics Diagnostics provides information for embedded software developers and can be useful for troubleshooting the operation of the ECU and of System Monitor itself. The Diagnostics tools are accessed from the Tools menu. The following tools are available: - **Read RAM** — Opens a window that displays the content of the memory for a particular controller in the attached ECU. (Shortcut: `ALT+R`) - **Single Message** — Allows messages to be sent to the ECU and to view the response. (Shortcut: `ALT+M`) - **Force Complete Reprogram** — Downloads all the data and code for the ECU whether it has changed or not. (Shortcut: `ALT+C`) Only available when an ECU is connected. ## System Log System Monitor creates files of errors and events that are caused by its own operation. (These are not the same as the Errors and Events which are due to the operation of the ECU.) The contents of the files typically correspond to the on-screen messages but with much more detail. The following files are created: - `Pgv_xxx.txt` — log of messages generated when Creating Program Versions. - `Cmd_xxx.txt` — log of commands issued from Menu commands or ActiveX. - `Warn_xxx.txt` — log of warnings, in particular Parameters dropped from configurations, etc. - `Unit_xxx.txt` — log of communications with the unit. Where `xxx` is the username of the person logged onto the computer running System Monitor. The directory for log files is defined in Advanced Settings. If no location is defined, the files are not created. ### Message Logs A further level of detail is available in a Message Log which is used for debugging. To create a Message Log: 1. Enable logging by selecting a log level in the **Message log level (for debugging)** Advanced Setting. This is normally set to Not Logging. 2. Write the log to the directory by clicking the Log IP Transaction button on the Standard Tool Bar. The following file is created: - `SM_Msgs_xxx.txt` All log files are in ASCII format and can be read by any text editor. ### Error Messages During Operation The messages linked to this topic should not occur during normal operation. They indicate that there may be a problem with your System Monitor installation or with your ASAP or Hex files. ## Productivity Tools Productivity Tools allow repetitive processes to be automated and also provide links with other applications. The following tools are available: - **Custom Menus** — Run applications other than System Monitor from menu commands in System Monitor. - **Macros** — Record and play back sequences of keystrokes. ### Custom Menus System Monitor includes a facility to run external applications. Custom Menus creates additional entries in the Tools menu, which run executable files. The Custom Menu facility can be used for scripts and the McLaren Applied support team may use this facility to provide you with special functions in System Monitor. Custom Menus are stored in the Windows Registry as part of the System Monitor application and are available to all Projects. To open the Custom Menus dialog box, select: Tools > Customize. Create a new custom menu entry: 1. Click `image\Custom_Menu_new_icon.gif`. 2. Enter the text that you want to appear in the Tools menu and press ENTER to accept it. 3. In the Command text box enter or browse for the name of the executable file that you want to be run by the menu entry. 4. Enter Arguments or Working Directory, if required. 5. Click Close to accept the menu entries. Existing entries can be edited if required. When an entry is selected in the Menu Contents list, its Command, Arguments and Working Directory text boxes can be edited. To rename an entry, select it in the Menu Contents list, then click on it. - `image\Custom_Menu_delete_icon.gif` — Deletes the selected entry. The menu entries appear in the Tools menu in the same order as they appear in the Menu Contents list. - `image\Custom_Menu_up_icon.gif` — Moves the selected entry one place up the list. - `image\Custom_Menu_down_icon.gif` — Moves the selected entry one place down the list. ### Macros System Monitor includes a facility to record keystrokes and to replay them as Macros. Macros are stored in the Windows Registry as part of the System Monitor application and are available to all Projects. !!! note Mouse movements are not recorded. To open the Macros dialog box, select: Tools > Macros. Record a new macro: 1. Click Record to open the Record New Macro dialog box. 2. Enter a Description. 3. Enter a Shortcut key sequence. The shortcut consists of one or more control keys (`CTRL`, `ALT` or `SHIFT`) and a function key or any letter or number. 4. Select the number of times you want the Macro to repeat on playback. The default is `0`. Check Repeat Forever if you want the Macro to continue to repeat until you press `CTRL+ESC`. 5. Select the Playback Speed that you want the Macro to run at. The default is `x1`. 6. Click OK. The dialog boxes close and all keystrokes are recorded until you press `F2`. The Description, Shortcut, Repetition and Playback Speed of the selected macro can be edited in the Macro dialog box. To delete a macro, select it from the Macros list and click Delete. To play or stop a macro: - To play a macro, select it from the Macros list and click Play. Alternatively, you can play a macro by pressing the shortcut key sequence; the Macro dialog box must be closed. - To stop a macro that is playing, press `CTRL+ESC`. ## FIA Upload System Monitor includes a utility for verifying that the code in the ECU meets FIA requirements. The utility is a separate program that runs outside System Monitor. It is installed with System Monitor in the same location as the System Monitor files. For more information on using the FIA Upload utility, please contact Technical Support and ask for the Working Practice Document “Uploading Code and Data for FIA Software Checks” TWPD0066. --- # Overview During its operation the ECU generates various statistics about the operation of the vehicle that it is controlling. History Report extracts these statistics and displays them. System Monitor can also control aspects of the way the statistics are collected and stored. History Report is only available if the relevant Customer Base has permission, as set in Customer Base Advanced Settings. Before History Reports can be used, a History Report file must be associated with the Project. Select **Tools > History Report** to access the submenu options. **Menu Options** | Option | Description | |-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Open filename** | Opens the History Report dialog box for the History Report File currently associated with the project. If there is no History Report file, this command is not available. To open a different History Report file. | | **Upload** | Extracts history data from the ECU and opens the History Report dialog box showing that data. The contents of the History Report file associated with the project are overwritten. If there is no History Report file, a new one is created. | | **Download** | Sends the history data in the currently opened History Report to the ECU. The previous set of history data in the ECU is deleted. Use this facility if you want to continue gathering data for a vehicle in which you have changed the ECU. This command also sends Setup data to the ECU.
**Note:** Only available when a History Report dialog box is open. | | **Reset Engine Stats** | Sets the value of all the history data for engines in the ECU to Zero. The previous set of history data for engines is deleted.
**Note:** Only available when a History Report dialog box is open. | | **Reset Chassis Stats** | Sets the value of all the history data for chassis in the ECU to Zero. The previous set of history data for chassis is deleted.
**Note:** Only available when a History Report dialog box is open. | ## Selecting a History Report File An existing History Report File can be selected from the File Type Selection dialog box. When selected the History Report dialog box opens, showing data from the file. --- # Details ## Files The History Report files are stored in the `\HistReps\` sub-directory of the Multi-Application Base. The default name for the History Report is `XXXXXnnn.hrs`, where: - **XXXXX**: The first five characters of the first Application Name in the project. - **nnn**: A three-digit hex number, starting with 001, which is incremented whenever a new History file is saved. The History Report File name is displayed in the Titlebar. When **Tools > History Report > Upload** is selected, the new History Report is given a file name with the next sequential number for the Active Application. History Report Files can be saved. If you make any changes to the history data, including uploading from the ECU, you will be prompted to save the History Report file when you close the History Report Dialog Box. Use **Save As** on the File Menu if you want to save a History report file under a different name. The History Report dialog box is opened when: - An existing History Report File is opened using the **File > Open** command. - **Tools > History Report > Upload** is selected. - A new History Report File is created using the **File > New** command. ![History Report Dialog](./assets/new-report.png) The dialog box has two tabs: - **Summary**: Shows totals of statistics for all the Chassis and Engines for which history data has been obtained and gives access to detailed reports. - **Setup**: Allows modification of how history data is collected. ## Summary The Summary tab shows totals of statistics for all the Chassis and Engines for which history data has been obtained. The ASAP file defines the number of variants of Chassis and Engine for which history data is analyzed. Each Chassis variant has a row in the Chassis panel, and each Engine variant has a row in the Engine panel. The check box in the Active column indicates which data set is currently being collected by the ECU. ![History Report Summary Dialog](./assets/summary.png) ### Chassis Panel The No, Chassis, Driver, and Notes columns contain text boxes to annotate the history data. ### Engine Panel In the Engine panel, the text boxes are in the Engine and Notes columns. Click on a text box to open it for editing. The Dist, Time, and Cycles (Engine panel only) columns show the total distance, time, and number of cycles run by the Engine or Chassis since the History Report was last reset. To see the Chassis Details, select a row in the Chassis panel and press ENTER or double-click in the Active column. To see the Engine Details, select a row in the Engine panel and press ENTER or double-click in the Active column. To send changes to the ECU, select: **Tools Menu > History Report > Download**. ## Chassis Details To open the Chassis Details dialog box for the selected chassis data set, click on a row in the Chassis panel in the Summary Sheet of the History Report. The dialog box displays read-only statistics for the selected Chassis. ![Chassis Details Dialog](./assets/chassis-details.png) - **Chassis**: Text information from the row for that Chassis in the History Report - Summary. - **From**: The date that the Chassis data was last reset. - **To**: The date that the history data was uploaded. - **Total Distance**: The distance accumulated by the chassis between the From and To dates. - **Total Time**: The time accumulated by the chassis while history data was being recorded. - **Time in Gear**: A breakdown of the time the chassis was in each gear. ## Engine Details To open the Engine Details dialog box for the selected engine data set, click on a row in the Engine panel in the Summary Sheet of the History Report. The dialog box has the following tabs: - **Time**: Shows a breakdown of the amount of time that the selected engine was in selected combinations of engine speed and throttle opening. - **Cycles**: Shows a breakdown of the number of cycles during which the selected engine was in selected combinations of engine speed and throttle opening. - **Auxiliary**: Shows diagnostic data for the selected engine. ![Engine Details Dialog](./assets/engine-details.png) Some aspects of the data shown in the Engine Details dialog box can be modified in the Setup tab of the History Report Dialog Box. Click **Print** to print a text version of the information on the selected tab. ### Time The Time tab shows a breakdown of the amount of time that the selected engine was in selected combinations of engine speed and throttle opening. The dialog box displays read-only statistics for the selected Engine. - **Engine**: Shows the text information from the row for that Engine in the History Report - Summary. - **From**: The date that the Engine data was last reset. - **To**: The date that the history data was uploaded. - **Total Cycles**: The number of cycles accumulated by the engine while history data was being recorded. - **Total Time**: The time accumulated by the chassis while history data was being recorded. **Features**: - The upper pane shows a breakdown of the time spent by the selected engine in each combination of engine speed and throttle opening. - The lower pane shows the same data as a percentage of the total time. - The Breakpoint values can be modified in the Setup tab of the History Report dialog box. ### Cycles The Cycles tab shows a breakdown of the number of cycles during which the selected engine was in selected combinations of engine speed and throttle opening. The dialog box displays read-only statistics for the selected Engine. - **Engine**: Shows the text information from the row for that Engine in the History Report - Summary. - **From**: The date that the Engine data was last reset. - **To**: The date that the history data was uploaded. - **Total Cycles**: The number of cycles accumulated by the engine while history data was being recorded. - **Total Time**: The time accumulated by the chassis while history data was being recorded. **Features**: - The upper pane shows a breakdown of the number of cycles during which the selected engine was in each combination of engine speed and throttle opening. - The lower pane shows the same data as a percentage of the total time. - The Breakpoint values can be modified in the Setup tab of the History Report dialog box. ### Auxiliary The Auxiliary tab shows diagnostic data for the selected engine. The dialog box displays read-only statistics for the selected Engine. - **Engine**: Shows the text information from the row for that Engine in the History Report - Summary. - **From**: The date that the Engine data was last reset. - **To**: The date that the history data was uploaded. - **Total Cycles**: The number of cycles accumulated by the engine while history data was being recorded. - **Total Time**: The time accumulated by the chassis while history data was being recorded. The list shows a breakdown of the Battery Voltage and Alternator Current against engine speed. Note: The Breakpoint and Condition values can be modified in the Setup tab of the History Report dialog box. ### Conditions Panel The Conditions panel shows the Total Time and the Longest individual period that the engine was in the following states: - Oil temperature (T_Oil) is lower than condition while Oil pressure (P_Oil) is greater than condition. - Oil pressure (P_Oil) is lower than condition while Engine Speed (Revs) is greater than condition. - Water Temperature (T_Water) is greater than condition. ### Statistics - **Max Revs seen**: The highest engine speed reached while the history data was being recorded. - **Soft Rev-limit**: The number of occurrences of Soft rev limits (i.e., engine speed high enough for the ECU to retard the ignition). Also shown as a percentage of the Total Cycles. - **Hard Rev-limit**: The number of occurrences of Hard rev limits (i.e., engine speed high enough for the ECU to cut the ignition). Also shown as a percentage of the Total Cycles. - **Num Rev-limits**: The number of times that the number of consecutive Rev-limit incidents exceeded condition. The maximum number of consecutive Rev-limit incidents is also shown. ## Setup The Setup tab allows you to modify how history data is collected. ![Setup Tab](./assets/setup.png) ### Rev Breakpoints Edit the breakpoints to modify the ranges of engine speed used to display statistics in the Chassis Details and Engine Details dialog boxes. The default for Number of Breakpoints is 16, which can be edited to a lower value (must be a positive integer). The breakpoints between the new number and 16 are labeled "Unset". All breakpoint values, except the first, can be edited. ### Throttle Breakpoints Edit the breakpoints to modify the ranges of throttle opening used to display statistics in the Engine Details dialog box. The Number of Breakpoints is fixed at 5. Each breakpoint value, except the first and the last, can be edited. ### Conditions Edit the values used for conditions in the Auxiliary Engine Details dialog box: - **Low Oil Temp/High Oil Pres Condition**: Oil temperature is lower than T_Oil ºC and Oil pressure is greater than P_Oil bar. - **Low Oil Pressure/Normal Running Condition**: Oil pressure is lower than P_Oil bar and Engine Speed is greater than Revs rpm. - **High Water Temperature condition**: Water Temperature is greater than T_Water ºC. - **No. of long rev-limits condition**: The number of times that the number of consecutive Rev-limit incidents exceeds FiresInLimit. To send changes made on this dialog box to the ECU, select: **Tools > History Report > Download**. --- # Overview Organise and retain vast telemetry datasets in a structured, scalable format. ## SQL Race Management Studio (SRMS) SRMS streamlines the storage and access of racing telemetry. Designed for teams handling large data volumes, it enables efficient session organisation, import automation, and secure database‑backed retention. ### Key Functionality - Session management tools - Centralised data storage with user controls - Bulk import - Auto‑import for routine ingestion --- # Overview SQL Race is the session data access layer used by ATLAS to manage telemetry data. It provides a structured way to store, retrieve and manipulate session data, supporting both real-time and historical analysis. SQL Race integrates tightly with the tools in the ATLAS product suite, enabling efficient data handling, and customisation via the [SQL Race API](../../../developer-resources/atlas/sql-race/index.md). SQL Race Database is a custom SQL Server implementation specific for data store. It supports MS SQL Replication where replication of data is required over multiple sites. ## Core Capabilities 1. Session Management: Handles creation, loading and association of sessions, including composite sessions for comparitive analysis. 2. Data Access Layer: Provides APIs for reading, and writing data. Including parameters, events and session summaries. 3. Filestream integration: Uses SQL Server with filestreams for efficient storage of large binary data such as channel and event logs.
- __Setup with SQL Race__ --- Learn how to install SQL Server and SQL Race Database. [→ Getting started](installation.md) - __Start using the SQL Race API__ --- Automate access to sessions, parameters and events. [→ API Guide](../../../developer-resources/atlas/sql-race/index.md)
--- # Getting Started This guide will cover the process of installing SQL Server, setting up a SQL Race Database, and connecting your database to ATLAS ready to start reading and writing data. ## Installing SQLServer Download and install SQL Server 2017 or 2019 from the [Microsoft SQL Server Downloads](https://www.microsoft.com/en-us/sql-server/sql-server-downloads) page. !!! note Express will work too but it has limited size so it's not recommended to use for larger datasets. ![Custom Installation](assets/developer_edition.png){: style="max-width:75%;"} Once the installer has been downloaded, run it and select the 'Custom' installation type and follow the prompts to complete the installation after which the installation center will open. ![installation center](assets/setup_wizard.png){: style="max-width:75%;"} Go to **Installation** and select **New SQL Server stand-alone installation** at the top. Click through the wizard, if you need SQL Server Replication, make sure the checkbox is selected on the **Feature Selection** page. ![Feature Selection](assets/feature_selection.png){: style="max-width:75%;"} Give the instance a name on the **Instance Configuration** page. It's recommended to use a named instance (like `LOCAL`) instead of leaving it blank, which does not work well with SQL Race Functions. ![Custom Installation](assets/database_config.png){: style="max-width:75%;"} In **Database Engine Configuration** ensure Windows authentication mode is selected and add the current user to the SQL Server administrators. Go the the **FILESTREAM** tab and enable all three options, and add your instance name to the Windows share name (e.g. `LOCAL`). ![Filestrean](assets/filestream.png){: style="max-width:75%;"} ## Installing SQL Server Management Studio (SSMS) You can download SQL Server Management Studio (SSMS) from the [Download SQL Server Management Studio (SSMS)](https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms) page. Or from the INstalaller center after installing SQL Server by going to **Tools > Install SQL Server Management Tools**. ## Configuring Firewall If you want to access the SQL Server instance from another machine, you need to configure the firewall to allow incoming connections via certain ports. Open PowerShell as an administrator and run the following commands: ```powershell New-NetFirewallRule -DisplayName "SQL Server" -Direction "Inbound" -Protocol "TCP" -LocalPort "1433" -Action "Allow" New-NetFirewallRule -DisplayName "SQL Server" -Direction "Inbound" -Protocol "UDP" -LocalPort "1434" -Action "Allow" New-NetFirewallRule -DisplayName "SQL Server" -Direction "Inbound" -Program "%ProgramFiles%\Microsoft SQL Server\MSSQL12.LOCAL\MSSQL\Binn\sqlservr.exe" -Action "Allow" ``` ## Enabling TCP/IP Open SQL Server Configuration Manager and navigate to **SQL Server Network Configuration > Protocols for [YourInstanceName]**. Double-click on **TCP/IP** and select **Enable**, **Apply**, and **OK**. ![TCP/IP](assets/tcp.png){: style="max-width:75%;"} You need to restart the SQL Server service for the changes to take effect. You can do this in SQL Server Configuration Manager by going to **SQL Server Services**, right-clicking on your SQL Server instance, and selecting **Restart**. ![Restart](assets/restart.png){: style="max-width:75%;"} ## Installing SQL Race Database You can find the latest version of SQL Race Database on the [Portal](https://portal.mclarenapplied.com/portal/Downloads/Software#SQL%20Race%20Database). Download and run the installer, accepting the license agreement. The installer should automatically detect your SQL Server instance. If it doesn't, you can manually enter the instance name you specified during installation (e.g. `SERVER_NAME\INSTANCE_NAME`). ![SQL Race](assets/sqlrace.png){: style="max-width:75%;"} Leave all default options and click **Install**. ### Setting Filestream SQL Race doesn't use native filestreams, so you must create your own directory. Create a new folder on your C: drive called `SQLRace_FileStream`. When connecting to SQL Race Database from a remote machine, you'll need to access the folder you just created. Right click on the folder, go to **Properties > Sharing > Advanced Sharing**, check **Share this folder**, and click **Permissions**. Ensure that the user you are connecting with has at least **Read/Write** permissions. ![FileStream Permissions](assets/permissions.png){: style="max-width:50%;"} The folder will be available on the network at `\\YOUR_COMPUTER_NAME\SQLRace_FileStream`. ![Manual Filestream](assets/manual_filestream.png){: style="max-width:50%;"} !!! note If the user accessing the filestream is in a different AD group then follow the steps [here](#cross-domain-access) to enable cross domain access. ## Connecting to SQL Race Database from ATLAS In ATLAS, go to **Tools > Database Connection Manager** and Click **Add**. ![ATLAS](assets/atlas.png){: style="max-width:75%;"} !!! note It's preferrable to match the case of the host machine. So if the machine name is all caps, use all caps in the connection string. Click **Test Connection** to make sure the filestream is writeable. !!! failure If your filestream is readonly, you will see the message "You are able to access only data to read." In this case, make sure you have read/write access to the folder. !!! success If the connection is successful, you should receive the message "Established connection to database successfully. FIle stream folder is writeable." ### Cross Domain Access It is also possible to access SQL Race from a client machine in a different AD group. #### Windows Credentials From **Credentials manager > Windows credentials > Add a Windows Credential** ![ATLAS](assets/credential_manager.png){: style="max-width:100%;"} When asked for an Internet or network address, you need to provide the IP or fully-qualified domain name (FQDM), followed by port 1433. Then give a valid username and password for the REMOTE server, ensure the user have permission to view the databases and access the filestreams. #### SQL Server Settings In SSMS, right click the **Server instance > Properties > Connections > Allow remote connections to this server** ![ATLAS](assets/ms_ssms.png){: style="max-width:100%;"} In **Sql Server configuration manager > SQL Native Client 11.0 Configuration**. Ensure TCP/IP is enable and move it to order 2. ![ATLAS](assets/sscm_tcp.png){: style="max-width:100%;"} In **Sql Server configuration manager > SQL Server Network Configuration > Protocols for {instance name}** (for a default instance, the instance name would be MSSQLSERVER). Ensure TCP/IP is enabled. ![ATLAS](assets/sscm_tcp2.png){: style="max-width:100%;"} Right click **TCP/IP > Properties**. Make all TCP Dynamic Ports blank and TPC Port to be 1433 ![ATLAS](assets/tcp_left.png){: style="max-width:60%;"} ![ATLAS](assets/tcp_right.png){: style="max-width:60%;"} Apply all setting and restart the service. On the server, run command ```netstat -a -p``` tcp and make sure port 1433 is listening. #### Test Remote Connection with SSMS On your local machine, launch SSMS and use the following connection string: **{IP/ FQDM},1433**. For example: 10.1.123.123,1433 or 10.1.123.123,1433\MSSQLSERVER ![ATLAS](assets/connect_server.png){: style="max-width:100%;"} Use Windows Authentication as the authentication method, the Windows Credential that we have set up earlier means that when you connect to the remote it will use that credential instead. !!! tip If you encounter this error message: ![ATLAS](assets/connect_server_error.png){: style="max-width:100%;"} That means your local machine cannot connect to the remote server. Ensure that remote connect is enabled and the protocols and port numbers are set correctly as outlined above. !!! tip If you encounter this error message: ![ATLAS](assets/connect_server_error2.png){: style="max-width:100%;"} Check your Windows Credential, this could be because the local credentials were passed, or because the username didn't include the domain prefix or used the wrong domain name. Double check all of the properties of the stored credential and recreate it if you need to (you can modify the username and password, but if the server name is wrong, you'll need to re-create it). #### Enter connection details in ATLAS The setting below might be required when using .NET Framework connecting to a SQL server in a MultiSubnet Environment. Follow the [link](https://techcommunity.microsoft.com/blog/sqlserversupport/connection-timeout-issue-with-net-framework-4-6-1---transparentnetworkipresoluti/318791) for more info. ```TransparentNetworkIPResolution = False``` To connect to this database, navigate to the Connection Manager in ATLAS as normal, and enter **{server IP}\{instance name},1433** The server’s name will not be present in the dropdown. But after entering the server name and let ATLAS connect to the database, the database name should be populated with all the available databases. ![ATLAS](assets/a10connection_manager.png){: style="max-width:100%;"} This should also create a corresponding entry in the the ```C:\Users\...\Documents\McLaren Electronic Systems\SQL Race Server\Config\ConnectionManagerV3.config``` file. #### Using the SQLRace API To read and write to databases cross domain, an entry would be required for the corresponding database in ConnectionManagerV3. Refer to the last section for the correct format. In the code, the connection string to be used for the SessionManager should be: ``` const string ConnectionString = @"Data Source=tcp:10.1.162.152\LOCAL,1433; TransparentNetworkIPResolution = False; Initial Catalog=SQLRACE02_155;Integrated Security=True;"; ``` #### Access to the filestream Depending on the SQL Race filestream is access controlled, you may have to follow similiar steps to allow access to the filestream folder using a different domained account. --- # Performance Guide This guide covers settings on how to get the most out of your SQL Race installation by adjusting settings for better performance. ## Cache Settings The SQL Race cache is a key component of SQL Race memory management, designed to optimize data access time by keeping recently added or loaded data in memory. Operations in RAM are much faster than continuously reading and writing data directly from disk. Raw channel and event data is stored in memory in “slots” created during recording new data or loading historical data. The cache settings can be modified and are stored by default in the configuration file SettingsStoreDefault.config located at: `C:\Users\{USER.NAME}\Documents\McLaren Electronic Systems\SQLRace\Config\SettingsStoreDefault.config` Applications can override the configuration file name. If the settings have never been changed, manually insert the following block of XML into the file within the `` tag: ``` ``` To modify the settings, change the value for each setting in the XML. ### Available Settings **CacheSize:** Maximum amount of memory (in MB) that SQL Race will use to cache raw data (default 1024). !!! warning "Changing the settings below is not advisable. Contact MA for further support." **PercentageToKeepFree:** The memory manager will attempt to keep at least this amount of cache available (default 10). **PercentageToFree:** Amount of the cache that should be freed when space needs to be made in the cache for new data (default 20). **MaximumSlotSize:** Maximum size of memory slot (in bytes) that can be cached. A smaller slot size may mean less memory is wasted, but it also means that there will not be as much read-ahead benefit when loading the data back into the cache. Setting this to 85000 or higher will likely result in excessive allocations on the LOH, so it is recommended to keep below that (default 80000). **MaximumTimeInMemoryBeforeFlushingInSeconds:** Amount of time data will be kept in memory before flushing to disk (default 300). **MaximumTimeInMemoryBeforeFreeingInSeconds:** Amount of time data will be kept in memory before the memory is released (default 300). **ParameterDataPoolingSize:** The maximum size (in MB) of the pool of ParameterValues objects used to fulfill data requests. The objects are reused if possible, to reduce memory allocations and subsequent garbage collection (default 512). ### When to change the settings For optimal performance, increase the “CacheSize” setting, assuming your system has enough free memory. A system with 16GB RAM should cope with the cache size increased to 2048. If ATLAS is used intensively with significant data throughput and the messages still appear, increase it to 3072 or even 4096, assuming the machine has enough available RAM. | RAM Size (GB) | Recommended Cache Size (MB) | |---------------|-----------------------------| | 8 | 1024 | | 16 | 2048 | | 32 | 3072 | | 64 | 4096 | !!! tip If you see messages like: > “Not enough memory freed. Flushing 14.864 MB immediately. Consider increasing your cache size” in the log file, the SQL Race cache is filling up too quickly. This results in excessive memory allocations and disk activity as ATLAS and SQL Race move more data around to stay within the limit. ## Cache Folder Caching data is a technique used to improve the performance and efficiency of applications. It involves storing frequently accessed data in a temporary storage location, such as memory or disk, which is faster to access than the original source. This reduces the time and resources needed to fetch data repeatedly from the main storage. Consider deleting the SQL Race cache folder improved performance, better disk space management, reduced startup time, and enhanced stability by resolving issues related to corrupted cache files. ## C# Garbage Collection C# uses an automatic memory management system called garbage collection (GC), managed by the Common Language Runtime (CLR). When you create a new object, memory is allocated for it on the managed heap, a region of memory reserved for .NET applications. The GC periodically checks for objects no longer in use by examining the application’s roots, such as static fields, local variables, and CPU registers. Unreachable objects are considered garbage, and their memory is reclaimed. After reclaiming memory, the GC compacts the remaining objects to reduce fragmentation and make memory allocation more efficient. ### SQL Race and Memory Disposal SQL Race doesn’t release memory immediately when a session is disposed. Instead, the garbage collector (GC) plays a crucial role in managing memory. Typically, resources are released after about 5 minutes. However, if the system requires memory sooner, the GC will release the resources earlier to accommodate the demand. This approach helps optimize memory usage and ensures that the system runs efficiently. ### Force GC Option Forcing garbage collection (GC) in ATLAS can have both benefits and drawbacks. On the positive side, it allows for immediate memory reclamation, which can be useful when you need to free up resources quickly. This can help optimize performance by reducing memory usage and preventing potential memory leaks. However, there are also some downsides. Forcing GC can introduce performance overhead, as the process can be resource-intensive and may temporarily slow down the application. It can also lead to unpredictable behaviour, disrupting the normal GC cycle and potentially causing issues if not managed carefully. ![Force GC](assets/force_gc.png){: style="max-width:50%;"} !!! note The Force GC feature was implemented to assist our developers in managing stress and benchmarking tasks, enabling quicker resource release between tests. --- # SQL Race, SQLite & SSN2 This document provides a clear comparison between SQL Race Database, SQLite and SSN2, focusing on their roles within motorsport telemetry and general data management. ## SQL Race Database SQL Race is a proprietary database access layer developed by Motion Applied, designed to handle high-frequency, structured telemetry data from motorsport environments. SQL Race sits on top of Microsoft SQL Server and provides a structured way to interact with telemetry and session data via SQL Race Database. **Key Features:** - Facilitates high-performance access to telemetry data stored in SQL Server. - Supports multi-session synchronisation, composite session management, and statistical operations. - Provides an API for programmatic access in Python, C#, MATLAB. - Optimised for large datasets and high concurrency. **Best For:** - Trackside and engineering teams needing robust performance and analytics at scale. - Scenarios requiring live telemetry and complex queries. ## SQLite SQLite is a SQL database used for simpler, local data storage needs embedded within ATLAS. It does not require a separate server process. The only way to create SQLite session (SSNDB) is using an ATLAS Recorder (`Database Engine = SQLite`) or directly through SRMS (SQL Race Management Studio). !!! tip ATLAS uses SQLite cache for storing telemetry or recorded data as local (temporary) cache. Users can move the data from local cache (SQLite) to SSN2 or SQL Race Database (using SRMS) if the data needs to be persisted. **Key Features:** - Zero configuration – no server setup required. - Portable and easy to integrate into standalone tools. - Limited concurrency (single-writer, multi-reader). **Best For:** - Testing, or small-scale tools. - Solo workflows or low-concurrency environments. ## SSN2 SSN2 is an ATLAS session file format that bundles session metadata and data streams together for portability. **Key Features:** - Stores session data in a single file. - Ideal for sharing and archiving sessions. - No separate filestream (unlike SQLite and SQL Race). **Best For:** - Offline workflows and data exchange between teams. - Quick review of sessions without database infrastructure. ## Feature Summary | Feature | **SQL Race Database** | **SQLite** | **SSN2** | |---------------------|-----------------------------------------------|----------------------------------------|-----------------------------------| | **Type** | SQL Race ontop of SQL Server | SQL Race ontop of SQL Server | File-based session format | | **Concurrency** | High (multi-session, multi-user) | Limited (single-writer, multi-reader) | Single-user | | **Performance** | Optimised for large, concurrent workloads | Good for small/local workloads | Basic (file read/write) | | **Setup** | Requires SQL Server infrastructure | None (Embedded in ATLAS Client) | None | | **Use Case** | Production-grade analytics, live telemetry | Lightweight local storage, prototyping| Sharing and archiving sessions | | **File Stream** | Yes | Yes | No | --- # Overview Distribute live telemetry seamlessly across multi‑client workflows. ## [Open Data Package](open_streaming_architecture/index.md) Provides open access to ATLAS data streams through standardised formats, enabling collaboration and integration with external systems and bespoke engineering tools. ### Key Functionality - Modern microservice architecture - Open telemetry data formats - Lightweight processing libraries - Virtual parameter calculation ## [Data Server Package](ads/index.md) Enables high‑performance distribution of live telemetry across multiple clients via Ethernet or Wi‑Fi. Designed for distributed teams working collaboratively. ### Key Functionality - Daisy‑chain server setups - Integration with Control Limited workflows - Multicast to multiple ATLAS clients - Coverage cursor for data integrity checks --- # Overview ATLAS Data Server is a core component of the ATLAS ecosystem, providing a robust and scalable solution for storing and managing large volumes of telemetry data. It is designed to handle high-throughput data ingestion, efficient storage, and fast retrieval, making it ideal for applications that require real-time data processing and analysis. ## Key Features 1. **Daisy-Chaining**: Built for distributed teams, ATLAS Data Server's can be daisy chained together to accommodate distributed teams without having to compromise on live data requirements. 2. **High Availability**: Designed for fault tolerance, ensuring that data is always accessible even in the event of hardware failures. 3. **Data Integrity**: Implements strong data validation and integrity checks to ensure the accuracy and reliability of stored data. 5. **Integration with ATLAS Tools**: Seamlessly integrates with other ATLAS tools and services, enabling a unified data management experience. ## System Requirements | Requirement | Minimum Supported | Recommended | |------------------|----------------------------------|-------------------------------| | Operating System | Windows 11 | Windows 11 | | Memory | 1 GB | 8 GB or higher | | Processor | Intel Core Duo 2GHz | Intel Core i7 Processor | | Disk Space | HDD with 200 GB free disk space | High Capacity SSD 512GB or more | ## Licensing ATLAS Data Server is available to users with Premium (1) or Data Platform variants of ATLAS. For more information contact the sales team [sales@motionapplied.com](mailto:sales@motionapplied.com). { .annotate } 1. Data Server Package is available to Premium users as an add-on. --- # Getting Started This guide walks you through installing and configuring the ATLAS Data Server (ADS) to manage and store telemetry data. ## Installation Steps 1. **Download the Installer** Get the latest ATLAS Data Server installer from the [Motion Applied Portal](https://portal.motionapplied.com/portal/Downloads/Software#Standalone%20ADS). 2. **Run the Installer** Launch the installer and follow the on-screen instructions. 3. **Configure the Server** After installation, open the Standalone ADS application: `Start > All > McLaren Applied Technologies > Data Server`. To add a SQLRace Database as a data source: `Tools > SQLRace > Connection Manager` and click **Add** to enter your SQL Server instance details. !!! note If you use ADS with SQLRace, install and configure SQLRace first. See the [SQLRace setup guide](../../store/sqlrace/installation.md). ## Data Server Setup The steps below configure ADS through its interface, which is the right approach for a single machine. !!! tip "Setting up more than one machine?" Every setting in this section can also be applied from an `ATLAS.ini` file, so a whole ADS configuration can be deployed by copying one file and restarting ADS — no clicking through the dialogs. See the [ATLAS.ini reference](atlas-ini.md). ### Server Settings File The Server Settings file contains all of the setup information for ATLAS Data Server. It's important to spend time configuring these settings so that you don't need to repeat the process every time you open ATLAS Data Server. 1. Create a Server Settings File by creating a new file `AtlasServer.sbk` and saving it to a known location. 2. Go to `Tools > Options` 3. Under `Recording > General` find `Server Settings File` and enter the path to your `AtlasServer.sbk` file. ### Log File The ATLAS Data Server creates a log file to record all events during operation. This log file is essential for troubleshooting and monitoring the server's performance. 1. Under `Recording > Logging (Debug)` find Log File Folder. 2. By default the log file is created in your Documents folder, at `McLaren Electronic Systems\ATLAS 9\log`. To update the location, change the path. ### Setting up IP addresses 1. Go to `Tools > Options > Data Server` 2. Find `Wide Band local address` and copy your IP address 3. Click OK to save and close the Options window. ### System Monitor Paths Setting up the System Monitor paths allows ATLAS Data Server to locate the necessary configuration and data files for proper processing. 1. Go to `Tools > Options > System Monitor Folders` 2. In the System Monitor Base Folder field enter the path to the System Monitor project (.pgv). 3. In the Logging Configuration Folder field enter the path to the logging configuration folder (LogCfgs) inside of the System Monitor project. 4. Click OK to save and close the Options window. --- # Recorders In ADS, a “recorder” is the configured input/output endpoint that defines how data enters ADS (protocol/device/source) and how/where it gets stored/offloaded/distributed. To access recorders, click `Setup > Add` | Source | Description | |--------|-------------| | Ethernet Telemetry | Data is continuously uploaded while the car is connected via Ethernet. | | Ethernet Wirelink | The contents of the data logging memory are uploaded when the car is connected via Ethernet. | | RF Telemetry | Data is continuously uploaded while the car is transmitting Wide Band Telemetry. | ### General tab The **General** tab is the first step in setting up ADS on your network. ![Recorder General tab](assets/recorder_general.png) Configure the following settings: - Network parameters - Status port for custom status monitoring services - *(Optional)* Broadcast Streams for global telemetry and wirelink settings ### All tab Use the **All** tab to define default session settings (details, export location, and so on) that apply to all future recorders on this ADS instance. ![Recorder All tab](assets/recorder_all.png) ## Recorder types ### Dual Recorder Dual Recorder writes to two different locations by duplicating the primary recorder. To enable Dual Recorder: 1. Go to **Tools** > **Options** > **Recording** > **General** 2. Set **Dual Recorder** to `true` 3. Set the **Dual Recording SQL Race DB** to Recorder 4. Set the **Dual Recording SQL Race Recording Folder** path to where you want to save the data 5. Click **OK** to save ![Dual Recorder Options](assets/dual_recorder.png) ### DST Multi Source Recorder A Multi Source Recorder automatically uploads data from available sources (Ethernet Telemetry, Ethernet Wirelink, or RF Telemetry). This ensures sessions use the best available data source at any time. Typically, Ethernet Telemetry operates in the pit, RF Telemetry on track, and Ethernet Wirelink offloads missed data. ![DST Multi Source Recorder](assets/dst.png) Configure these options: | Option | Description | |--------|-------------| | **Car number** | 1-4. Must match the value in System Monitor. | | **Car** | Select a Car Definition from the drop-down list. | | **Data source** | **DATA Lab Memory** (chassis parameters), **Burst Memory** (engine parameters), or **Burst and DATALab** (both). | ### TLM‑Px Multi Source Recorder Before configuring the TLM-Px Recorder in ADS, The Telemetry Controller PC must be configured (VPN Agent Manager, Telemetry Controller Dashboard, and Telemetry Controller Interface). Once the Telemetry Controller is set up, follow these steps to configure the TLM-Px Multi Source Recorder in ADS: 1. Click `Setup > Add` and select TLM‑Px Multi Source Recorder 2. Select the car by unit IP (example: `172.27.92.26`) 3. In **Advanced settings** > **TLM‑Px** tab, set **Port number** to match the Control Telemetry Output UDP port(example: `4001`) 4. Configure other settings as needed 5. Click **OK** to save ![TLM-Px Setup Guide](assets/tlmpx.png) --- # Options To access the options dialog box, go to `Tools > Options` in ATLAS Data Server (ADS). ![Options](assets/options.png) Every setting on this page is also a key that can be set in the `ATLAS.ini` file or written directly to the registry, which is how an ADS installation is configured without using this dialog. The `ATLAS.ini` key column below gives the name to use — see the [ATLAS.ini reference](atlas-ini.md) for how the file works. !!! note "Version note" Settings, keys and defaults below reflect ADS 9.87.x. If you are running an older version, a setting or its default may differ from what is shown here. !!! tip "Reading the tables" - **`ATLAS.ini` key** is both the INI key name and the registry value name. Unless stated otherwise it lives under `[ADSAdvancedSettings]`. - **Default** shows the ADS default, which for some settings differs from the ATLAS Viewer default for the same key. - **Restart** marks settings that only take effect after ADS is restarted. - Settings shown as TRUE/FALSE here are written as `1`/`0` in `ATLAS.ini`. ## Recording Tab The Recording tab of the Options dialog box contains advanced settings used to configure the Recording system. Settings are grouped in the dialog under the headings used below. ### Coverage Cursor | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Coverage Minimum Percentage (0-100) | `CoverageMinimumPercentage` | `98` | | This option sets the minimum percentage of quads received before the Coverage Cursor moves to its next position. | | Coverage Timeout (s) | `CoverageTimeout` | `60` | | The maximum time allowed for the Coverage Minimum Percentage to be reached. The Coverage Cursor moves to its next position when the Coverage Timeout is exceeded. | | Coverage Cursor Span (%) | `CoverageCursorSpanPercentage` | `50` | | Controls how the coverage cursor is calculated on an incomplete block (under construction); not used for completed blocks. See the coverage cursor section for more info. | | Quad sources excluded from coverage | `CoverageExcludedQuadSources` | `7` | | Comma-separated list of streams to exclude from the coverage cursor calculation. See the coverage cursor section for more info. | | Plot Coverage Percentage | `PlotCoveragePercentage` | FALSE | | Plots coverage cursor % as a green line in the Data Server Statistic Panel; line height = percentage, time = delay. | ### Custom Decode DLL | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Custom quad buffer size in quads per source | `CustomQuadBufferSizeQuads` | `100000` | | Buffer size to cover payload when data quads are missing (used for custom data logging units). | | Load decode DLL at Startup | `CustomQuadDecoderload` | FALSE | | When TRUE, loads the Decode DLL on application startup. | | Unload decode DLL after recording | `CustomQuadDecoderUnload` | FALSE | | When TRUE, unloads the Decode DLL after recording. | | Max quads generic recorder groups before sending | `GenericRecorderMaxQuadsInGroup` | `10` | | Limits max quads sent in one group so the generic recorder sends more often. | ### Data Server | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Data Servers | `WideBandDataServers` | Blank | | List of ATLAS Data Server names and corresponding ports in the form `{Server:Port}`. | | Wide Band local address | `WideBandLocalAddress` | ANY | | Selects which network card is used for sending/receiving data; shows IPs of all NICs present. | | Data Server connection timeout (ms) | `ServerConnectionRetryTime` | `20000` | | Time ATLAS waits for data from a Data Server before stopping recording. | | Server auto-refresh frequency (ms) [0=disable] | `ServerAutoRefreshFreq` | `2000` | | Periodic refresh of current data servers shown in the Record dialog. | | Server auto-refresh reply timeout (ms) | `ServerAutoRefreshReplyTimeout` | `100` | | Time before a data server is deemed offline when no activity is detected. | | Timeout for searching for Data Servers (ms) | `SearchForServersTimeout` | `0` | | Time clients search for available Data Servers before timing out. | | Server reply timeout [ms] | `ServerReplyTimeout` | `3000` | Yes | Time ATLAS waits for a reply from a Data Server. | | Data Server Buffer size Mb | `WideBandBufferSize` | `20` | Yes | Size of client storage buffer for incoming data; should be large to avoid loss on poor networks. | | Data Server Retries Enabled | `WideBandRetryMode` | FALSE | Yes | When TRUE, client may retry loading data from the network. | ### General | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Close record dialog after start/stop recording | `CloseRecDlgStopStart` | TRUE | | If TRUE, Record dialog closes when Start/Stop clicked; if FALSE, stays open. | | Show Record Error Messages | `ShowRecordErrors` | FALSE | | When TRUE disables recording error pop-ups and writes errors to the Data Server log file. | | Minimum Disk Space (MB) | `MinimumDiskSpace` | `500` | | If free disk space drops below this value, recording stops (with warnings beforehand). Minimum value is 500MB. | | Enable Record to Disk | `EnableRecordToDisk` | TRUE | | When FALSE, data is only multicast to clients; no session is saved. | | Quad sources excluded from lap trigger | `LapExcludedQuadSources` | Blank | | Rejects lap triggers sourced from listed streams (comma-separated). | | Virtual Memory Block Size (KB) | `MemoryBlockSize` | `1024` | | Virtual memory slot size used by memory mapping in ATLAS. | | Server Settings File | `SettingsFile` | `\AtlasServer.sbk` | Yes | Location of the server workbook (.sbk) holding Data Server configuration settings. | | Auto Record connection check interval (ms) | `MonitorUnitStatusInterval` | `1000` | | Time ATLAS waits before stopping Automatic Recording. | | Unload unused Pgvs on Start Recording | `ClearPgvCacheOnStartRec` | FALSE | | When TRUE, PGVs are removed from the cache when recording starts. | | DTV Rename Cached Files | `RenameDtvCachedFiles` | FALSE | | When TRUE, cached DTV files include `_Cached` in the filename; set 'Unload Unused PGVs on Start Recording' if cached file not required. | | Auto Increment Run Number | `AutoIncrementRunNumber` | FALSE | | Automatically increments Session Number name (e.g., R1, R2) for sequential recordings. | | Hours to Delete Configs 0 = Disabled | `TimeInHoursToDeleteConfig` | `0` | | Automatically delete obsolete config files to optimize resources. | | Enable 'still processing parameters' Dialog | `EnableAbortSavedParamsDlg` | TRUE | | Warns that parameter processing from config files is incomplete if recording ends prematurely. | | Default disabled 'still processing parameters' to continue | `SavedParamsDlgDefaultContinue` | TRUE | | Sets which button the 'still processing parameters' dialog defaults to. When TRUE the default is to continue, so an unattended recording is not held up waiting for a response. | | Automatic Multicast Address | `AutomaticMulticastAddress` | FALSE | Yes | When TRUE, server derives Multicast IP from last three integers of Local IP bind. | | Setup Recorders From Registry | `SetupRecordersFromRegistry` | FALSE | Yes | Saves Recorder settings in registry rather than with a Workbook. | | Temporary directory for recording files (default: blank) | `TempRecordingDirectory` | Blank | Yes | If set, session files are written here first and copied to the data path when recording stops. If blank, they are written directly to the data path. | | Transmission timeslice (ms) | `TransmissionTimeslice` | `10` | | Period ADS sends multicast to the network (reducing may even peak load); 10ms should be minimum. | | Delay Wirelink upload after recording (s) | `DelayWirelinkUpload` | `0` | | Delay (seconds) for data upload from unit at end of recording. | | Auto-dismiss config search dialog | `AutoDismissConfigSearch` | TRUE | | When TRUE, config search box auto-closes. | | Warn when config is missing | `WarnMissingConfig` | TRUE | | Warns when a configuration file needed for a recording cannot be found. | | Send missing configs to clients | `SendMissingConfig` | TRUE | | Sends missing configuration files to clients that request them. | | Recording event script | `RecordingEventScript` | Blank | | User-defined script called in a new process on recording events (LIVE_STARTED, LIVE_STOPPED, OFFLOAD_STARTED, OFFLOAD_STOPPED). | | SQL Data Service Instance | `SQLDataServiceInstance` | Blank | Yes | Name of the SQL Data Service instance used for SQL Race recording. Leave blank to use the default instance. | | DTV Offload Exclusion List | `DTVOffloadExclusionList` | Blank | Yes | Keywords specifying DTVs to exclude from offload (supports application-level and unit-specific formats). | | PGV Exclusion List | `ADSLDFPGVExclusionList` | Blank | | Keywords specifying PGVs to exclude from processing at start of recording (application-level and unit-specific formats). | | PGV Inclusion List | `PGVInclusionList` | Blank | | Keywords specifying PGVs to include in processing; inclusion list overrides exclusion list. | | PGV Standalone List (Hex) | `PGVStandaloneList` | Blank | Yes | Hexadecimal keywords identifying PGVs to be treated as standalone. | | Auto Merge Option | `EnableAutoMerge` | Disabled | | Drop-down: Disabled (sources only from .cfg), Maximum (all sources added), Minimum (one parameter added but merge performed on all sources; not selectable in ADS). | | Process Weather.prm | `ProcessWeatherPgv` | FALSE | | Processes `Weather.prm` during recording. | | Process Scalar | `ProcessScalar` | FALSE | | Processes scalar parameters during recording. | | Enable Pre-Record Config Transfer | `EnablePreRecordConfigTransfer` | TRUE | Yes | When TRUE, allows sending config files when not recording. | | Pre-Record Config Transfer Boost Time (s) | `PreRecordConfigTransferBoostTime` | `10` | Yes | Duration (seconds) that config send rate increases to boost rate. | | Pre-Record Config Transfer Rate (KBps) | `PreRecordConfigTransferRate` | `50` | Yes | Raised config send rate used during boost time. | | Enable New Unique Filter | `EnableNewUniqueMethod` | FALSE | Yes | Enables new unique filter and disables Missing Data Upload; allows Quad ID to wrap while still ensuring uniqueness on network. | | Enable dual recording | `EnableDualRecording` | FALSE | Yes | Enables the out-of-process dual recorder, which writes a second copy of the recording via a separate process. | | Dual recording end point | `DualRecordingEndPoint` | `[::1]:9798` | Yes | Endpoint of the dual recording process, in `host:port` form. The default is IPv6 loopback. | | Dual recording SQL Race DB | `DualRecordingDB` | Blank | Yes | SQL Race connection used for the dual recording. | | Dual recording SQL Race Recording Folder | `DualRecordingSqlRaceRecordingFolder` | `\Server` | Yes | Folder the dual recorder writes SQL Race session files to. | | Dual recording server listener IP | `DualRecordingServerListenerIP` | `127.0.0.1` | Yes | Local IP address the dual recording listener binds to. | | Dual recording server listener port | `DualRecordingServerListenerPort` | `6800` | Yes | Port the dual recording listener binds to. | | Enable FTP download | `EnableFTPDownload` | FALSE | | Enables downloading data from a logging unit over FTP. | | Enable Bridge Service | `EnableRemoteDataFeed` | FALSE | Yes | Master switch for the Bridge Service. When disabled, ADS makes no Bridge Service connection and the status indicator stays grey. | | Bridge Service Endpoint | `RemoteDataFeedEndpoint` | `[::1]:9697` | Yes | Address of the Bridge Service in `host:port` form. Square brackets are required around an IPv6 literal; the default is IPv6 loopback. | | Local Bridge Service | `LocalRemoteDataFeed` | FALSE | Yes | When enabled, ADS starts and stops the Bridge Service process itself. When disabled, it connects to a Bridge Service that is already running and leaves it running on exit. | | *(not shown in dialog)* | `TempSqlRaceRecordingDirectory` | `\Server` | | Folder used for in-progress SSN2 recordings. Set automatically from `Temporary directory for recording files`, or the data folder, and not editable in the dialog. *Not editable in the dialog.* | ### Handshake Server | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Handshake Minimum Time [ms] | `HandshakeMessageTimeoutMin` | `100` | | Minimum timeout for handshake message from data server to car. | | Handshake Maximum Time [ms] | `HandshakeMessageTimeout` | `200` | | Maximum timeout for handshake message from data server to car. | | Handshake Message Queue Length | `HandshakeMessageQueueLength` | `2` | | Maximum message queue length awaiting transmission. | | Handshake Message Size | `HandshakeMessageSize` | `640` | | Must match message size supported by embedded system. | | Use Low Rate Message | `HandshakeMessageUseLowRateMsg` | FALSE | | When TRUE, uses compressed handshaking messages. | | Low Rate Maximum Time [ms] | `HandshakeMessageLowRateTimeout` | `2000` | | Maximum time between low-rate handshaking messages. | | Low Rate Maximum Message Size | `HandshakeMessageLowRateSize` | `18` | | Maximum size of low-rate handshaking messages. | ### Advanced Data Feed Settings for the Advanced Data Feed, which publishes recorded data to Kafka. All of these require a restart. | Setting | `ATLAS.ini` key | Default | Description | |---|---|---|---| | Enable Advanced Data Feed Producer | `ADFProducerEnable` | FALSE | Enables the Advanced Data Feed Kafka producer. | | client.id | `client.id` | `ADS` | Client identifier presented to the Kafka brokers. | | metadata.broker.list | `metadata.broker.list` | Blank | Comma-separated list of Kafka brokers to connect to. Required when the producer is enabled. | #### Kafka client properties The remaining 73 settings in this group are **librdkafka** producer properties. The key name *is* the librdkafka property name, and the value is passed to the Kafka client unchanged, so the librdkafka configuration reference is the authority on what each one does. They are shown in the dialog but cannot be edited there — set them through `ATLAS.ini` or the registry if you need to change them. | `ATLAS.ini` key | Default | |---|---| | `builtin.features` | Blank | | `message.max.bytes` | `1000000` | | `receive.message.max.bytes` | `100000000` | | `metadata.request.timeout.ms` | `60000` | | `topic.metadata.refresh.interval.ms` | `300000` | | `topic.metadata.refresh.fast.cnt` | `10` | | `topic.metadata.refresh.fast.interval.ms` | `250` | | `topic.metadata.refresh.sparse` | TRUE | | `topic.blacklist` | Blank | | `debug` | Blank | | `socket.timeout.ms` | `60000` | | `socket.blocking.max.ms` | `100` | | `socket.send.buffer.bytes` | `1048576` | | `socket.receive.buffer.bytes` | `1048576` | | `socket.keepalive.enable` | FALSE | | `socket.max.fails` | `3` | | `broker.address.ttl` | `1000` | | `broker.address.family` | `any` | | `reconnect.backoff.jitter.ms` | `0` | | `statistics.interval.ms` | `0` | | `log_level` | `6` | | `log.thread.name` | FALSE | | `log.connection.close` | TRUE | | `internal.termination.signal` | `0` | | `quota.support.enable` | FALSE | | `protocol.version` | `0` | | `security.protocol` | `plaintext` | | `ssl.cipher.suites` | Blank | | `ssl.key.location` | Blank | | `ssl.key.password` | Blank | | `ssl.certificate.location` | Blank | | `ssl.ca.location` | Blank | | `sasl.mechanisms` | `GSSAPI` | | `sasl.kerberos.service.name` | `kafka` | | `sasl.kerberos.principal` | `kafkaclient` | | `sasl.kerberos.kinit.cmd` | `kinit` | | `sasl.kerberos.keytab` | Blank | | `sasl.kerberos.min.time.before.relogin` | `60000` | | `group.id` | Blank | | `partition.assignment.strategy` | `range,roundrobin` | | `session.timeout.ms` | `30000` | | `heartbeat.interval.ms` | `1000` | | `group.protocol.type` | `consumer` | | `coordinator.query.interval.ms` | `600000` | | `queued.min.messages` | `100000` | | `queued.max.messages.kbytes` | `1000000` | | `fetch.wait.max.ms` | `100` | | `fetch.message.max.bytes` | `1048576` | | `max.partition.fetch.bytes` | `1048576` | | `fetch.min.bytes` | `1` | | `fetch.error.backoff.ms` | `500` | | `offset.store.method` | `broker` | | `queue.buffering.max.messages` | `100000` | | `queue.buffering.max.ms` | `1000` | | `message.send.max.retries` | `2` | | `retry.backoff.ms` | `100` | | `compression.codec` | `none` | | `batch.num.messages` | `1000` | | `delivery.report.only.error` | FALSE | | `request.required.acks` | `1` | | `request.timeout.ms` | `5000` | | `message.timeout.ms` | `300000` | | `produce.offset.report` | FALSE | | `auto.commit.enable` | TRUE | | `enable.auto.commit` | TRUE | | `auto.commit.interval.ms` | `60000` | | `auto.offset.reset` | `largest` | | `offset.store.path` | Blank | | `offset.store.sync.interval.ms` | `-1` | | `offset.store.method` | `file` | | `consume.callback.max.messages` | `0` | | `opaque` | FALSE | | `ADFCompleteSessionsTimeout` | `60000` | ### Live Data Feed | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | ldf.metadata.broker.list | `LiveData2ConnectionBrokerList` | Blank | Yes | Comma-separated list of Kafka brokers used by the Live Data Feed connection. | | Enable Live Data Feed | `ADSLDFEnable` | FALSE | Yes | When TRUE, data can be transferred via live data feed. | | Send All Data | `ADSLDFSendAllData` | FALSE | | Sends all data over the Live Data Feed rather than only the requested subset. | | Live Data Feed Retrieval Path | `ADSLDFFilePath` | `\McLaren Electronic Systems\ATLAS 9\LDS` | | Location used to store intermediate (LDR) files when using LDF. | ### Logging (Debug) | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Log File Folder | `LogFileFolder` | `\McLaren Electronic Systems\ATLAS 9\log` | | Location for log files. | | Raw Data Logging | `RawDataLogging` | FALSE | | Saves raw telemetry data during recording to .raw and .raw_tm files (for diagnostics). | | Raw Data Root Folder | `RawDataRootFolder` | Blank | | Location for files created by Raw Data Logging. | | Raw Data Logging [VTS] | `RawVTSDataLogging` | FALSE | | Saves VTS raw data during recording to .raw and .raw_tm files (for diagnostics). | | Performance Statistics | `PerformanceCounterStats` | FALSE | | When TRUE, creates performance log file (developer analysis/debugging). | | Message log level (for debugging) | `MessageLogSetting` | Not Logging | | Level of diagnostic message logging. Intended for support and development use. | | Message log file size (MB) | `MaxLogFileSizeMB` | `5` | Yes | ATLAS log file size limit (0MB = infinite; new file created when limit reached). | ### Offload | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Enable Update Progress On Clients (3 Source) | `WireLinkUpLoadStatusBroadcast` | TRUE | | Enables client update progress bar on ADS dialog during data upload. | | Enable Multicast for Wirelink (3 Source) | `WireLinkMulicastControl` | TRUE | | When TRUE, all three recording sources enabled in ADS Advanced Multi Source setup (may hinder network performance). | | Exclude Offload Streams | `ExcludeOffloadStreams` | Blank | | Prevent ADS from offloading specific unit streams (comma-separated list). Overrides may apply with Telemetry Streams setting. | | Enable Reverse Upload | `EnableReverseUpload` | FALSE | | Uploads missing data in reverse order, newest first. | | Apps To Exclude From Clear Max/Min/Errors | `ExcludeAppsFromClear` | `CBT610;CBR610` | | Semicolon-separated list of applications excluded when max/min values and errors are cleared on a logging unit. | | Produce Encrypted Export Files | `ProduceEncryptedExportFiles` | FALSE | | Encrypts export files produced at the end of a recording. | ### Remote Data Server | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Enable Remote Data Server | `ADSRDREnable` | FALSE | | When TRUE, data can be transferred via the remote data server. | | Remote Data Server Retrieval Path | `ADSRDRFilePath` | `\McLaren Electronic Systems\ATLAS 9\RDS` | | Location used to store intermediate (RDR) files when using RDS. | | RDS Write to Log File | `RDS Write to Log File` | TRUE | | When TRUE, RDS diagnostics written to an RDS log file in the main ATLAS log folder. | | RDS Logging level | `RDSMessageLogSetting` | Minimum | Yes | Verbosity of Remote Data Server diagnostic logging. | | Auto delete PARTIAL Sessions | `ADSAutoDeletePartialSSN` | TRUE | | After playback when .ssn is created, delete PARTIAL live session file (when unlocked). | | Enable RDR Transfer During Live (Master Only) | `ADSRDRTransferDuringLive` | FALSE | | Enables ADS to transmit RDR data simultaneously with live recording if spare bandwidth available. | | Enable Multi-Session RDR | `ADSRDREnableMultiSession` | FALSE | | Allows the Remote Data Server to transfer more than one session at a time. | | Enable VTS Session Save | `EnableRdsSaveVtag` | FALSE | | Saves vTAG session data received over the Remote Data Server connection. | ### Telemetry | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Telemetry Guard Time (s) | `TelemetryGuardTime` | `0` | | Reject data if timestamp difference between packets exceeds this value. | | Telemetry Guard Filter Length | `TelemetryGuardSynchronizationLength` | `10` | | Telemetry Guard synchronization length. | | Telemetry Guard Quad Range Limit 0 = Disabled | `TelemetryGuardQuadRange` | `0` | | Maximum allowable gap between sequential Quad IDs within a stream. | | Live Telemetry Timeout 0 = Disabled | `LiveTelemTimeout` | `50` | | For Marelli DST Receiver only: time after which ATLAS stops recording if no live data received (overridden by active Ethernet recorders). | | Session Completion Timeout 0 = Disabled | `SessionTimeout` | `0` | | Time (seconds) to stop recording after last data sample; 0 disables. | | Enable VTS Session Completion Timeout | `VTSSessionTimeoutEnable` | TRUE | | Allows VTS Recorder to timeout per Session Completion Timeout; FALSE overrides timeout. If timeout is 0, this has no effect. | | Telemetry Streams To Complete Missing Offload | `TelemetryStreams` | Blank | | Streams required for missing telemetry uploads before closing (comma-separated); uploaded first; live record status cleared after last stream completes; overrides Exclude Offload Streams. | | Transmitter Status Timeout 0 = Disabled | `CBTStatusTimeout` | `0` | | Time (seconds) that a status from a CBR600 is valid; after this time, last status ignored until a new one arrives. | | Sequence number anti-wrap time (hours) 0 = Disabled | `AntiWrapTime` | `0` | | Threshold delta time where sequence numbers auto-unwrap; applies only to current instance (forwarded data unaffected). | | DST Interface Version | `DSTInterfaceVersion` | 3 | Yes | Version of the Magneti Marelli DST receiver interface to use. Must match the receiver in use. | | SCS Logging Control Streams | `ScsTelemetryLoggingControlStreams` | `1,2` | | Comma-separated list of streams used for Standard Communication System logging control. | ### Unit Comms | Setting | `ATLAS.ini` key | Default | Restart | Description | |---|---|---|---|---| | Ethernet Receive Timeout (ms) | `EthernetReceiveTimeout` | `500` | | Time ATLAS waits before retrying a receive message. | | Ethernet Send Timeout (ms) | `EthernetSendTimeout` | `500` | | Time ATLAS waits before retrying a send message. | | Ethernet retries | `EthernetRetries` | `1` | | Number of times ATLAS retries a message before aborting and showing a warning. | | Ethernet upload message interval timeout (ms) | `EthernetUploadMsgTimeout` | `3` | | Time before Quad Ethernet Telemetry data is re-requested during an upload (auto-extended if data receipt detected beyond timeout). | | Ethernet upload message extended timeout (ms) | `EthernetUploadExtendTimeout` | `500` | | Time ATLAS waits before Quad Ethernet Telemetry offload times out. | | Ethernet offload connection retries | `EthernetUploadConnectionRetries` | `10` | | Number of reconnection attempts during offload if connection is lost. | | Ethernet Minimum UDP Missing Request command size Limit | `EthernetMinimumMissingRequestLimit` | `512` | | Minimum command size when requesting missing data to avoid IP fragmentation issues (steps down from maximum when timeouts occur). | | Ethernet Maximum UDP Missing Request command size Limit | `EthernetMaximumMissingRequestLimit` | `1472` | | Maximum command size when requesting missing data; may need adjustment if VPN introduces fragmentation. | | Ethernet upload config unique response | `EthernetUploadCfgResponse` | FALSE | | Setting must match client-specific BIOS configurations. | | Enable missing data upload | `EnableMissingDataUpload` | TRUE | | When TRUE, only data found to be missing is uploaded; when FALSE, all data uploaded regardless. | | Ethernet Buffer Size (MB) | `EthernetBufferSize` | `5` | Yes | Size of local cache used for Direct Ethernet Wirelink. | | Default Timeout (ms) | `DefaultTimeout` | `200` | Yes | Default timeout applied to unit communications that have no more specific timeout. | | Enable Unit Status Message | `EnableUnitCfgStatusMessage` | TRUE | Yes | Enables the configuration status message sent to logging units. | ## Recorder Folders Tab The Recorder Folders tab sets the paths ADS uses for configurations and parameter files. These keys live under **`[Preferences]`**, not `[ADSAdvancedSettings]`. | Setting | `ATLAS.ini` key | Description | |---|---|---| | System Monitor 7 base folder | `SystemMonitor7BaseFolder` | Path where System Monitor files are stored. Multiple paths can be separated with semicolons. | | Logging configuration folder | `ConfigFolder` | Path where ADS writes logging configuration files. | | Ancillary parameters folders | `AncilliaryParametersFolders` | Paths where logging configurations created in System Monitor are stored, semicolon-separated. Note the doubled `l` in the key name. | | Parameter unlock list path | `ParameterUnlockListDir` | Path ADS searches for parameter unlock lists. Also passed to SQL Race as its unlock list search path. | | Custom PGV path | `CustomPGVFolder` | Path where the LDF recorder stores received PGV files. | | Cars database file | `CarDatabaseFile` | Path and file name of the car definitions file. Stored under `HKCU\Software\McLaren Electronic Systems`, not under the ATLAS 9.0 key, so it cannot be set from `ATLAS.ini`. | Other `[Preferences]` paths that ADS uses but that are not on this tab — including `DataFolder`, `WorkbookFolder` and `LocalFolder` — are listed in the [ATLAS.ini reference](atlas-ini.md#preferences). ## Slave ADS Connection Tabs Two tabs, **Slave ADS Connection 1** and **Slave ADS Connection 2**, configure outgoing connections to slave data servers. !!! note "Licence gated" These tabs only appear if the **Remote Data Server** licence option is present. Without it, both connections are forced off regardless of what is configured. In the keys below `N` is `0` for the first tab and `1` for the second. The two connections must not use the same port. `Allow connection` cannot be changed while a recording is in progress. | Setting | `ATLAS.ini` key | Default (tab 1 / tab 2) | Description | |---|---|---|---| | Allow connection | `SlaveADSConnectionNEnable` | FALSE | Enables this slave connection. | | Port | `SlaveADSConnectionNPort` | 5010 / 5011 | Port used for the connection. | | Auto data rate | `SlaveADSConnectionNAutoRate` | TRUE | Adjusts the data rate automatically rather than using the fixed maximum. | | Max data rate | `SlaveADSConnectionNMaxRate` | 500 | Maximum data rate in KB/s. | | Min data rate | `SlaveADSConnectionNMinRate` | 0 | Minimum data rate in KB/s. | | Compression | `SlaveADSConnectionNCompression` | FALSE | Compresses data sent over the connection. | | Send VTAG data | `EnableVtagDataRdsConnectionN` | FALSE | Includes vTAG data in the connection. | | Quad sources excluded | `SlaveADSQuadSourceNExcluded` | Blank | Comma-separated list of streams to exclude from this connection. | | *(no control)* | `SlaveADSConnectionNRetransmission` | FALSE | Enables retransmission of lost data. | | *(no control)* | `SlaveADSConnectionNMaximumSize` | 100 | Maximum block size before sending. | | *(no control)* | `SlaveADSConnectionNMaximumTime` | 1 | Maximum time to wait before sending a partial block. | ## Live Data Feed Connection Tab Configures the Live Data Feed connections. The broker list for the current connection is on the Recording tab as `LiveData2ConnectionBrokerList`; the remaining keys are on this tab. | Setting | `ATLAS.ini` key | Default | Description | |---|---|---|---| | Allow | `LiveData2ConnectionEnable` | FALSE | Enables the Live Data Feed connection. | | Send VTAG data | `LiveData2SendVtagData` | TRUE | Includes vTAG data in the feed. | | Data rate | `LiveData2ConnectionMaxRate` | Blank | Maximum data rate in KB/s. Blank applies no limit. | | Quad sources excluded | `LiveData2ConnectionStreamsExcluded` | Blank | Comma-separated list of streams to exclude. | !!! warning "Deprecated connections" The two numbered connections below are labelled **Deprecated** in ADS and are retained only for existing installations. Use the connection above for new configurations. | Setting | `ATLAS.ini` key (`N` = 0 or 1) | Default (1 / 2) | Description | |---|---|---|---| | Allow | `LiveDataConnectionEnableN` | FALSE | Enables this legacy connection. | | Port | `LiveDataConnectionPortN` | 5012 / 5013 | Port used for the connection. | | Data rate | `LiveDataConnectionMaxRateN` | 150 | Maximum data rate in KB/s. | | Send VTAG data | `LiveDataConnectionNSendVtagData` | TRUE | Includes vTAG data in the feed. | | Quad sources excluded | `LiveDataConnectionStreamsExcludedN` | Blank | Comma-separated list of streams to exclude. | | *(no control)* | `LiveDataConnectionMinRateN` | 50 | Minimum data rate in KB/s. | | *(no control)* | `LiveDataConnectionCompressionN` | FALSE | Compresses data sent over the connection. | | *(no control)* | `LiveDataConnectionAutoRateN` | FALSE | Adjusts the data rate automatically. | | *(no control)* | `LiveDataConnectionEnableRowData` | TRUE | Includes row data in the legacy feed. | !!! note `LiveDataConnectionNSendVtagData` follows a different pattern to the other keys — the index comes immediately after `LiveDataConnection`, giving `LiveDataConnection0SendVtagData`. ## Library Tab Sets the function libraries loaded by ADS, and whether session constants are sent to clients. | Setting | `ATLAS.ini` key | Description | |---|---|---| | Use server constants | `SendSessionConstantsToClients` | Sends session constants from the server to connected clients. Lives under `[AtlasDataServer]`. | | Library list | `Defaults`, `DefaultsAttrib`, `Order`, `DefaultsType` | Function library entries, under the `ADSLibrary` key. ADS uses its own library key, separate from the ATLAS Viewer's. | !!! note The library list values are multi-string registry values and **cannot** be set from `ATLAS.ini`. Configure them in the interface, or write them to the registry as `REG_MULTI_SZ`. ## Decode DLLs Tab The Decode DLL tab is used to allocate customer-created DLLs to a data stream. A total of 15 data streams are available. ![decode_DLLs](assets/decode_dll.png) | Setting | `ATLAS.ini` key | Description | |---|---|---| | Decode DLL Path | `CustomQuadDecoderFilepath` | Enter the full path and filename of the Decode DLL, or browse to locate it. The installer seeds this with the bundled `CustomDLLs\AtlasApiDecoder.dll`. | Related settings are on the Recording tab under [Custom Decode DLL](#custom-decode-dll). ## System Monitor Folders Tab The System Monitor Folders tab controls how ADS works with System Monitor configurations. These keys are under **`[Preferences]`** and are the same values as the [Recorder Folders tab](#recorder-folders-tab). | Setting | `ATLAS.ini` key | Description | |---|---|---| | System Monitor 7 base folder | `SystemMonitor7BaseFolder` | Sets the path where System Monitor files are stored. Multiple paths can be separated with a semi-colon. | | Logging configuration folder | `ConfigFolder` | Sets the path where ADS writes Logging configuration files. File structure must be: `…ATLAS V8\Config\Logging\AutoConfig` | | Car definitions file | `CarDatabaseFile` | Sets the path and file name of the Car definitions file that defines network communication. | | Ancillary Parameters Folder | `AncilliaryParametersFolders` | Sets the path where Logging Configurations created in System Monitor are stored. | | Parameter Unlock List Path | `ParameterUnlockListDir` | Sets the path where ADS searches for Parameter Unlock Lists. | | Custom PGV Path | `CustomPGVFolder` | Sets the path where the ADS LDF Recorder stores received PGV files. | ## Settings with no dialog control These are read from `[ADSAdvancedSettings]` but have no control in the Options dialog. They are included for completeness — most are internal or legacy, and should only be changed on advice from support. | `ATLAS.ini` key | Group | Default | Description | |---|---|---|---| | `WideBandInfoPort` | Data Server | 9696 | UDP port used to broadcast to and discover data servers. | | `EnableADSTestMode` | General | FALSE | Enables ADS test mode. | | `NarrowBandDataServer` | General | Blank | Legacy narrow band data server list. | | `NarrowBandSequenceWindowLength` | General | 4 | Legacy narrow band sequence window length. | | `NarrowBandWindowLength` | General | 10000 | Legacy narrow band window length. | | `PreciseLapTimes` | General | TRUE | Enables precise lap time calculation. | | `EnableAddFixedParameters` | General | TRUE | Adds fixed parameters to recorded sessions. | | `EnableEthernetQuadTelemetry` | General | TRUE | Enables quad Ethernet telemetry. | | `HandshakeMessagePLMN` | Handshake Server | 23415 | PLMN provider code used in handshake messages. | | `HandshakeMessageTechnology` | Handshake Server | 2 | Network technology type used in handshake messages. | | `HandshakeMessageUseSlowRowIP` | Handshake Server | FALSE | Uses the slow row for the IP address in handshake messages. | | `broker.version.fallback` | Advanced Data Feed | `0.9.0.0` | Kafka broker version assumed when it cannot be determined. | | `ADSSyncRDRFolderCheck` | Remote Data Server | TRUE | Checks the RDR folder synchronously. | | `RDRPerformanceCounterSubsample` | Remote Data Server | 16 | Subsampling factor for RDR performance counters. | | `AutoTelemTimeout` | Telemetry | 20 | Timeout used by automatic telemetry detection. | | `EnableDstParameters` | Telemetry | FALSE | Enables DST-specific parameters. | | `ScsLoggingControlCount` | Telemetry | 10 | Standard Communication System logging control count down. | ## Settings no longer in this dialog If you are working from an older document or an existing `ATLAS.ini`, note that the following are **not** ADS Options settings: - **SQL Race logging and performance counters** — `SQL Race Logging Level`, `SQL Race Log File Path`, `SQL Race Logging Period`, `Enable SQL Race Perf Counters`, `Enable SQL Race Perf Counters Log to File`, `Enable SQL Race Background Metrics` and `SQL Race Cache Size` are no longer registered as ADS settings and have no effect if set. Configure SQL Race logging through SQL Race itself. - **Session File Type** — chosen per recording in the Record dialog destination, not here. - **Viewer-only settings** — `Wideband Min Lap Time`, `Record Telemetry Data From Start`, `Record Only Live Data`, `Unload Session on Start Recording`, `Save Client Config File to Disk`, `Use Only the Highest Rate Continuous Channel in Merging`, `Regenerate Merged Channels at the end of a Recording` and `PGV Request Timeout` exist in the ATLAS Viewer but are not shown or used by ADS. ## See also - [ATLAS.ini reference](atlas-ini.md) — setting any of these keys from a file, for scripted deployment - [Recorders](recorders.md) — recorder types and their settings - [RDS](rds.md) — Remote Data Server configuration --- # ATLAS.ini File `ATLAS.ini` is a plain text file that sets ATLAS Data Server (ADS) configuration without using the GUI. Any setting normally changed in `Tools > Options` can be written into this file instead, which is what makes a scripted or automated ADS deployment possible. Values supplied by the file also become read-only in the interface, so a configuration can be applied consistently and left locked. !!! note "Version note" Keys, defaults and behaviour on this page reflect ADS 9.87.x. If you are running an older version, a default may differ from what is shown here. ## Why use ATLAS.ini Configuring ADS through the GUI is fine for a single machine, but it does not scale and it cannot be scripted. `ATLAS.ini` solves three problems: - **Repeatable deployment.** One file, copied to each machine, replaces a long click-through of the Options dialog. - **Locked-down settings.** A setting that comes from `ATLAS.ini` is displayed greyed out and cannot be edited or overwritten from the interface. This is useful for values that must not be changed. - **Multi-user and service accounts.** The ADS installer writes its default settings under `HKEY_CURRENT_USER` for the installing user only. If ADS runs under a different account, those defaults never materialise. ## Where ATLAS.ini lives By default ADS looks for the file at: ``` McLaren Electronic Systems\ATLAS 9\Atlas.ini ``` Two things about this path: - **It is per-user.** The file must be in the Documents folder of the account ADS runs under, not the account that installed it. - **Folder redirection applies.** If Documents is redirected — most commonly by OneDrive Known Folder Move — the path follows the redirect, for example `C:\Users\...\McLaren Electronic Systems\ATLAS 9\Atlas.ini`. This PowerShell snippet writes the file to the correct location for the current user: ```powershell $dir = Join-Path ([Environment]::GetFolderPath('MyDocuments')) 'McLaren Electronic Systems\ATLAS 9' New-Item -ItemType Directory -Force $dir | Out-Null @' [ADSAdvancedSettings] EnableRemoteDataFeed = 1 '@ | Out-File (Join-Path $dir 'Atlas.ini') -Encoding ascii ``` ### Moving the file The location can be changed with a registry value that holds the full path **including the file name**: | Registry key | Value | Type | |---|---|---| | `HKEY_CURRENT_USER\Software\McLaren Electronic Systems\ATLAS\9.0\Preferences` | `ATLASIniFile` | String | In the ATLAS Viewer this is exposed as `Tools > Options > Standard`. ADS has no equivalent control, so set it in the registry directly. !!! warning "Two limits on relocating the file" `ATLASIniFile` is read from the registry only — it cannot be set from inside `ATLAS.ini`, because the file has not been located at the point it is needed. A small number of settings are read very early during start-up, before the relocation is applied — including `RemoteDataFeedEndpoint` and the Bridge Service log path. If you relocate `ATLAS.ini`, those two are still taken from the default location. Unless you have a reason to move the file, leave it at the default path; if you must move it, set the Bridge Service values in the registry instead. Unlike the ATLAS Viewer, **ADS does not accept an `.ini` path on the command line**. The default location and `ATLASIniFile` are the only two options. ## How overrides are applied ### Precedence For every setting ADS reads, the order is: 1. **`ATLAS.ini`** — if the section and key are present, this value is used. 2. **Registry** — otherwise the value under the ADS registry key is used. 3. **Built-in default** — otherwise the shipped default is used, and is written into the registry. A value that comes from `ATLAS.ini` is never copied into the registry, and ADS never writes back to`ATLAS.ini`. ### Settings from the file are read-only When a setting is supplied by `ATLAS.ini`, ADS marks it read-only. In `Tools > Options` it appears greyed out, and attempts to change it are discarded. This is the quickest way to confirm a key has been picked up. ### Changes require a restart `ATLAS.ini` is read once per process and cached until ADS exits. **Editing the file while ADS is running has no effect** — restart ADS to apply changes. Separately, many individual settings are themselves flagged as requiring a restart; these are marked in the [Options reference](options.md). ## File syntax `ATLAS.ini` is a standard Windows INI file, read with the Win32 private profile functions. The rules follow from that. ### Sections and keys A section name is the **registry sub-key**, relative to `HKEY_CURRENT_USER\Software\McLaren Electronic Systems\ATLAS\9.0`. A key name is the **registry value name**. So this file: ```ini [ADSAdvancedSettings] LogFileFolder = D:\ATLASLogs ``` sets the same thing as the registry value `LogFileFolder` under `HKCU\Software\McLaren Electronic Systems\ATLAS\9.0\ADSAdvancedSettings`. - Section and key names are case-insensitive. - Nested registry keys use a backslash in the section name, for example `[ADSAdvancedSettings\RecorderSettings]`. - A section header may appear only once per file. Repeating it means the later block is ignored. - Comments start with a semicolon (`;`). A hash (`#`) is **not** a comment character and will be read as part of a value. - Whitespace around `=` is trimmed, and surrounding double quotes are stripped. ### Value types | Type | How to write it | |---|---| | String | Verbatim text, up to 1024 characters. Do not quote paths. | | Number | Decimal only. Hexadecimal (`0x…`) and digit separators are not accepted. | | Boolean | `1` or `0`. | !!! warning "`true` and `false` do not work" Boolean settings must be written as `1` or `0`. The interface displays them as True/False, but the file is parsed numerically — `true` is read as `0`, silently giving you the opposite of what you intended. Save the file as **ANSI/ASCII**, not UTF-16. In PowerShell use `-Encoding ascii`; `Out-File` and `>` default to UTF-8, and `Set-Content` defaults to the system code page. ### Values that cannot be set from the file - **`-1` and `-2`.** These two values are indistinguishable from "key not present", so a setting written as `-1` is ignored and the registry or default value is used instead. Where a setting needs to be disabled, use the documented disable value (usually `0`). - **Multi-string values.** The function library lists under `ADSLibrary` (`Defaults`, `Order`, `DefaultsAttrib`) are `REG_MULTI_SZ` and have no INI representation. Set them in the registry. - **Binary values.** Recorder session details are stored as a binary blob (see [Recorder settings](#adsadvancedsettingsrecordersettings)). - **Root-level values.** A handful of values live directly under the `ATLAS\9.0` key rather than in a sub-key, and cannot be reached from the file because a section name is required. - **`HomeFolder`.** Read from the registry only. - **Environment variables.** There is **no** expansion of `%USERPROFILE%` or any other variable — write literal absolute paths. Older sample files that use `%USERPROFILE%` are incorrect. ## Sections reference ADS reads four sections. | Section | Contains | Gate | |---|---|---| | `[Preferences]` | Folder and file paths | — | | `[ADSAdvancedSettings]` | Everything in `Tools > Options` | — | | `[AtlasDataServer]` | `Setup > Configure Interfaces` network settings | `InitializeFromRegistry = 1` | | `[ADSAdvancedSettings\RecorderSettings]` | Recorder definitions | `SetupRecordersFromRegistry = 1` | ### [Preferences] Paths used across ADS. These correspond to `Tools > Options > Recorder Folders` and to folders the installer seeds. | Key | Purpose | |---|---| | `DataFolder` | Root folder for recorded session data. | | `LocalFolder` | Temporary/working folder. | | `WorkbookFolder` | Workbook folder. Also supplies the default location of the server settings file, `\AtlasServer.sbk`. | | `ConfigFolder` | Logging configuration folder. | | `CircuitsFolder` | Circuit files. | | `BitmapsFolder` | Bitmap files. | | `CustomDisplaysFolder` | Custom display definitions. | | `FunctionLibraryFolder` | Function library. | | `FunctionDataFolder` | Function data. | | `FunctionDLLFolder` | Function DLLs. | | `ExportJobFolder` | Export job definitions. | | `DefaultConstantsFile` | Default constants file. | | `SystemMonitor7BaseFolder` | System Monitor base folder. Multiple paths may be separated with semicolons. | | `AncilliaryParametersFolders` | Ancillary parameter folders, semicolon-separated. Note the spelling of this key — it contains a doubled `l`. | | `ParameterUnlockListDir` | Parameter unlock list folder. Also passed to SQL Race as its unlock list search path. | | `CustomPGVFolder` | Folder where the LDF recorder stores received PGV files. | | `DefaultSqlRaceConnection` | Default SQL Race connection. | | `ATLASIniFile` | Location of this file. **Registry only** — cannot be set here. | !!! note `HomeFolder` is deliberately excluded — it is read from the registry only and ignores this file. ### [ADSAdvancedSettings] This is the main section and covers every setting on the `Tools > Options` tabs. Because there are roughly 200 of them, they are documented with their descriptions and defaults in the **[Options reference](options.md)**, which lists the `ATLAS.ini` key for each setting alongside its label in the interface. ### [AtlasDataServer] The network interface settings from `Setup > Configure Interfaces > General`. !!! warning "Only read when `InitializeFromRegistry` is set" ADS normally loads these from the server settings workbook (`.sbk`). This section is ignored entirely unless `InitializeFromRegistry = 1`. There are also no built-in defaults here: a key you leave out is read as zero or empty, not as a sensible default. Set the whole group together. | Key | Type | Purpose | |---|---|---| | `InitializeFromRegistry` | Boolean | Must be `1` for the rest of this section to be read. | | `Stream` | String | Data server name, as seen by clients. | | `Port` | Number | Network port. The multicast port is implicitly `Port + 1`. | | `StatusPort` | Number | Status port. | | `LocalIPBind` | String | IP address of the network adapter to bind to. | | `MulticastIP` | Number | Multicast address, stored as a packed number. See below. | | `TelemTTL` | Number | Telemetry time to live, 0–10. | | `TelemMaxDataRate` | Number | Telemetry maximum data rate in KB/s, 0–15000. | | `TelemBuffer` | Number | Telemetry buffer in MB, 1–1024. | | `WireTTL` | Number | Wirelink time to live. | | `WireMaxDataRate` | Number | Wirelink maximum data rate in KB/s. | | `SendSessionConstantsToClients` | Boolean | Corresponds to `Use Server Constants` on the Library tab. | `MulticastIP` is stored as a packed number rather than dotted-quad text, and the first octet must be in the range 224–239. Rather than calculating it, set it once in `Setup > Configure Interfaces` and read the stored value back — see [Capturing a configuration](#capturing-a-configuration-from-a-configured-machine). !!! note `WireBuffer` exists as a registry value and is written when settings are saved, but the wirelink buffer is fixed at 1 MB when read. Setting it has no effect. ### [ADSAdvancedSettings\RecorderSettings] Recorder definitions, so that a machine comes up with its recorders already configured. !!! warning "Only read when `SetupRecordersFromRegistry` is set" Put `SetupRecordersFromRegistry = 1` in `[ADSAdvancedSettings]` first, otherwise recorders are loaded from the workbook and this section is ignored. With this enabled, ADS also *saves* recorders back to the registry when settings are saved. Any recorder in `ATLAS.ini` is re-applied on the next start, so recorders added through the interface will appear to be lost. | Key | Purpose | |---|---| | `RecorderCount` | Number of recorders to create. | | `RecorderType` | Recorder type, `N` starting at 1. Must exactly match one of the strings below. | | `Recorder__NumberOfSettings` | Number of sources this recorder has. See the warning below. | | `Recorder__0_` | Per-recorder properties, for example `Recorder_1_0_SessionDirectory`. | Commonly used properties include `AutoRecord`, `SessionDirectory`, `SessionName`, `SessionDestination`, `DataStream`, `DataSource`, `AppendSession`, `RecordOnlyToMemory`, `DetailsComeFrom`, `CarName`, `UnitName`, `SqlRaceConnectionIdentifier`, `RemoteDataServerIPAddress` and `RemoteDataServerPort`. `SessionName` supports the same substitutions as the Record dialog, such as `%y%m%d%H%M%S` for a timestamp or `$vCar$` for a parameter value. !!! warning "`NumberOfSettings` must match the recorder" `Recorder__NumberOfSettings` must equal the number of sources the recorder type actually has — `1` for a single-source recorder. If it does not match, the **entire** `Recorder__0_*` block is silently ignored and the recorder is created with default settings. Session detail blobs are also stored in binary form and cannot be expressed in the file, so drive naming through `SessionName`, `SessionDirectory` and `DetailsComeFrom` instead. If in doubt, configure one recorder in the interface and read the values back, as described below. #### Valid recorder type strings `RecorderType` is matched against the recorder types registered by ADS, exactly and including spaces and capitalisation. Several are only registered if the corresponding licence option is present; if the string does not match, the recorder is skipped and `Recorder Type: , not found` is written to the log. | `RecorderType` | Licence option required | |---|---| | `File Telemetry` | — | | `Generic Recorder` | — | | `Remote Data Server` | — | | `Remote Data Server Playback` | — | | `Ethernet Multi-Session Offload` | — | | `Data Server Telemetry` | — | | `VTSRecorder` | vTAG Server Recorder | | `DST Multi Source Recorder` | Wide Band | | `Generic SQLRace Recorder` | SQLRace read | | `Advanced Data Feed Recorder` | Advanced Data Feed Recorder | | `TLM-Px Multi Source Recorder` | TLM-Px Recorder | !!! note `CBR610 Multi Source Recorder` and `PiDB Multi Source Recorder` are valid strings elsewhere in ATLAS but are **not** registered as ADS recorder types. ## Worked examples ### Enable the Bridge Service The minimum file to turn on the Bridge Service and have ADS manage the process itself: ```ini ; \McLaren Electronic Systems\ATLAS 9\Atlas.ini [ADSAdvancedSettings] EnableRemoteDataFeed = 1 LocalRemoteDataFeed = 1 RemoteDataFeedEndpoint = [::1]:9697 ``` To connect to a Bridge Service running on another machine instead, point the endpoint at it and let that machine own the process: ```ini [ADSAdvancedSettings] EnableRemoteDataFeed = 1 LocalRemoteDataFeed = 0 RemoteDataFeedEndpoint = bridge-host.example.com:9697 ``` ### Relocate data and log folders Moving recording and diagnostic output onto a dedicated data volume: ```ini [Preferences] DataFolder = D:\ATLASData WorkbookFolder = D:\ATLASData\Workbooks ConfigFolder = D:\ATLASData\Config [ADSAdvancedSettings] LogFileFolder = E:\ATLASLogs SettingsFile = D:\ATLASData\Workbooks\AtlasServer.sbk MinimumDiskSpace = 2000 TempRecordingDirectory = D:\ATLASData\Temp MaxLogFileSizeMB = 50 ``` ### A complete unattended configuration Network interfaces, one auto-starting recorder and the Bridge Service, all applied on first start: ```ini ; ADS configuration - applied at start-up, read-only in Tools > Options. ; Restart ADS after editing. Save as ANSI, not UTF-16. [Preferences] DataFolder = D:\ATLASData WorkbookFolder = D:\ATLASData\Workbooks [ADSAdvancedSettings] ; --- diagnostics --- LogFileFolder = E:\ATLASLogs MaxLogFileSizeMB = 50 ; --- Bridge Service --- EnableRemoteDataFeed = 1 LocalRemoteDataFeed = 1 RemoteDataFeedEndpoint = [::1]:9697 ; --- gates for the two sections below --- SetupRecordersFromRegistry = 1 [AtlasDataServer] InitializeFromRegistry = 1 Stream = TrackADS1 Port = 5000 StatusPort = 5002 LocalIPBind = 10.20.0.5 MulticastIP = 4009754848 TelemTTL = 1 TelemMaxDataRate = 500 TelemBuffer = 20 WireTTL = 1 WireMaxDataRate = 500 [ADSAdvancedSettings\RecorderSettings] RecorderCount = 1 Recorder1Type = Data Server Telemetry Recorder_1_NumberOfSettings = 1 Recorder_1_0_AutoRecord = 1 Recorder_1_0_SessionDirectory = D:\ATLASData\Telemetry Recorder_1_0_SessionName = %y%m%d%H%M%S Recorder_1_0_SessionDestination = 0 Recorder_1_0_DetailsComeFrom = 0 ``` !!! tip Take the `MulticastIP` value from a machine you have already configured rather than computing it. The value above is an example and will not be correct for your network. ## Capturing a configuration from a configured machine The most reliable way to build an `ATLAS.ini` — particularly for recorders and `MulticastIP` — is to configure one machine through the interface, then read the values back and transcribe them. Configure ADS as required, close it so that settings are flushed, then export: ```powershell reg export "HKCU\Software\McLaren Electronic Systems\ATLAS\9.0\ADSAdvancedSettings" ads-settings.reg ``` Or list a section's values to read them directly: ```powershell reg query "HKCU\Software\McLaren Electronic Systems\ATLAS\9.0\AtlasDataServer" ``` Each value name becomes a key in the matching `ATLAS.ini` section, and each sub-key becomes a section with a backslash in its name. Remember that multi-string and binary values cannot be transcribed. ## Checking that your file was read ADS logs whether it found the file each time it starts. Look in `ATLASWideLog.txt` in the folder given by `LogFileFolder`: ``` ATLAS ini file found; C:\Users\jsmith\Documents\McLaren Electronic Systems\ATLAS 9\Atlas.ini ``` If the file was not located, the same line reports `ATLAS ini file is not found` with the path that was tried — which tells you where ADS expected it. Then confirm individual settings: open `Tools > Options` and check that the settings you supplied are greyed out. A setting that is still editable did not come from the file. ## Troubleshooting | Symptom | Likely cause | |---|---| | Nothing changed at all | ADS was not restarted, or the file is in a different account's Documents folder. Check the log line above. | | Log says the file was not found | Wrong path, wrong file name, or Documents is redirected. The logged path is the one ADS tried. | | A boolean behaves as if disabled | Written as `true`/`false` instead of `1`/`0`. | | A setting is still editable in Options | The section or key name does not match, or the value was rejected. Section names are the registry sub-key, key names the registry value name. | | A whole section is ignored | Missing gate: `InitializeFromRegistry` for `[AtlasDataServer]`, `SetupRecordersFromRegistry` for recorders. | | A section appears partly applied | The section header is duplicated in the file; only the first block is read. | | A recorder was not created | `RecorderType` does not exactly match a registered type, or its licence option is absent. The log records `Recorder Type: , not found`. | | A recorder has default settings | `Recorder__NumberOfSettings` does not match the recorder's source count, so the property block was skipped. | | Values look garbled | The file was saved as UTF-16. Save as ANSI. | | Bridge Service uses the wrong endpoint | `ATLAS.ini` was relocated with `ATLASIniFile`. Set `RemoteDataFeedEndpoint` in the registry, or move the file back to the default path. | ## Setting values in the registry instead Every key described here is a registry value under `HKEY_CURRENT_USER\Software\McLaren Electronic Systems\ATLAS\9.0`, so a deployment script can write the registry directly: ```powershell $key = 'HKCU:\Software\McLaren Electronic Systems\ATLAS\9.0\ADSAdvancedSettings' New-Item -Path $key -Force | Out-Null Set-ItemProperty -Path $key -Name 'EnableRemoteDataFeed' -Value 1 -Type DWord Set-ItemProperty -Path $key -Name 'LocalRemoteDataFeed' -Value 1 -Type DWord Set-ItemProperty -Path $key -Name 'RemoteDataFeedEndpoint' -Value '[::1]:9697' -Type String ``` Use booleans and numbers as `DWord` and text as `String`. Choose between the two approaches on this basis: | | `ATLAS.ini` | Registry | |---|---|---| | Settings remain editable in the interface | No — greyed out | Yes | | Multi-string and binary values | Not supported | Supported | | Applies per user | Yes, via that user's Documents folder | Yes, via that user's hive | | Easy to copy between machines | Yes, one file | Requires a `.reg` export or script | Because it locks values and travels as a single file, `ATLAS.ini` is the better fit for deployment. Use the registry for the few things the file cannot express, and where operators still need to be able to change a setting. !!! note "The installer does not configure ADS" The ADS installer cannot apply configuration. It is a bootstrapper that installs prerequisites and then the ADS package, and it declares no variables that would let it pass settings through — so only the standard silent-install switches (`/quiet`, `/passive`, `/log`) apply, and none of them configure ADS. Treat configuration as a separate step: install ADS, then write `ATLAS.ini` or the registry values, then start ADS. ## See also - [Options reference](options.md) — every ADS setting with its `ATLAS.ini` key, default and description - [Getting Started](installation.md) — installing and first-run setup - [Recorders](recorders.md) — what each recorder type does - [RDS](rds.md) — Remote Data Server configuration --- # RDS The Remote Data Server enables listening to another ADS server across different networks over WAN connections. This functionality is essential for distributed teams, allowing factory floor personnel to access real-time data being transmitted from remote locations. ## Configuring Remote Data Server 1. On the ADS machine, go to `Tools > Options > Recording` 2. Find `Enable Remote Data Server` and set it to `True` ![RDS_options](assets/rds_options.png) 3. Go to the `Slave ADS Connection 1` tab and Allow Connection. 4. Click `OK` to save and close the Options window. 5. On the RDS machine, go to `Setup > Add > Remote Data Server` Recorder. 6. Configure the recorder by entering the IP address and port of the ADS machine (as defined in Slave ADS Connection settings). ![RDS_recorder_setup](assets/rds_recorder.png) 7. Turn Auto-Record on if you want the Remote Data Server to receive telemetry automatically when the main ADS starts a session. 8. Click `OK` to save the recorder settings. !!! info "RDS Playback" When you enable **Sync RDS Playback File**, the system creates a playback file to recover lower-priority packets that were deprioritized during the session. This playback file can be upserted to the main session using the RDS Playback Recorder. ## Remote Data Server (RDS) Options | Setting | Description | |---------|-------------| | **Allow connection** | Select to allow a Slave ADS to connect to the local ADS | | **Port** | Port to be used by a Slave ADS | | **Auto data rate** | Automatically adjusts data rate for optimum bandwidth usage | | **Max data rate** | Maximum data rate in Kb/s (0 = unlimited) | | **Min data rate** | Minimum data rate when Auto data rate is disabled | | **Compression** | Select to apply data compression | | **Data distribution** | Eliminates data banding by evenly distributing received packets | ## Data Server Telemetry Recorder The Data Server Telemetry Recorder daisy-chains multiple data servers together by mirroring the primary Data Server stream to downstream clients. !!! tip "Networking Considerations" - **Same Network:** Can be used when all Data Servers reside on the same network - **Different Networks:** Must use Remote Data Server recorders instead of DST recorders for cross-network deployment Remote Data Server lets you listen to another ADS server on a different network over WAN. This is used by teams so that people in the factory have access to the data currently being transmitted. --- # Overview A standard, open set of rules and services for streaming telemetry data over a broker. Includes full definition of the Protobuf schema for transmitting messages, along with documentation describing the messages. !!! tip "New to Open Streaming?" See the [Getting Started Guide](../../../developer-resources/secu4/getting-started.md) to get data flowing in 30 minutes using the Docker-based quick start, or explore the [example repository](https://github.com/atlas-dev-hub/example-stream-api-kafka-setup) for a hands-on walkthrough. ![open streaming architecture](assets/open_streaming_light.png#only-light) ![open streaming architecture](assets/open_streaming_dark.png#only-dark) ## Components ### [Stream API](../../../developer-resources/secu4/stream_api/index.md) A standard API to expose streaming data from SECU units and abstract away proprietary implementation details. Consumers of this API will be able to write software that interfaces with any viewer, network protocol, or storage technology of their choice. All the data from the SECU unit (after RDA filtering) will be available on the stream. This will be integrated with a protocol for streaming engineering (calibrated) telemetry, interoperating with ATLAS clients and the surrounding data processing ecosystem. ### [Bridge Service](../../../developer-resources/secu4/bridge_service/index.md) Standalone service that decodes a raw stream of quads from an ADS, converting the data to engineering values then into the Stream Protocol via the Stream API. Will include components to manage sessions, configuration, metadata etc. All RDA protected data will be filtered out at this point, based on the active team license, so any downstream components will only receive data available to the team. ### Virtual Parameter Service Virtual parameters are defined as simple FDL functions in the ECU logging config, and are traditionally calculated on demand by clients rather than the values being persisted like other parameters. However, this requires knowledge of the proprietary FDL grammar, so a service will be developed as part of the Motion Applied Bridge Service to calculate virtual parameters in real time and written to the broker, eliminating the need for downstream clients to recalculate them. ### Multicaster Service An optional service to reduce network bandwidth in situations where a large number of clients are reading the same set of data from the same broker. Can be configured to consume specific streams of data from the broker, and forward all the packets over multicast to listening clients. ## Historical Data Storage (Parquet) In addition to the ATLAS session formats, a Parquet format will be supported for storing open engineering data directly from the stream or as an export from ATLAS. Parquet is an open-source file format for storing large datasets in an optimized way. Columnar storage allows for fast data processing and good compression rates. Parquet files can be easily accessed using a variety of programming languages and APIs, to allow teams to access their data independent of ATLAS. --- # Overview Tooling to add upon existing raw data to augment and enrich for more advanced simulation ## [vTAG Server](vtag/index.md) vTAG Server is a computation engine that runs Simulink‑based logic on live or historical telemetry to generate new derived data without relying on physical hardware. ### Key Functionality - Live & historical data processing: Replays telemetry streams or session files and executes Simulink‑compiled algorithms on them to produce new computed channels. - External computation offload: Shifts non‑critical or CPU‑heavy logic away from on‑car ECUs, reducing hardware load and enabling safer experimentation. - Data publishing into ATLAS: Outputs vData back into the ATLAS ecosystem so processed signals appear as native channels for analysis, distribution, and recording. --- # Overview vTAG Server is part of the ATLAS ecosystem and provides a way to run Simulink‑based logic on real data—either live telemetry or historical sessions—without requiring a physical ECU or embedded unit. The aim is to offload computationally expensive or non‑critical logic from hardware and to provide a safe, flexible environment for algorithm development. Many engineering teams reach a point where: - The physical unit’s CPU is at capacity. - Telemetry bandwidth limits how much data can be streamed. - Testing on real hardware introduces delays or risks. - They need to interrogate or transform existing session data without modifying the original datasets. vTAG Server solves these problems by acting as an external computation engine: it replays data, executes your Simulink logic on it, and publishes new derived parameters. This allows engineers to test ideas, optimise algorithms, or shift load off embedded devices. ## Key Features - **Data Integration**: The complete data stream is displayed in ATLAS as if all data came from the car - **Network Distribution**: One ATLAS Client typically runs the vTAGServer and supplies processed data (vData) to the VTS Data Recorder in the ADS, which distributes it across the network - **Session Processing**: In stand-alone historical use, vTAGServer processes data from a session (.ssn) file and creates vTAG data in another vTAG session (.ssv) file !!! note "What's the difference between vTAG Server and vTAG PC?" Although vTAG Server and vTAG PC use the same build mechanism, they serve different engineering goals: vTAG Server: - Processes real data (historical or live). - Acts as an external computation engine. - Suitable for validation of logic on production telemetry. - Publishes outputs into the ATLAS ecosystem. vTAG PC: - A software‑in‑the‑loop emulator. - Runs like an ECU but entirely on your computer. - Optimised for debugging and functional testing of embedded code, not replaying data. Engineers typically use vTAG PC to test behaviour, and vTAG Server to test behaviour against real data. --- # Getting Started This guide provides the fastest, simplest path to building and running your first vTAG Server application — from a blank Simulink model to live outputs in ATLAS. ## 1. Install & Verify Required Components Before you begin, ensure you have the following: **Software** - MATLAB + Simulink - System Monitor - ATLAS - ADS (Atlas Data Server) **Licenses** - MCT licence (needed for Measurement Write blocks) - MCT Atlas Toolbox licence (required for Atlas 10 Source block) - MCT vTAG HSP licence (required to build vTAG applications) !!! note To learn more about licences, please contact [sales](mailto:sales@motionapplied.com). **Download** Download the latest vTAG Server installation package from [Motion Applied Portal](https://portal.mclarenapplied.com/portal/Downloads/Software#vTAG%20Server). ## 2. Create Your First vTAG Server Model 1. Open Simulink → select **New model**. 2. In the Library Browser, find `MCT ATLAS Toolbox > ATLAS10`. 3. Add an **Atlas 10 Source block**: - Pick a parameter (e.g., `vehicle_speed`) - Select the correct originating application - You can define sample times, modes, data types, etc. as needed by clicking into the block. 4. Add simple logic — for example, a **Gain block** multiplying by 2. 5. Add a **Measurement Write block**: - Name the output parameter (e.g., `vehicle_speed_x2`) 6. Save the model. You now have the simplest valid vTAG model: ``` Input → Logic → Output ``` !!! question "What are the blocks in MCT ATLAS Toolbox?" - ATLAS 10 Source – Reads a parameter from a live ADS data stream or a historical session file and provides it as an input to your Simulink model. - ATLAS 10 Source Type – Outputs a status code indicating whether the connected data source is live, historical, or unavailable to support diagnostics or logic switching. - ATLAS 10 Control – Allows you to run a Simulink simulation driven by ATLAS while a session is loaded, effectively acting as a local replacement for vTAG Server during testing. *Not required for vTAG Server.* - ATLAS 10 Sink – Sends Simulink‑generated signals back into ATLAS during simulation so they appear as channels in the viewer.*Not required for vTAG Server.* ## 3. Build the Application (pgv) Before building, configure the model: 1. Open **Model Settings** → **Code Generation**. 2. On **System Target File** Select **vTAG TLC**. 3. Under **Build Process > Toolchain** Choose your Microsoft Visual C compiler. 4. Click **Ok**. Run the build command in MATLAB to create the PGV file: ```matlab vtag.build( ... 'modelName', 'your_model_name', ... 'Application', 9, 'PGVVersion', 'MyFirstVtagApp', ... 'SMBase', 'path/to/SystemMonitor/base' ... ); ``` For more information on the build command and its arguments type `help vtag.build` in MATLAB Command Window. After running the build command, go to the SMBase folder and find the folder generated with PGVVersion (e.g., `MyFirstVtagApp`). Inside, you will find the generated `.pgv`, `.dtv` and `.dvb` files. To learn more about projects, see [System Monitor](../../configure/system-monitor/projects/index.md) documentation. ## 4. Reprogram the vECU Now that we have the PGV file, we need to reprogram it onto the virtual unit. 1. Open **System Monitor**. 2. Create a **New Project** (`File > New > Project). 3. Select BIOS Program Version - If vtag.build didn't specify a BIOS version, the default is 4; select a 1000 BIOS Version. 4. Select all of the App IDs in the selected base and load the generated PGVs. 5. Click **Finish** 6. Connect to the **vTAG Server IP**. - The vTAG Server IP can be found in vTAG Server under `Own IP`. 7. In vTAG Server, click **Reset > Reset vECU**. 8. Reprogram the unit (F3). ## 5. Configure ADS Configure ADS: 1. Open **ADS**. 2. Add a **File Telemetry Recorder** (simulated data source). 3. Add a **VTS Recorder** (records vTAG Server output). 4. Start **ADS recording**. ![VTS Recorder](assets/vts-recorder.png) Check that **vTAG Server** shows active in System Monitor. ## 6. View in ATLAS Viewer 1. Open ATLAS Viewer and a workbook. 2. Open Session Browser (`Ctrl + L`) and add a new [Data Server Telemetry Recorder](../../analyse/viewer/handling-data/recorders/data-server.md), enable auto-record and add to set. 3. View your live output parameter streaming in. This setup lets you simulate a live telemetry environment without hardware. --- # Operating vTAG Server vTAG Server contains 6 tabs plus a bottom control section showing Process State and Progress. All vTAG Server settings are saved within the workbook. vTAG Server has the following tabs: - **Settings** – configure vTAGServer - **Status** – view processing and recording information - **Reset** – vECU control - **APP Info** – view information on APPs - **Log** – view logging output - **Control section** – operate the model and view progress ![vTAG Server](assets/vtag-server.png) ## Settings Tab ### Live Mode In Live Mode you can: - Set the IP address - Check/uncheck parameter options - Stop the model ### Historical Mode In Historical Mode you can: - Set .ssv file name - Set calculation period - Play and stop the model ### General Workflow 1. Choose Live or Historical from the Mode list. 2. Choose your own IP address (the machine running ATLAS). 3. In Historical Mode, specify a session name (part used by extension manager). If omitted, the .ssv file name matches the .ssn file. 4. Define the Calculation Period: - Session - Current Lap - Fastest Lap - All Timed Laps - Current Time Base 5. Wait for parameters up to X seconds. 6. Re‑run model on cable data – when enabled, VTS recorder records during ATLAS wire‑link upload and generates a new .ssv file. 7. Use coverage cursor (Live only) – recommended. Ensures Apps run on complete data. 8. Channel latency (up to 4s) is considered. 9. Auto reload historical run – automatically loads .ssv after generation. 10. Delay – apply constant delay without using coverage cursor. ## Status Tab Displays: - Logging Status - Memory Size (MB) - Memory Used (MB) - Parameter List – parameters created by vTAGServer ### Possible Logging States - Cleared - Cleared to Logging - Stopped - Logger Only - Telemetry Only - End of Logging Pending - Download Request - Download Pending - Download - Full Reset Request - Erase - Disabled - Reconfiguration Request - Reconfiguration ## Reset Tab - **Reset virtual ECU** – reinitialises applications, BIOS, and logging memory. - **Clear vECU** – removes PGVs and logging configuration. Use whenever programming a new PGV via System Monitor. ## APP Info Tab Shows: - Models running per APP - BIOS usage - Parameters and PGVs used ## Log Tab Displays the log, also saved as a standard log file. ### Options: - **Autoscroll** – disable to stop auto‑scroll - **Filter out trace** – filters debug information ## Control Section ### Live Mode - Start/Stop the model ### Historical Mode - Play/Stop the model - Progress bar shows processing progress - Progress may pause temporarily while searching for PGVs ### Process States - Stop - Stopped on Error - Running Live - Running after Upload - Running Historical !!! tip - The Process States and the progress are shown in the 'control' part. - When running live, no progress is shown. When running on Wire Link Upload or Historical, you can see the progress. - Note that in Historical mode the progress bar sometimes 'holds' for a few seconds. vTAGServer is then searching for the PGV's (this will be shown in the Status Bar of ATLAS Viewer) --- # Live & Historical Usage vTAG Server can operate in two ways depending on the data source: - **Live Mode** – processes parameters from an active ADS telemetry stream. - **Historical Replay Mode** – replays a previously recorded session file (SSN or SQL Race). The behaviour of your vTAG model is the same in both modes: it reads parameters via Atlas 10 Source blocks, executes the Simulink logic, and publishes outputs via Measurement Write. ## Live Usage (ADS Telemetry) When a live telemetry stream is active in ADS, the vTAG Server instance connected via VTS Recorder will begin producing output parameters in real time. ### What you will see - ATLAS will display live parameters from the telemetry stream. - Additional derived parameters from your vTAG Server model appear alongside them. - If ADS is configured with Auto‑Record, both the actual telemetry and the VTS Recorder output are recorded continuously. !!! note - vTAG Server processes samples as they arrive. - If live telemetry contains coverage gaps (network dropouts), vTAG Server cannot backfill the missing samples later. - The output during live operation will therefore include any gaps present in the incoming signal. ## Wire‑Link Upload (End‑of‑Session Offload) When telemetry coverage is imperfect, some data is only available after the wire‑link offload at the end of a run. After offload, once the complete session is available: - The system can automatically switch to Historical Replay Mode. - vTAG Server can replay the session from start to finish using the full, gap‑free dataset. - It regenerates the entire set of derived parameters using the higher‑quality data. This ensures that the simulated parameters are based on the complete dataset, not just what was available live. If ATLAS detects that additional data has arrived after live recording: - The vTAG model may be re-run over the full session with the improved dataset. - The newly generated outputs are written to the same directory as the original VTS data. - The version number (iteration counter) increments automatically to distinguish reprocessed files. ## Stand‑Alone Historical Usage (Full Session Replay) If you have access to a complete SSN or SQL Race session, you can perform a stand‑alone vTAG replay directly inside ATLAS. To run a historical replay: 1. Open ATLAS. 2. Load the session you want to analyse. 3. Select the vTAG Server layer as the data source. 4. Choose the correct vTAG Server instance (IP address under Options → Choose). 5. Define the time span if required (Current Lap, Fastest Lap, Entire Session, etc.). 6. Start playback — ATLAS will display the derived parameters from vTAG Server. This allows you to run diagnostic or engineering models using complete, offline session data without ADS or live telemetry. ## Session File Naming (VTS Recorder) The VTS Recorder generates output session files containing the vTAG‑derived parameters. If the active recorder outputs a base session name `xxx.ssn`, the VTS Recorder produces: ``` xxx.VTS.001.ssv xxx.VTS.002.ssv xxx.VTS.003.ssv ... ``` Where: - `xxx` = base session name (taken from the active recorder or the name entered in VTS Recorder). - `VTS` = the identifier for the vTAG source (configurable). - `nnn` = iteration number (starting at 001). Each time the vTAG Server performs a fresh playback (live or historical), the version number increments. When running a replay without ADS you can enter the base session name manually. Each new playback produces a new version automatically. ## Opening Multiple Sessions in ATLAS ATLAS can display actual data and vTAG‑generated data together. ### How ATLAS handles simulated session versions - ATLAS shows one column for each unique vTAG identifier (the VTS part of the filename). - It automatically finds all versions of simulated output files created for the same base session. - The highest version is selected by default. - Users can switch to earlier iterations if needed. This allows easy comparison of re‑processed sessions (e.g., live vs. offloaded vs. "clean" full replay). !!! note "Session Persistence" - vTAG Server does not remember state between sessions. - If you need multi‑session continuity (e.g., cumulative metrics), you must: - Create a synthetic "start block" (short session containing required initial values). - Stitch sessions together using SQL Race APIs. This is necessary because vTAG Server always starts a replay from zero state. --- # Overview The **Virtual Parameter Service (VPS)** simplifies the calculation of virtual parameters — parameters defined as FDL (Function Definition Language) functions in ECU logging configurations. Traditionally calculated on demand by downstream clients such as ATLAS, the VPS performs these calculations **in real time** and writes the results back to the broker, removing the need for client-side recalculation. ## Key Features | Feature | Description | |---|---| | **Real-time virtual parameter calculation** | Automatically computes FDL-defined virtual parameters as live telemetry data arrives. | | **Stream API integration** | Reads source sessions from Kafka and writes computed results back as new virtual sessions. | | **Session lifecycle management** | Detects live sessions, creates associated virtual sessions, and manages their full lifecycle. | | **Prometheus metrics** | Exposes detailed operational metrics for monitoring via Prometheus and Grafana. | | **Docker-first deployment** | Ships as a public Docker image, alongside a Windows binary for running outside Docker. | | **Configurable buffering** | Tunable buffering window and sliding window percentage to balance latency and throughput. | ## How It Works At a high level, the VPS operates as a background service that: 1. **Connects** to a Kafka broker via the Stream API. 2. **Listens** for live telemetry sessions published by ADS through the Bridge Service. 3. **Reads** configuration packets containing parameter definitions and FDL expressions. 4. **Compiles** FDL expressions into executable functions at runtime. 5. **Subscribes** to source parameter data streams and buffers incoming samples. 6. **Calculates** virtual parameter values in real time using the compiled functions. 7. **Writes** the computed results back to the Stream API as an associate session. Downstream clients can then consume these pre-calculated virtual parameters directly from the stream, instead of computing them locally. ```mermaid flowchart LR ADS["ADS (ATLAS)"] -->|Telemetry| BS["Bridge Service"] BS -->|Publishes to| K["Kafka"] K -->|Sessions & Data| VPS["Virtual Parameter Service"] VPS -->|Computed Virtuals| K K -->|Virtual Data| C["ATLAS Viewer"] ``` ## Use Cases - **Eliminating redundant calculations** — Instead of every client independently computing the same virtual parameters, the VPS computes them once and publishes the results. - **Ensuring consistency** — All consumers receive identical computed values, removing discrepancies between different client implementations. - **Reducing client load** — Downstream applications no longer need to include FDL evaluation logic, simplifying their architecture. --- # Getting Started This guide walks through deploying the Virtual Parameter Service using Docker (recommended) or running it manually. ## Prerequisites Before deploying the Virtual Parameter Service, ensure the following dependencies are in place. ### Required Software | Software | Minimum Version | Purpose | |---|---|---| | **ADS (ATLAS Data Server)**, running the [Bridge Service](../../../developer-resources/secu4/bridge_service/index.md) | `9.85.2.183` | The VPS's actual data source — Bridge Service publishes live telemetry to Kafka; ADS is what runs it | | **Docker** | Latest stable | Container runtime for deploying the service stack | | **Kafka** | Latest (via Confluent images) | Message broker for the Stream API | ### Bridge Service The VPS doesn't talk to ADS directly — it consumes live telemetry from Kafka topics published by the **Bridge Service**, which runs alongside ADS. Set up and enable Bridge Service following its own [configuration guide](../../../developer-resources/secu4/bridge_service/index.md); several of its settings need to line up with the VPS's own configuration, or the VPS will be listening in the wrong place: - **`DataSource`** — the VPS's `DataSource` setting (see [AppConfig Reference](configuration/appconfig-reference.md)) must match the DataSource name Bridge Service publishes under. - **`BrokerUrl`** — the VPS's `StreamApiConfig.BrokerUrl` must point to the same Kafka broker Bridge Service is configured to publish to. - **`StreamCreationStrategy`** — the VPS's `StreamApiConfig.StreamCreationStrategy` must match Bridge Service's (`1` = partition-based, `2` = topic-based), since it determines where in Kafka the data actually lands. - **`PartitionMappings`** — only relevant when `StreamCreationStrategy` is `1` (partition-based); the VPS's `StreamApiConfig.PartitionMappings` must match Bridge Service's mapping so the VPS reads from the same partitions Bridge Service writes to. Not used for topic-based (`2`). ### Docker Access The VPS Docker image is public, hosted on Docker Hub as [`atlasplatformdocker/virtual-parameter-service-host`](https://hub.docker.com/repository/docker/atlasplatformdocker/virtual-parameter-service-host/general). No special access is required — `docker login` is only needed to avoid Docker Hub's anonymous pull rate limits. !!! note "Can't reach Docker Hub?" If your environment can't pull from Docker Hub at all (e.g. an offline/air-gapped machine), see [Offline Install](setup-guide.md#offline-install) below. ### Network Requirements | Port | Service | Description | |---|---|---| | `9094` | Kafka | Broker listener for external clients | | `10010` | VPS Metrics | Prometheus metrics endpoint | | `13579` | Stream API | Default Stream API port (configurable) | !!! tip All ports are configurable. The above are the conventional defaults if you set up this stack yourself, matching the `AppConfig.json`/image defaults documented elsewhere on this page. ## Docker Deployment (Recommended) 1. **Create a working directory** (e.g. `C:\dev\vps`) and an `AppConfig.json` inside it. See the [AppConfig Reference](configuration/appconfig-reference.md) for the full schema — at minimum, set: - `DataSource` to match the DataSource name Bridge Service publishes under. - `StreamApiConfig.BrokerUrl` to point to the Kafka broker Bridge Service publishes to (e.g. `YOUR_MACHINE_IP:9094` — use your machine's IPv4 address, not `localhost`, if Kafka runs in Docker). 2. **Pull and run the image**: ```bash docker run -d \ --name virtual-parameter-service \ -p 10010:10010 \ -v C:\dev\vps\AppConfig.json:/config/AppConfig.json \ atlasplatformdocker/virtual-parameter-service-host:latest \ -c /config/AppConfig.json ``` See [Docker Run Command Explained](#docker-run-command-explained) below for what each flag does, or [Docker Compose Environment Variables](#docker-compose-environment-variables) if you're running it from your own `docker-compose.yaml` alongside Kafka and Bridge Service. 3. **Verify** — check the container is running: ```bash docker ps ``` ### Offline Install If your environment can't reach Docker Hub at all (e.g. an offline/air-gapped machine), export the image on a machine that can reach Docker Hub, then transfer and load it on the offline machine: ```bash # on a machine with Docker Hub access docker pull atlasplatformdocker/virtual-parameter-service-host:latest docker save atlasplatformdocker/virtual-parameter-service-host:latest -o vps-image.tar # transfer vps-image.tar to the offline machine, then: docker load --input vps-image.tar ``` Then run the container as in step 2 above. ### Docker Run Command Explained ```bash docker run -d \ --name virtual-parameter-service \ -p 10010:10010 \ -v C:\AppConfig.json:/config/AppConfig.json \ atlasplatformdocker/virtual-parameter-service-host:latest \ -c /config/AppConfig.json \ -l /logs/vps-svc-log.txt ``` | Flag | Purpose | |---|---| | `-d` | Run in detached (background) mode | | `--name` | Assign a container name for easier management | | `-p 10010:10010` | Expose the Prometheus metrics port | | `-v ...:/config/AppConfig.json` | Mount your local config file into the container | | `-c /config/AppConfig.json` | Tell the service where to find its configuration | | `-l /logs/vps-svc-log.txt` | Specify the log file path inside the container | ### Docker Compose Environment Variables Instead of command-line arguments, you can use environment variables in `docker-compose.yaml`: ```yaml virtual-parameter-service: image: atlasplatformdocker/virtual-parameter-service-host:latest ports: - "10010:10010" volumes: - ./virtual-parameter-service/AppConfig.json:/config/AppConfig.json environment: - CONFIG_FILE_PATH=/config/AppConfig.json - LOG_FILE_PATH=/logs/vps-svc-log.txt ``` See the [Environment Variables](configuration/environment-variables.md) reference for all supported variables. ## Manual Deployment For running the service outside of Docker: 1. **Build or download** the `MA.DataPlatforms.VirtualParameterService.Host` binaries. 2. **Place `AppConfig.json`** in a `Configs` folder alongside the executable: ``` MA.DataPlatforms.VirtualParameterService.Host.exe Configs/ └── AppConfig.json ``` Or specify a custom path at startup with `-c`: ```bash MA.DataPlatforms.VirtualParameterService.Host.exe -c C:\path\to\AppConfig.json ``` 3. **Run the executable**: ```bash MA.DataPlatforms.VirtualParameterService.Host.exe ``` !!! warning "Administrator privileges required" The Prometheus HTTP listener requires the application to run with elevated (Administrator) privileges. 4. **Optional: specify the log file path**: ```bash MA.DataPlatforms.VirtualParameterService.Host.exe -c C:\config\AppConfig.json -l C:\logs\vps.txt ``` If no log path is specified, logs are written to: ``` %ProgramData%\Mclaren_Applied_Ltd\VirtualParameterService\Logs\vps-svc-log.txt ``` ## Verifying the Deployment Once the service is running: 1. **Check the metrics endpoint** — Navigate to `http://YOUR_MACHINE_IP:10010/metrics` in a browser. You should see Prometheus-formatted metrics. 3. **Look for the data source gauge** — Search for `vps_running_gauge_of_data_sources` in the metrics output. Its value should match the number of data sources you configured. 4. **Optional: Prometheus and Grafana** — if you've set up your own Prometheus/Grafana instances against the VPS metrics endpoint, confirm the VPS target shows `UP` in Prometheus and query the metrics from Grafana. See the [Metrics & Prometheus](monitoring/metrics.md) and [Grafana Dashboards](monitoring/grafana.md) guides. --- # AppConfig Reference The VPS is configured via a single JSON file — `AppConfig.json`. This page documents every section and setting. ## File Location By default, the service looks for the config file at: ``` Configs/AppConfig.json ``` This path is relative to the executable or working directory. Override it using: - **Command-line argument:** `-c /path/to/AppConfig.json` - **Environment variable:** `CONFIG_FILE_PATH=/path/to/AppConfig.json` ## Full Example ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "192.168.0.6:9094", "PartitionMappings": [{}], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "BatchingResponses": false, "StreamApiPort": 13579 }, "VirtualParameterServiceConfig": { "DataSource": "Default", "BufferingWindowLengthInMs": "3000", "SlidingWindowPercentage": 5, "SessionTypesToProcess": [ "Session", "VTS" ], "MetricPort": 10010, "CalculationMode": "Default", "MaxWriteBatchSizeInKb": 100 }, "Serilog": { "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File" ], "MinimumLevel": "Debug", "WriteTo": [ { "Name": "Console" }, { "Name": "File", "Args": { "path": "C:/logs/vps-svc-log.txt" } } ] } } ``` ## VirtualParameterServiceConfig Controls the core behaviour of the virtual parameter calculation engine. | Setting | Type | Default | Description | |---|---|---|---| | `DataSource` | `string` | `"Default"` | The data source name to listen for. Only **one** data source is supported per service instance. Must match the data source configured in ADS. | | `BufferingWindowLengthInMs` | `uint` | `3000` | The time window (in milliseconds) over which incoming sample data is buffered before processing begins. Increasing this value allows more data to accumulate before calculation, which can improve accuracy at the cost of latency. | | `SlidingWindowPercentage` | `byte` | `5` | The percentage of the buffering window that controls how frequently computed data is emitted. For example, with a 3000 ms window and 5%, results are emitted approximately every 150 ms. | | `SessionTypesToProcess` | `string[]` | `["Session", "VTS"]` | The ATLAS session types the service processes. Historical sessions and the service's own virtual-session output are filtered out automatically regardless of this setting. | | `MetricPort` | `int` | `10010` | The port the Prometheus metrics endpoint listens on. | | `CalculationMode` | `string` (`Default` or `Snapshot`) | `Default` | Selects the virtual parameter calculation strategy. See [Calculation Modes](#calculation-modes) below. | | `MaxWriteBatchSizeInKb` | `uint` | `100` | Only used when `CalculationMode` is `Snapshot`. Caps the size of each packet written for a virtual — once a virtual's pending samples reach this limit within a batch, they're flushed immediately and accumulation restarts, so no single packet exceeds the limit. Set it to match the Kafka `message.max.bytes` setting. | !!! info "Single data source" The VPS only supports a single `DataSource` per instance. To process multiple data sources, deploy separate VPS instances. ## Calculation Modes `CalculationMode` controls how the VPS decides when to recalculate a virtual parameter, and what happens when a source parameter is slow, quiet, or delivers samples out of order. === "Default" Waits until every source parameter has data covering a given timestamp before calculating, so a virtual's output timestamp is only produced once all of its inputs are aligned. Timestamps that can't yet be aligned are deferred to a later buffering window. Use Default when a virtual must never be computed from an input that doesn't actually cover its output timestamp — for example, processing recorded or replayed sessions, or sources that can redeliver samples out of order. === "Snapshot" Keeps only the latest known value per source parameter. Any time a source changes, the virtual recalculates immediately using that change together with the latest known value of its other sources (forward-filling the ones that haven't changed). Nested virtuals that depend on the result cascade in the same pass. Use Snapshot for live streaming: memory use stays flat regardless of session length, and a virtual keeps producing values even while one of its sources ticks slowly. The trade-off is in the warning below. !!! warning "Snapshot mode never accepts out-of-order data" If a sample's timestamp is older than the latest value Snapshot has already recorded for that parameter, the sample is rejected rather than retroactively applied. Use `Default` mode for sources that can deliver historical or out-of-order samples. | | Default | Snapshot | |---|---|---| | **Recalculates when** | Every source for a timestamp has arrived | Any one source changes | | **Memory use** | Grows with source lag over the life of a session | Flat, independent of session length or sample rate | | **Slow or quiet sources** | Virtual stalls until the slow source catches up | Keeps producing, using the last known value | | **Out-of-order samples** | Tolerated within the buffering window | Rejected | ```json title="AppConfig.json — enabling Snapshot mode" linenums="1" "VirtualParameterServiceConfig": { "CalculationMode": "Snapshot", "MaxWriteBatchSizeInKb": 100 } ``` !!! note "Add or update — don't replace the whole file" Add these two settings to your existing `VirtualParameterServiceConfig` section. ## Serilog Controls application logging. Standard [Serilog configuration](https://github.com/serilog/serilog-settings-configuration) is supported. !!! warning "Precedence" The `Serilog` section in `AppConfig.json` takes precedence over command-line arguments (`-l`) and environment variables (`LOG_FILE_PATH`) for log file paths. ### Common Settings | Setting | Description | |---|---| | `MinimumLevel` | Log level: `Verbose`, `Debug`, `Information`, `Warning`, `Error`, `Fatal` | | `Using` | Serilog sink packages to load | | `WriteTo` | Array of configured sinks | ### Example: Console + File (Debug Level) ```json { "Serilog": { "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File" ], "MinimumLevel": "Debug", "WriteTo": [ { "Name": "Console" }, { "Name": "File", "Args": { "path": "C:/logs/vps-svc-log.txt" } } ] } } ``` ### Example: File Only (Information Level) ```json { "Serilog": { "Using": ["Serilog.Sinks.File"], "MinimumLevel": "Information", "WriteTo": [ { "Name": "File", "Args": { "path": "/logs/vps-svc-log.txt" } } ] } } ``` --- # Environment Variables The Virtual Parameter Service supports configuration via environment variables, which are particularly useful when deploying with Docker or Docker Compose. ## Supported Variables | Variable | Default | Description | |---|---|---| | `CONFIG_FILE_PATH` | `Configs/AppConfig.json` | Path to the `AppConfig.json` configuration file. | | `LOG_FILE_PATH` | `%ProgramData%\Mclaren_Applied_Ltd\VirtualParameterService\Logs\vps-svc-log.txt` | Path to the log file. Must be a full file path ending in `.txt`. | | `METRIC_PORT` | `10010` | The port on which the Prometheus metrics HTTP endpoint is exposed. | !!! warning "LOG_FILE_PATH must include a filename ending in .txt" If the path doesn't end in `.txt`, the service does **not** treat it as a target directory and append `vps-svc-log.txt` inside it. Instead: - A path with no `.txt` extension but with a parent directory (e.g. `/logs/output` or `/logs`) has its last segment dropped, and `vps-svc-log.txt` is written into what's left of the path (e.g. `/logs/output` → `/logs/vps-svc-log.txt`; `/logs` → the filesystem root). - A path with no `.txt` extension and no parent directory at all (a bare name like `logs`, relative to the working directory) is treated as a subfolder, with `vps-svc-log.txt` appended inside it. - A path with some other extension (e.g. `.log`) is rejected outright — the service logs a console warning and reverts to the default path. Always set `LOG_FILE_PATH` to a full path that ends in a `.txt` filename, e.g. `/logs/vps-svc-log.txt`. ## Precedence Order - **`CONFIG_FILE_PATH` / `LOG_FILE_PATH`:** the `-c` / `-l` command-line arguments set these environment variables for the process before they're read, so a command-line argument always wins over a manually-set environment variable, which in turn wins over the built-in default. If a `Serilog` section is present in `AppConfig.json`, it takes over logging configuration entirely and `LOG_FILE_PATH` (env var or `-l`) is ignored — see [Logging](logging.md). - **`METRIC_PORT`:** unlike the two settings above, this also has an `AppConfig.json` equivalent — `VirtualParameterServiceConfig.MetricPort` (see [AppConfig Reference](appconfig-reference.md#virtualparameterserviceconfig)). If `METRIC_PORT` is set, it overrides the `MetricPort` value from `AppConfig.json`. ## Usage with Docker Compose ```yaml virtual-parameter-service: image: atlasplatformdocker/virtual-parameter-service-host:latest ports: - "10010:10010" volumes: - ./virtual-parameter-service/AppConfig.json:/config/AppConfig.json environment: - CONFIG_FILE_PATH=/config/AppConfig.json - LOG_FILE_PATH=/logs/vps-svc-log.txt - METRIC_PORT=10010 ``` ## Usage with Docker Run ```bash docker run -d \ --name virtual-parameter-service \ -p 10010:10010 \ -e CONFIG_FILE_PATH=/config/AppConfig.json \ -e LOG_FILE_PATH=/logs/vps-svc-log.txt \ -e METRIC_PORT=10010 \ -v C:\path\to\AppConfig.json:/config/AppConfig.json \ atlasplatformdocker/virtual-parameter-service-host:latest ``` !!! tip "Environment variables vs command arguments" When using Docker Compose, environment variables are generally preferred over command arguments (`-c`, `-l`) because only one `command` directive can be specified in the compose file. --- # Logging The Virtual Parameter Service uses [Serilog](https://serilog.net/) for structured logging and the standard ASP.NET Core logging framework for host-level messages. ## Application Logging (Serilog) Application logs capture the operational activity of the VPS — session lifecycle events, parameter processing, errors, and diagnostics. ### Configuration Serilog is configured via the `Serilog` section in `AppConfig.json`. See the [AppConfig Reference](appconfig-reference.md#serilog) for the full schema. ### Log Levels | Level | When to Use | |---|---| | `Verbose` | Very detailed tracing information. Use only for deep debugging. | | `Debug` | Internal system events useful for diagnosing problems. | | `Information` | General operational events (session started, session ended). | | `Warning` | Unexpected but recoverable situations. | | `Error` | Failures that prevent part of the system from functioning. | | `Fatal` | Critical failures that cause the service to stop. | !!! tip "Recommended production level" Use `Information` for production deployments. Switch to `Debug` when investigating issues. ### Default Log File Location When no log path is configured, logs are written to: === "Windows" ``` C:\ProgramData\Mclaren_Applied_Ltd\VirtualParameterService\Logs\vps-svc-log.txt ``` === "Docker" Specify explicitly via environment variable or command argument: ``` /logs/vps-svc-log.txt ``` ## Host Logging (ASP.NET Core) The underlying .NET host produces its own log messages (startup diagnostics, hosting lifecycle events). These are configured separately via `appsettings.json`. ### Reducing Host Log Noise By default, `appsettings.json` is configured to minimise host log output: ```json { "Logging": { "LogLevel": { "Default": "Warning", "Microsoft.Hosting.Lifetime": "Warning", "Microsoft.AspNetCore.Hosting.Diagnostics": "None" } } } ``` This prevents the host framework from flooding the console with startup and request-handling messages, keeping the output focused on VPS-specific logs. !!! info `appsettings.json` is located in the root directory alongside the executable. It is separate from `AppConfig.json`. --- # Troubleshooting & FAQ Common issues and their solutions when working with the Virtual Parameter Service. ## Common Issues ### VPS container starts but no sessions are detected ??? note "Checklist" 1. **Is ADS running?** Ensure ADS is active and streaming live data. 2. **Is the Bridge Service running?** Verify the Bridge Service container is healthy: `docker ps`. 3. **Is the Bridge Service configured correctly?** See its own [configuration guide](../../../developer-resources/secu4/bridge_service/index.md) for enabling it and setting `BrokerUrl`. 4. **Is the `DataSource` correct?** The VPS's `DataSource` (`AppConfig.json`) must match the DataSource name Bridge Service publishes under. 5. **Is Kafka reachable?** The `BrokerUrl` in `AppConfig.json` must point to a valid, running Kafka broker — the same one Bridge Service is configured to publish to. 6. **Do `StreamCreationStrategy` and `PartitionMappings` match Bridge Service?** If `StreamApiConfig.StreamCreationStrategy` doesn't match Bridge Service's setting, the VPS is looking for data in the wrong place (topics vs. partitions). If it's `1` (partition-based), `PartitionMappings` must match Bridge Service's mapping too — it's ignored for `2` (topic-based). ### Kafka is not initialised in time before the Bridge Service This is a **known issue** when starting the full Docker Compose stack. The Bridge Service may start before Kafka is fully ready. ??? success "Solution" Restart the `bridge-service` and `virtual-parameter-service` containers manually: ```bash docker restart bridge-service docker restart virtual-parameter-service ``` Alternatively, add health checks and `depends_on` conditions to your `docker-compose.yaml`. ### Prometheus target shows DOWN ??? note "Checklist" 1. **Is `prometheus.yml` configured correctly?** The target must be your machine's IPv4 address, not `localhost`. 2. **Is the VPS running and healthy?** Check `docker ps` for the container status. 3. **Is port 10010 exposed?** Confirm the port mapping with `docker ps` or test with `curl http://YOUR_IP:10010/metrics`. 4. **Firewall?** Ensure port `10010` is not blocked by a firewall rule. ### Metrics endpoint returns an error when running manually The Prometheus HTTP listener requires **Administrator privileges** when running outside Docker. ??? success "Solution" Run `MA.DataPlatforms.VirtualParameterService.Host.exe` as Administrator. ### Virtual parameter definitions are failing to build Check the `vps_virtual_parameter_definition_failed_counter` metric for the count of failures. ??? note "Possible causes" - **Missing source parameters** — The FDL expression references parameters that are not present in the session. - **Invalid FDL expressions** — The expression syntax is incorrect or unsupported. - **Dependency resolution failure** — A virtual parameter depends on another virtual that has not yet been defined. Check the service logs (set `MinimumLevel` to `Debug` in the Serilog config) for detailed error messages about which definitions failed and why. ### Incomplete virtual parameter definitions (gauge is non-zero) The `vps_virtual_parameter_definition_incomplete_gauge` indicates virtual definitions waiting for their source dependencies to be defined. ??? note "Explanation" This is normal during session startup — virtual parameters that depend on other virtuals may be temporarily incomplete until all definitions are received. The gauge should return to `0` once all configuration packets have been processed. If it remains non-zero indefinitely, check that all required source parameter definitions are being published in the session's configuration packets. ### A virtual parameter's values stop appearing after a source replays old data (Snapshot mode) ??? success "Explanation" In `Snapshot` calculation mode, a source sample older than the latest value already recorded for that parameter is rejected rather than applied retroactively — this is expected behaviour, not a bug. If your source can redeliver historical or out-of-order samples (e.g. replaying a recorded session), use `CalculationMode: "Default"` instead. See [Calculation Modes](configuration/appconfig-reference.md#calculation-modes). ### Log file is not being created ??? note "Checklist" 1. **Does the Serilog config override the path?** The `Serilog` section in `AppConfig.json` takes precedence over `-l` arguments and `LOG_FILE_PATH` environment variables. 2. **Is the path valid?** The log file must end in `.txt`. If an invalid extension is provided, the service reverts to the default path. 3. **Does the directory exist?** The directory must exist — the service does not create intermediate directories. 4. **Permissions?** The service process must have write access to the target directory. ## FAQ ### Can I run multiple VPS instances for different data sources? Yes. Each VPS instance supports a single `DataSource`. To process multiple data sources, deploy separate instances with different `AppConfig.json` files, each specifying a different `DataSource` value. Ensure each instance uses a unique `StreamApiPort` and `METRIC_PORT`. ### Which calculation mode should I use? Use `Snapshot` for live streaming — it keeps memory flat over long-running sessions and keeps producing values even when a source ticks slowly. Use `Default` for replayed/recorded sessions, or any source that can deliver samples out of order, since `Snapshot` rejects out-of-order samples. See [Calculation Modes](configuration/appconfig-reference.md#calculation-modes) for the full comparison. ### Does the VPS process historical sessions? No. The VPS only processes **live** sessions. Historical sessions and sessions of type `VirtualSession` are automatically filtered out to avoid reprocessing its own output. ### What happens if the VPS goes down during a live session? The VPS does not persist state between restarts. If the service restarts, it will detect any active live sessions and begin processing them from that point forward. Data generated during the downtime will not be retroactively computed. ### Can I change the buffering window while the service is running? No. Configuration is loaded at startup. To change `BufferingWindowLengthInMs` or `SlidingWindowPercentage`, update `AppConfig.json` and restart the service. ### What Kafka topics does the VPS use? The VPS uses the Stream API to manage topic creation and data flow. Topics are created automatically based on the configured `StreamCreationStrategy`. You do not need to manually create Kafka topics. ### How do I verify that virtual data is being written correctly? 1. **Check metrics** — Ensure `vps_virtual_parameter_packet_info_packager_packaged_counter` is incrementing. --- # Overview Visualise, compare, and understand high‑frequency telemetry — live or post‑run. ## [ATLAS Viewer](viewer/index.md) ATLAS Viewer provides a powerful environment for inspecting high‑resolution time‑series data. Built for motorsport engineers, it enables confident, rapid decision‑making through rich visualisation tools and flexible customisation. ### Key Functionality - Advanced visualisation and layout customisation - Alarms for rapid event recognition - Optimised live data viewing - MATLAB integration - Compare‑set analysis - 19 built‑in display types - Functions and virtual channels - Secure and reliable data handling --- # Overview The Standalone Stream Recorder is a Windows console application that listens to live telemetry published through the [Stream API](../../../developer-resources/secu4/stream_api/index.md) and saves each session to disk, without needing ATLAS Viewer to act as the recorder. ## At a glance | | | |---|---| | **Application** | `MA.DataPlatforms.DataRecorder.Host` | | **Platform** | Windows (x64) | | **Input** | A Kafka broker carrying Stream API data | | **Output** | SSN2 files, or a SQL Race database (SQLite or SQL Server) | | **Configuration file** | `Config/AppConfig.json`, next to the executable | | **Logs** | Console and daily rolling files in `logs/` (default) | ## What it does When the recorder starts it connects to the Kafka broker, waits for sessions published to the configured data source, and records each one as it arrives. When a session ends it waits for the next one. ## Choose your output format The first decision is `WritingConfig.SessionFormat`: | Format | Use it when | Also needs | |---|---|---| | `SSN2` | You want one SSN2 file per session in a folder. | `WritingConfig.RecordingFolder` | | `Database` | You want sessions stored in a SQL Race database, and want to load live sessions through the recorder's Server Listener. | `SqlRaceConfig` (connection string, engine, data source, delete option) | ## The three things you must configure 1. **Where to read from** — `StreamApiConfig` (`BrokerUrl`, `StreamCreationStrategy`) and `StreamReadingConfig.DataSource`. These must match the producer's Stream API settings. 2. **Where to write to** — `WritingConfig` and, for `Database`, `SqlRaceConfig`. 3. **Which sessions to record** — `StreamReadingConfig.SessionIdentifierPattern` and `RecordVpsSessions`. ## Next steps - [Getting Started](getting-started.md) — run the recorder for the first time - [Configuration Guide](configuration-guide.md) — task-based recipes - [Configuration Reference](configuration-reference.md) — every field - [Troubleshooting](troubleshooting.md) --- # Getting Started This page takes you from a fresh install to a recorded session. ## Before you begin - A Windows machine to run the recorder. - A Kafka broker that a Stream API producer (for example the [Bridge Service](../../../developer-resources/secu4/bridge_service/index.md)) is publishing to. - The producer's **Stream Creation Strategy**, **broker address** and **data source name**. The recorder must use the same values. See [Server Configuration](../../../developer-resources/secu4/stream_api/reference_docs/configuration/server-config.md). ## 1. Edit AppConfig.json The recorder reads `Config/AppConfig.json` from the folder containing the executable. This is the shipped file, which records to SSN2 files and works as a starting point: ```json title="AppConfig.json" linenums="1" { "$schema": "./AppConfig.schema.json", "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "localhost:9094", "PartitionMappings": [{}], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "BatchingResponses": false, "StreamApiPort": 13579, "Domain": "" }, "RecordVpsSessions": false, "WritingConfig": { "SessionFormat": "SSN2", "RecordingFolder": "C:\\Standalone Stream Recorder\\SSN2", "UseTemporaryRecordingFolder": false, "TemporaryRecordingFolder": "", "UseStreamApiSessionIdentifier": true, "UseStreamApiSessionDetails": true }, "StreamReadingConfig": { "ReadingMode": "Live", "DataSource": "Default", "SessionIdentifierPattern": "*", "GroupId": "" }, "SqlRaceConfig": { "ConnectionString": "DbEngine=SQLite;Data Source=C:\\Standalone Stream Recorder\\stream_recorder.ssndb;PRAGMA journal_mode=WAL;", "DbEngine": "SQLite", "DataSource": "C:\\Standalone Stream Recorder\\stream_recorder.ssndb", "DeleteSessionOnClose": "NoSessionDelete", "ServerListenerAddress": "127.0.0.1:7300" }, "MetricsConfig": { "Port": 10015, "IsEnabled": false }, "Serilog": { "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File" ], "MinimumLevel": "Information", "WriteTo": [ { "Name": "Console" }, { "Name": "File", "Args": { "path": "./logs/standalone-stream-recorder-.txt", "rollingInterval": "Day", "fileSizeLimitBytes": 10485760, "rollOnFileSizeLimit": true } } ] } } ``` Change at least `StreamApiConfig.BrokerUrl`, `StreamApiConfig.StreamCreationStrategy` and `StreamReadingConfig.DataSource` to match your producer, and `WritingConfig.RecordingFolder` to a folder you want. !!! note "Add or update — don't replace the whole file" To record to a SQL Race database instead of SSN2 files, change only `WritingConfig.SessionFormat` to `"Database"` and make sure the `SqlRaceConfig` values are right for your database. The shipped `SqlRaceConfig` already points at a SQLite file. See the [Configuration Guide](configuration-guide.md#record-to-a-database). ## 2. Start the recorder Run `MA.DataPlatforms.DataRecorder.Host.exe`. It takes no command-line arguments; all settings come from `AppConfig.json` (and environment variables, see below). If the configuration is invalid, the recorder logs each problem and does not start. See [Troubleshooting](troubleshooting.md#the-recorder-exits-straight-away). ## 3. Publish a session Start a session from your Stream API producer on the configured data source. The recorder logs to the console and to `logs/standalone-stream-recorder-.txt`. ## 4. Stop the recorder Press Ctrl+C in the console window. ## Overriding settings with environment variables Environment variables are read after `AppConfig.json`, so they override it. Use a double underscore for nesting, for example `WritingConfig__SessionFormat`. --- # Configuration Guide Task-based recipes for `Config/AppConfig.json`. Each recipe shows only the settings that change; for every field see the [Configuration Reference](configuration-reference.md). !!! note "Add or update — don't replace the whole file" The snippets below are deltas on the shipped `AppConfig.json` shown in [Getting Started](getting-started.md). ## Connect to my Kafka broker Use this when your producer's Stream API uses a different broker or stream layout from the sample. The recorder must match the producer. ```json title="AppConfig.json (StreamApiConfig)" linenums="1" "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "localhost:9094", "PartitionMappings": [{}] } ``` - `StreamCreationStrategy` is `1` (`PartitionBased`) or `2` (`TopicBased`); the names also work. - `PartitionMappings` is only used with `PartitionBased`; each entry needs a `Stream` name and a `Partition` number. - If Kafka uses SASL or SSL, see [Kafka Security](../../../developer-resources/secu4/stream_api/reference_docs/configuration/kafka-security.md). The remaining `StreamApiConfig` fields are documented in [Server Configuration](../../../developer-resources/secu4/stream_api/reference_docs/configuration/server-config.md#configuration-properties). ## Record to SSN2 files Use this when you want one SSN2 file per session. ```json title="AppConfig.json (WritingConfig)" linenums="1" "WritingConfig": { "SessionFormat": "SSN2", "RecordingFolder": "C:\\Standalone Stream Recorder\\SSN2", "UseTemporaryRecordingFolder": false, "UseStreamApiSessionIdentifier": true, "UseStreamApiSessionDetails": true } ``` `RecordingFolder` is required for SSN2. It may contain wildcards such as `%r`, `%c`, `%y%m%d` or `$Param$`, which are resolved for each session. ### Write to a temporary folder first Use this when the final folder is on a slow or shared drive. The file is written to the temporary folder while the session is live, then moved to `RecordingFolder` when it closes. ```json title="AppConfig.json (WritingConfig)" linenums="1" hl_lines="3 4" "WritingConfig": { "SessionFormat": "SSN2", "UseTemporaryRecordingFolder": true, "TemporaryRecordingFolder": "C:\\Standalone Stream Recorder\\Temp", "RecordingFolder": "C:\\Standalone Stream Recorder\\SSN2", "UseStreamApiSessionIdentifier": true, "UseStreamApiSessionDetails": true } ``` `TemporaryRecordingFolder` is required when `UseTemporaryRecordingFolder` is `true`. ## Record to a database Use this when you want sessions in a SQL Race database, or want to load live sessions from the recorder. === "SQLite" ```json title="AppConfig.json" linenums="1" "WritingConfig": { "SessionFormat": "Database", "UseTemporaryRecordingFolder": false, "UseStreamApiSessionIdentifier": true, "UseStreamApiSessionDetails": true }, "SqlRaceConfig": { "ConnectionString": "DbEngine=SQLite;Data Source=C:\\Standalone Stream Recorder\\stream_recorder.ssndb;PRAGMA journal_mode=WAL;", "DbEngine": "SQLite", "DataSource": "C:\\Standalone Stream Recorder\\stream_recorder.ssndb", "DeleteSessionOnClose": "NoSessionDelete", "ServerListenerAddress": "127.0.0.1:7300" } ``` === "SQL Server" ```json title="AppConfig.json" linenums="1" "WritingConfig": { "SessionFormat": "Database", "UseTemporaryRecordingFolder": false, "UseStreamApiSessionIdentifier": true, "UseStreamApiSessionDetails": true }, "SqlRaceConfig": { "ConnectionString": "server=\\;Initial Catalog=;Trusted_Connection=True;", "DbEngine": "SQLServer", "DataSource": "\\", "DeleteSessionOnClose": "NoSessionDelete", "ServerListenerAddress": "127.0.0.1:7300" } ``` Replace the `<...>` placeholders with your SQL Server name, instance and SQL Race database. This example uses Windows authentication (`Trusted_Connection=True`); use whichever SQL Server connection string your SQL Race database requires. With `Database`, `ConnectionString`, `DbEngine`, `DataSource` and `DeleteSessionOnClose` must all be set, otherwise the recorder will not start. `DbEngine` accepts `SQLite` or `SQLServer`, in any letter case. The recorder only starts its Server Listener when `SessionFormat` is `Database`. ## Change session identifier or details Use this when the producer's session identifier or details are not what you want stored. ```json title="AppConfig.json (WritingConfig)" linenums="1" hl_lines="2 3 4 5 6 7" "WritingConfig": { "UseStreamApiSessionIdentifier": false, "SessionIdentifierText": "%y%m%d%H%M%S", "UseStreamApiSessionDetails": false, "SessionDetails": { "MyDetailName": "MyDetailValue" }, "SessionFormat": "SSN2", "RecordingFolder": "C:\\Standalone Stream Recorder\\SSN2", "UseTemporaryRecordingFolder": false } ``` - `SessionIdentifierText` is required when `UseStreamApiSessionIdentifier` is `false`, and may contain date wildcards. - `SessionDetails` is a set of text key/value pairs applied when `UseStreamApiSessionDetails` is `false`. `MyDetailName` and `MyDetailValue` are placeholders — use your own. ## Record only some sessions Use `StreamReadingConfig.SessionIdentifierPattern` to filter by session identifier; `*` records everything. ```json title="AppConfig.json (StreamReadingConfig)" linenums="1" "StreamReadingConfig": { "ReadingMode": "Live", "DataSource": "Default", "SessionIdentifierPattern": "*" } ``` `DataSource` must match the data source name set on the producer. When the producer is the Bridge Service, this is the name of the ADS. ## Catchup on missed data Set `ReadingMode` to `LiveWithCatchUp` to read live data and also catch up on data missed earlier. With `Live` only live data is read. ## Resume recording after a disconnect Set `StreamReadingConfig.GroupId` so the recorder can continue from where it left off if it disconnects for any reason. Restart the recorder with the same `GroupId` to resume. It may contain only letters, digits, underscores and hyphens. ```json title="AppConfig.json (StreamReadingConfig)" linenums="1" hl_lines="5" "StreamReadingConfig": { "ReadingMode": "Live", "DataSource": "Default", "SessionIdentifierPattern": "*", "GroupId": "my-recorder" } ``` ## Record VPS sessions Set the top-level `"RecordVpsSessions": true`. It defaults to `false`. ## Change the Prometheus metrics port Use this when the default metrics port clashes with another service. The recorder always exposes Prometheus metrics on `Port`. ```json title="AppConfig.json (MetricsConfig)" linenums="1" "MetricsConfig": { "Port": 10015, "IsEnabled": false } ``` `Port` must be between 1 and 65535. `IsEnabled` is ignored in this release, so the endpoint is available on `Port` whether it is `true` or `false`. !!! warning "Administrator rights" On Windows the metrics endpoint may need the recorder to run as Administrator. If it does not have access, the recorder logs that it is continuing without Prometheus. ## Change logging Logging is configured by the `Serilog` section. The shipped file logs to the console and to a daily rolling file `./logs/standalone-stream-recorder-.txt` (10 MB limit, rolls on size). If the section is removed, the recorder logs at `Information` level to the console and to `logs/standalone-stream-recorder-.txt`, rolling daily. --- # Configuration Reference Every setting in `Config/AppConfig.json`. For task-based examples see the [Configuration Guide](configuration-guide.md). The shipped `Config` folder includes `AppConfig.schema.json`; the `$schema` entry at the top of `AppConfig.json` lets editors validate and auto-complete the file. Enum settings accept either the name or the number. The sections `StreamApiConfig`, `WritingConfig`, `StreamReadingConfig`, `SqlRaceConfig` and `MetricsConfig` must be present. `Serilog` is optional. ## Top level | Key | Type | Default | Description | |---|---|---|---| | `RecordVpsSessions` | bool | `false` | Record VPS sessions. | ## StreamApiConfig Connection settings for the Stream API. The recorder's key settings are below. | Key | Type | Notes | |---|---|---| | `StreamCreationStrategy` | `PartitionBased` (1) / `TopicBased` (2) | Must match the producer. | | `BrokerUrl` | string | Kafka broker address, for example `localhost:9094`. | | `PartitionMappings` | array of `{ "Stream": string, "Partition": int }` | Only used with `PartitionBased`. | All other `StreamApiConfig` settings, including `Security` and the Kafka broker tuning file paths, are described in the [Stream API Server Configuration properties](../../../developer-resources/secu4/stream_api/reference_docs/configuration/server-config.md#configuration-properties). See also [Kafka Security](../../../developer-resources/secu4/stream_api/reference_docs/configuration/kafka-security.md) and [Kafka Broker Tuning](../../../developer-resources/secu4/stream_api/reference_docs/configuration/kafka-broker-tuning.md). ## WritingConfig | Key | Type | Default | Description | |---|---|---|---| | `SessionFormat` | `Database` (0) / `SSN2` (1) | `Database` | Format sessions are recorded in. **Required.** | | `UseStreamApiSessionIdentifier` | bool | — | Use the session identifier from the Stream API. **Required.** | | `SessionIdentifierText` | string | empty | Identifier to use instead. Required when `UseStreamApiSessionIdentifier` is `false`; may contain wildcards such as `%y%m%d%H%M%S`. | | `UseStreamApiSessionDetails` | bool | — | Use session details from the Stream API. **Required.** | | `SessionDetails` | object of strings | empty | Details to apply when `UseStreamApiSessionDetails` is `false`. | | `RecordingFolder` | string | empty | Final folder for SSN2 files. Required for `SSN2`; ignored for `Database`. May contain wildcards (`%r`, `%c`, `%y%m%d`, `$Param$`). | | `UseTemporaryRecordingFolder` | bool | — | Write SSN2 files to `TemporaryRecordingFolder` while live, then move them to `RecordingFolder`. **Required.** | | `TemporaryRecordingFolder` | string | empty | Required when `UseTemporaryRecordingFolder` is `true` and the format is `SSN2`. | ## StreamReadingConfig | Key | Type | Default | Description | |---|---|---|---| | `ReadingMode` | `Live` (0) / `LiveWithCatchUp` (1) | `Live` | `Live` reads live data only; `LiveWithCatchUp` also catches up on missed data. **Required.** | | `DataSource` | string | — | Data source to read from, for example `Default`. **Required.** | | `SessionIdentifierPattern` | string | — | Which sessions to read, for example `*`. **Required.** | | `GroupId` | string | empty | Group id used to consume the data, so the recorder can continue where it left off after a disconnect. Letters, digits, `_` and `-` only. | ## SqlRaceConfig Only used when `WritingConfig.SessionFormat` is `Database`. With `SSN2` the section must still be present but may be empty. | Key | Type | Default | Description | |---|---|---|---| | `ConnectionString` | string | — | SQL Race connection string. Required for `Database`. | | `DbEngine` | `SQLite` / `SQLServer` | — | Case-insensitive. Required for `Database`. | | `DataSource` | string | — | File path (SQLite) or server address (SQL Server). Required for `Database`. | | `DeleteSessionOnClose` | `NoSessionDelete` (0) / `RecordedSessions` (1) / `SQLiteDatabase` (2) | — | Whether to delete the session from the database when the recorder closes or new sessions appear. Required for `Database`. | | `ServerListenerAddress` | string `host:port` | — | Address the Server Listener listens on, used to load live sessions from the recorder. Shipped value: `127.0.0.1:7300`. Only started for `Database`. | ## MetricsConfig | Key | Type | Default | Description | |---|---|---|---| | `Port` | int, 1–65535 | `10110` | Port for Prometheus metrics. The shipped file uses `10015`. | | `IsEnabled` | bool | `false` | Ignored in this release; the recorder starts the Prometheus endpoint regardless of this value. | !!! note "Metrics endpoint" Control the metrics endpoint with `Port` only. `IsEnabled` does not turn it on or off. ## Serilog Standard [Serilog configuration](https://github.com/serilog/serilog-settings-configuration). The shipped file uses the Console and File sinks with `MinimumLevel` `Information`; the file sink uses `rollingInterval` `Day`, `fileSizeLimitBytes` `10485760` and `rollOnFileSizeLimit` `true`. ## Validation rules The recorder checks the file at startup and does not start if any rule fails: - A required section is missing. - `UseStreamApiSessionIdentifier`, `UseStreamApiSessionDetails`, `UseTemporaryRecordingFolder` or `SessionFormat` is not set. - `SessionFormat` is `SSN2` and `RecordingFolder` is empty, or `UseTemporaryRecordingFolder` is `true` and `TemporaryRecordingFolder` is empty. - `UseStreamApiSessionIdentifier` is `false` and `SessionIdentifierText` is empty. - `SessionFormat` is `Database` and `ConnectionString`, `DbEngine`, `DataSource` or `DeleteSessionOnClose` is not set. - `DbEngine` is not `SQLite` or `SQLServer`. - `ServerListenerAddress` is set to a non-empty value that does not match `host:port`. An empty or non-IP value does not stop startup; see [Troubleshooting](troubleshooting.md). - `MetricsConfig.Port` is outside 1–65535. - `GroupId` contains characters other than letters, digits, `_` or `-`. --- # Troubleshooting ## The recorder exits straight away **Symptom:** the console logs `Invalid AppConfig.json configuration: ...` followed by `AppConfig.json contains invalid configuration. See the log for details. The application will not start.` **Cause:** a [validation rule](configuration-reference.md#validation-rules) failed. Each message names the setting, for example: - `SqlRaceConfig section is missing from AppConfig.json.` - `WritingConfig.RecordingFolder must be set when WritingConfig.SessionFormat is SSN2.` - `SqlRaceConfig.ConnectionString must be set when WritingConfig.SessionFormat is Database.` - `SqlRaceConfig.DbEngine must be either 'SQLite' or 'SQLServer'.` - `MetricsConfig.Port must be between 1 and 65535.` - `GroupId must only contain alphanumeric characters, underscores, or hyphens.` **Fix:** correct the named setting and restart. Keep `"$schema": "./AppConfig.schema.json"` at the top of the file to get editor warnings before you run. ## "Server Listener will NOT be started" **Symptom:** the log contains `Failed to parse ServerListenerAddress '...'. Server Listener will NOT be started.` **Cause:** `SqlRaceConfig.ServerListenerAddress` is empty, or it is not an IP address and port. A hostname such as `localhost:7300` is not accepted here. The recorder keeps running without the Server Listener. **Fix:** use an IP address and port, for example `127.0.0.1:7300`. A value that is not in `host:port` form at all stops startup instead; see [Validation rules](configuration-reference.md#validation-rules). ## No Prometheus metrics **Symptom:** the log says `Need Administrator privilege to run Prometheus. continue without Prometheus`. **Cause:** Windows denied the recorder permission to listen on the metrics port. **Fix:** run the recorder as Administrator, or do without metrics. ## Nothing is recorded Check that the recorder matches the producer: - `StreamApiConfig.BrokerUrl` and `StreamCreationStrategy` are the same as the producer's. - `StreamReadingConfig.DataSource` is the producer's data source name. - `StreamReadingConfig.SessionIdentifierPattern` matches the session identifier (`*` matches everything). - For VPS sessions, `RecordVpsSessions` is `true`. The log file (`logs/standalone-stream-recorder-.txt` by default) shows what the recorder is doing. --- # Overview The ATLAS Viewer is a core component of the ATLAS ecosystem, designed for the visualisation and analysis of telemetry data. It enables users to interact with both live and historical data streams, offering real-time insights and deep analytical capabilities tailored for motorsport and other telemetry-intensive applications. Key features include: - **Live Telemetry Access**: Monitor performance data in real time, whether trackside or remotely. - **Customised Analysis**: Analyse data specific to individual vehicles or subsystems. - **Historical Replay**: Revisit and study past sessions using stored telemetry files. - **Flexible Integration**: Supports a wide range of data formats including proprietary (.ssn2), SQL Race databases, CSV, MATLAB, and Parquet. - **Automation Support**: Through the ATLAS Automation API, users can automate session loading, display configuration, and parameter management using .NET-compatible languages like C#, and MATLAB. The client operates as a plugin-based composite application, allowing extensibility and integration with custom display modules. It is optimised for performance and supports service-oriented architecture principles to streamline user workflows. For more information on how to create your own plugins read the [Display API Documentation](../../../developer-resources/atlas/display-api/index.md) --- # Getting Started ## Launching ATLAS To start ATLAS, if installed in the default location: * Click Start > All Programs > McLaren Applied Technologies> ATLAS A new Workbook is opened with a single empty Page by default. ![ATLAS Workbook](assets/intro.png) ## Adding a Licence Use the Licence Manager utility to display and modify licence information. Licence codes may be added after ATLAS has been installed. The default location on the Start menu is: All Programs > McLaren Applied Technologies > Licence Manager. ![Licence Manager](assets/licence_manager.png) ## Using ATLAS The Main Window of ATLAS is displayed when you start the program and open a Workbook. It is highly customisable but has a number of key features by default as shown below. ![Main Window](assets/main-window.png) ### [Workbooks and Pages](system-operation/workbook.md) Workbooks in ATLAS define the layout and positioning of elements on the screen. Each Workbook contains one or more Pages, and each Page can host multiple Displays. Workbooks are saved as `.wbkx` files, which store the formatting, size, and position of Displays, allowing users to create customized workspaces. !!! note Data files such as Compare Sets, Session data, Bit Config Files, Functions files, Alarm Files, Circuit Display Definition files, SurfaceMap Display Definition files, and PCU Dash Display RAW CAN files are not stored in `.wbkx` files but are automatically opened by the Workbook. If these files are unavailable, Displays will appear empty. When ATLAS starts or a new Workbook is opened, it includes one Page by default. Users can select only one Page at a time, but multiple Pages can be viewed by floating and arranging them within a Workbook. Each Display on a Page can show data from different Compare Sets. The Page Bar, located at the top of the Main Window, displays tabs for each Page in the Workbook. If there are too many tabs to fit, horizontal scroll arrows appear. Pages can be selected by clicking on a tab or using the Page Up/Page Down keys. Right-clicking a Page tab opens a menu with options to: - **Set:** Switch or remove the Compare Set associated with the Page. Each Set is color-coded. - **No Association:** Remove the link to the current Compare Set. - **Lock to Current Association:** Lock the Page to the current Compare Set. - **Title:** Edit the Page name. - **Page Colour:** Underline the tab with a selected color. - **New:** Add a new Page. - **Paste:** Paste a copied Display to the Page. - **Import Display:** Import a saved Display to the Page. - **Duplicate:** Duplicate the Page and its Displays within the Workbook. - **Export:** Export the Page for use in other Workbooks. - **Close:** Close the Page tab. - **Close Others:** Close all other Page tabs. - **Close Tab Group:** Close all Pages. - **Float:** Float the Page into a movable window. - **Float Page Group:** Float the Page Group into a movable window. - **Pin Tab:** Toggle the pin status of the tab. - **New Horizontal Tab Group:** Add a horizontal Page tab group. - **New Vertical Tab Group:** Add a vertical Page tab group. ### [Sessions](handling-data/sessions/index.md) Use the Session Browser (*CTRL + L*) to record a new session or load an existing one. The Session Browser allows you to define different sets of data to use across the Workbook. Sessions, either Live or Historical, may be combined into a single set using Compare or Append mode. Add historical data to the set from the Sources tab or directly from files. The preview pane displays detailed session information to help identify the session you need. Access live data via Recorders. A Recorder streams raw telemetry data, and several types are available depending on the data source, protocol, and system architecture. ![Session Browser](assets/session_browser.png) ### [Displays](displays/index.md) Once data has been added, begin building your Workbook by adding Displays to Pages. Multiple types of Displays are available, each of which can be customised to meet specific requirements. Displays may be docked in various layouts to optimise visibility across the Workbook. For detailed information on types of Displays, shortcuts and different display options refer to the embedded documentation: Help > View Help (Preview) (*F1*). ### [Parameters](handling-data/parameters/index.md) Use the Parameter Browser (*P*) to select one or more parameters to display. Double-click a parameter or drag and drop it onto the Display. To remove a parameter, click on it and press *DELETE*. Once parameters are visible in the Display, customise the visual presentation using the Display Properties (*D*) panel. Key display options are also available by right-clicking anywhere on the Display. ![Workbook Example](assets/final_workbook.png) --- # Mouse Actions A mouse with a scroll-wheel is recommended, though many actions have [Keyboard Shortcuts](keyboard.md).
![Mouse scroll wheel](assets/mouse_3_button.gif){class="mouse-img"} Use the scroll wheel to zoom in and out on a Waveform Display or the Bing Map Display.
![Left click](assets/mouse_left.gif){class="mouse-img"} Left click to select menu items and click buttons.
![Right click](assets/mouse_right.gif){class="mouse-img"} Right click to display context sensitive (pop-up) menus where available.
![Hold left and right mouse buttons](assets/mouse_left_right.gif){class="mouse-img"} Hold both buttons and move the mouse left or right to scroll a Waveform Display left or right.
![Left click](assets/mouse_left_hold.gif){class="mouse-img"} Left click, hold and drag to scroll the time reference indicator on a Timeline in the direction of the mouse movement. This action can also be used to drag and drop parameters.
![Right click](assets/mouse_right_hold.gif){class="mouse-img"} Right click, hold and drag to select a section of a waveform which you can then choose to zoom in on by selecting Zoom from the pop-up menu. --- # Keyboard Shortcuts ## Main Window | Shortcut | Description | | ------------------------------------------------------------|---------------------------------------------------------------------------------------- | | ++ctrl+l++ | Open Session Browser | | ++p++ or ++insert++ | Open Parameter Browser | | ++ctrl+q++ | Quick-Access Assistant | | ++ctrl+n++ | New workbook page | | ++page-down++ | Next workbook page | | ++page-up++ | Previous workbook page | | ++ctrl+tab++ | Cycle between displays (forwards) | | ++ctrl+shift+tab++ | Cycle between displays (backwards) | | ++ctrl+enter++ | Toggles full screen mode | | ++ctrl+shift+r++ | Rebuild all functions | | ++ctrl+e++ | Show the Data Export Tool | ### Saving and Loading | Shortcut | Description | | ------------------------------------------------------------|---------------------------------------------------------------------------------------- | | ++ctrl+o++ | Open a workbook | | ++ctrl+s++ | Save a workbook | | ++ctrl+alt+s++ | Save a workbook as... | | ++ctrl+shift+c++ | Copies a Session Reference to the clipboard | | ++ctrl+shift+l++ | Load a Session Reference from the clipboard | ### Quick-Access | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++ctrl+q++ | Cycle between Quick-Access features | | ++down++ | Select the next Quick-Access item | | ++up++ | Select the previous Quick-Access item | | ++enter++ | Select the current Quick-Access item | | ++escape++ | Close the Quick-Access Assistant | ### Associating Compare Sets | Shortcut | Description | | ------------------------------------------------------------------|---------------------------------------------------------------------------------------- | | ++shift+1++ … ++shift+9++ | Select the Compare Set for the _active display_ | | ++shift+0++ | Select the Compare Set for the _active display_ (set 10*) | | ++shift++ + ++minus++ | Select the Compare Set for the _active display_ (set 11) | | ++shift++ + ++equal++ | Select the Compare Set for the _active display_ (set 12) | | ++shift+backspace++ | Select the Compare Set for the _active display_ (set 13) | | ++shift++ + ++semicolon++ | Select the Compare Set for the _active display_ (set 14) | | ++shift++ + ++single-quote++ | Select the Compare Set for the _active display_ (set 15) | | ++shift++ + ++comma++ | Select the Compare Set for the _active display_ (set 16) | | ++shift++ + ++period++ | Select the Compare Set for the _active display_ (set 17) | | ++shift++ + ++slash++ | Select the Compare Set for the _active display_ (set 18) | | ++shift+tab++ | Select the Compare Set for the _active display_ (set 19) | | ++shift+capital++ | Select the Compare Set for the _active display_ (set 20) | ++ctrl+1++ … ++ctrl+9++ | Select the Compare Set for all displays on the _active page_ | ++ctrl++ + same order of keyboard characters listed above | Select the Compare Set for all displays on the _active page_ (sets 10-20) | | ++ctrl+f1++ … ++ctrl+f9++ | Select the Compare Set for all displays in the _active page group_ | ++ctrl+shift+1++ … ++ctrl+shift+9++ | Select the Compare Set for _all displays_ | ++ctrl+shift++ + same order of keyboard characters listed above | Select the Compare Set for _all displays_ (sets 10-20) | | ++shift+delete++ | Clears the Compare Set for the _active display_ | | ++ctrl+delete++ | Clears the Compare Set for all displays on the _active page_ | | ++ctrl+shift+delete++ or ++ctrl+alt+shift+delete++ | Clears the Compare Set for all displays in the active _page group_ | ## Timeline and Navigation | Shortcut | Description | | ----------------------------------------------------------- | --------------------------------------------------------------------------------------- | | ++plus++ | Zoom in | | ++minus++ | Zoom out | | ++ctrl+shift+f++ | Select all laps | | ++ctrl+shift+t++ | Select all timed laps (excludes in/out laps) | | ++home++ | Move to start of session | | ++end++ | Move to end of session | | ++right++ | Move to next sample timestamp | | ++left++ | Move to previous sample timestamp | | ++ctrl+shift+right++ | Move to next lap | | ++ctrl+shift+left++ | Move to previous lap | | ++ctrl+f++ | Move to fastest lap | | ++ctrl+bracket-right++ | Go to next occurrence of event | | ++ctrl+bracket-left++ | Go to previous occurrence of event | | ++ctrl+alt+right++ | Go to next signal change | | ++ctrl+alt+left++ | Go to previous signal change | ## Displays | Shortcut | Description | | ------------------------------------------------------------|---------------------------------------------------------------------------------------- | | ++ctrl+q++ | Quick-Access Parameter or Function | | ++p++ or ++insert++ | Insert Parameter | | ++del++ | Delete Parameter | | ++d++ | Show the Display Properties | | ++ctrl+tab++ | Cycle between displays (forwards) | | ++ctrl+shift+tab++ | Cycle between displays (backwards) | | ++ctrl+f10++ | Toggles the current titlebar on/off | | ++ctrl+f11++ | Hides all titlebars | | ++ctrl+f12++ | Shows all titlebars | ### Circuit Display | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++ctrl+alt+t++ | Generate circuit from the current lap | ### Histogram Display | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++k++ | Cycle the X axis mapping instance | | ++ctrl+shift+k++ | Set the X axis mapping instance for all displays | | ++ctrl+k++ | Set the X axis mapping instance for all page displays | ### Numeric Display | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++ctrl+c++ | Copy to Clipboard | | ++ctrl+a++ | Select All | ### Scatterplot Display | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++r++ | Reference cursor | | ++h++ | Hide/show cursor | | ++t++ | Hide/show parameter trace | | ++plus++ | Zoom in | | ++minus++ | Zoom out | | ++ctrl+z++ | Undo zoom | | ++ctrl+alt+z++ | Undo all zooms | | ++ctrl+y++ | Redo zoom | | ++ctrl+p++ | Print | | ++ctrl+shift+p++ | Print Preview | ### Waveform Display | Shortcut | Description | | ------------------------------------------------------------|-----------------------------------------------------------------------------------------| | ++r++ | Reference cursor | | ++c++ | Cross-hair cursor | | ++g++ | Gradient cursor | | ++v++ | Value cursor | | ++h++ | Hide cursor | | ++t++ | Hide/show parameter trace | | ++ctrl+a++ | Selects legend items | | ++ctrl+c++ | Copies selected legend items to clipboard | | ++k++ | Cycle the X axis mapping instance | | ++ctrl+shift+k++ | Set the X axis mapping instance for all displays | | ++ctrl+k++ | Set the X axis mapping instance for all page displays | | ++f++ | Set zero time reference for TimeDiff | | ++e++ | Show delta statistics | | ++ctrl+t++ | Show the display docking area | | ++s++ | Show statistics | | ++m++ | Show minimum statistics | | ++x++ | Show maximum statistics | | ++n++ | Show mean statistics | | ++q++ | Show standard deviation statistics | | ++plus++ | Zoom in | | ++minus++ | Zoom out | | ++ctrl+z++ | Undo zoom | | ++ctrl+alt+z++ | Undo all zooms | | ++z++ | Zoom box | | ++escape++ | Cancel zoom box | | ++ctrl+p++ | Print | | ++ctrl+shift+p++ | Print Preview | ### Known issue (Windows 11) The command to switch a page group to the 10th compare set (++ctrl+shift+0++) may not work on some Windows 11 systems due to a system-level language hotkey using the same combination. To restore the shortcut, disable the language hotkeys: - Open Windows Settings > Typing > Advanced keyboard settings. - Click "Input language hot keys". - Select "Between input languages" or "Keyboard layout" and click "Change Key Sequence". - Set "Switch keyboard layout" to "Not assigned". This change removes the OS-level binding that can block ++ctrl+shift+0++, allowing the application shortcut to work as expected. Also to note, for switching to set 15, on US keyboards you'll need to use the # key instead of ++single-quote++. --- # Options SessionState Options Screen The Options screen lets you view and change various ATLAS settings. These settings apply at a system level and are stored with the ATLAS installation, not on a Workbook basis. To view the Options screen select Tools > Options. ![Options](assets/options.png) ## General | Option Name | Description | Area | Default/Notes | |-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|------------------------------------------------------------------------------------------------| | Function Folder(s) | Specifies the directory where all user-defined Functions are loaded from and saved to. You can edit this path directly or browse to a folder. Changing this affects where ATLAS looks for and stores custom function scripts, which are used for data processing or automation. | Functions | C:\Users\USERNAME\Documents\McLaren Electronic Systems\SQL Race\Functions | | Constants Folder Location | Sets the folder for loading and saving Constants, which are reusable values or parameters used in calculations or scripts. Like the Functions folder, this can be edited or browsed to. Ensures all constants are centralised for easier management and sharing. | Functions | C:\Users\USERNAME\Documents\McLaren Applied Technologies\ATLAS 10\Constants | | Resolve without App Name | When enabled, allows parameters to be loaded from sources that use different Application Groups, so you can see and use channels even if the application names don’t match exactly. Useful for working with data from multiple sources or legacy systems. | Data | Off by default | | Parameter Properties in Compare mode| Determines how parameter properties (such as units, scaling, etc.) are inherited when comparing multiple sessions:
• **Primary Session**: Secondary sessions inherit properties from the primary session.
• **User Defined**: Local changes to secondary session properties are retained.
• **Default Data**: Properties stored within each session are loaded. Useful for ensuring consistency or flexibility in data comparison. | Data | User selects mode | | Move to Next/Previous Signal Change Parameter | Selects the parameter whose value changes are used by the Move to Next/Previous Signal Change actions (toolbar and Ctrl+Alt+Left/Right). | Data | `NGear:Chassis` | | Recent Workbook Max Count | Sets the maximum number of recent Workbooks shown in the File > Recent Workbooks menu, making it easier to quickly reopen frequently used files. | Workbook | 15 | | Load Sessions with Workbooks | If enabled, when you open a Workbook, all sessions that were loaded when it was last closed are automatically reloaded. This helps restore your previous working context. | Workbook | Checkbox | | Open Last Workbook when Starting | When enabled, ATLAS will automatically reopen the last opened or saved Workbook at startup, streamlining workflow continuity. | Workbook | Checkbox | | Auto Fit Laps | In Compare Sets with multiple laps, this option aligns the start and end of each lap on the X-axis in the Waveform Display, making it easier to visually compare laps of different durations. | Workbook | Checkbox | | Auto Fit Limit % | Sets the maximum percentage difference between laps that Auto Fit will stretch data to match. Prevents excessive distortion when aligning laps. Default is 5%. | Workbook | 5% | | Load Recorders From | Determines whether saved Recorder configurations are loaded from the Workbook (unique to each file) or from a Local Configuration file (shared across all Workbooks). Useful for managing how recording setups are reused. | Workbook | LocalConfiguration | | Remove unused recorders from Workbook| If enabled, removes any Recorders when saving the Workbook that are not used in any Sets. Helps keep the Workbook tidy of unused old Recorders. | Workbook | Checkbox | | Remove set when last session unloaded| If enabled, deletes the current set when the last session is removed from it (except the final set). Helps keep the workspace tidy by removing empty sets. | Workbook | Checkbox | | Workbook Folder Enabled | Allows you to specify a dedicated folder for storing Workbook files, improving organisation and backup management. | Workbook | Checkbox | | Workbook Backups Enabled | Enables automatic, regular backups of your Workbook files to a specified folder, providing data recovery in case of corruption or accidental changes. | Workbook | Checkbox | | Workbook Backups Folder | Sets the directory where Workbook backup files are stored. You can edit or browse to this folder. Ensures backups are kept in a known, safe location. | Workbook | C:\Users\USERNAME\Documents\McLaren Applied Technologies\ATLAS 10\Automatic Workbook Backups | | Number of backups to keep per workbook| Specifies how many backup copies of each Workbook are retained. When the limit is exceeded, the oldest backups are deleted. Helps manage disk space while ensuring recovery options. | Workbook | 5 | | Waveform Data Request | Sets the frequency (in Hz) at which data is requested for Waveform displays. Higher rates provide more real-time updates but may use more resources. | Performance | 0.5–10 Hz (default 5 Hz) | | Backfill Request | Determines the time window (in seconds) for backfilling gaps in live telemetry data on the Waveform. Ensures continuity in data display during brief interruptions. | Performance | 1–10 sec (default 3 sec) | | Statistics Data Request | Sets the update rate for statistics (Lap, Display, Session, Reference). Options are Low, Medium, or High, balancing performance and responsiveness. | Performance | Medium | | Waveform Display Refresh | Controls how often the Waveform Display is refreshed (in Hz). Higher rates give smoother updates but may impact performance. | Performance | 10–20 Hz (default 15 Hz) | | Scatterplot Data Request | Sets the frequency (in Hz) for requesting data for Scatterplot displays. Balances update speed and resource usage. | Performance | 0.5–5 Hz (default 2.5 Hz) | | Scatterplot Display Refresh | Determines how often the Scatterplot Display is refreshed (in Hz). | Performance | 10–20 Hz (default 10 Hz) | | Renderer | Selects the rendering mode for ATLAS:
• **Hardware Accelerated**: Uses GPU for best performance.
• **Balanced**: Mix of performance and compatibility.
• **Software**: CPU rendering, recommended for virtual machines or systems without suitable GPU. | Performance | Hardware Accelerated/Balanced/Software | | Tcp Port Number | Sets the TCP port for the Automation API server, which allows external clients to connect and control ATLAS programmatically. Change if the default port is in use. | Automation Service | 9089 | | Authorization Port Number | For users with an RTA License, sets the OAuth 2.0 port for authentication. Adjust if needed for network or security requirements. | Service Connections | 59511 | | Tab Side | Changes which side of the screen the workbook page tabs are displayed (e.g., top, bottom, left, right), allowing for custom workspace layouts. | Appearance | - | | Tab Overflow | Determines how workbook page tabs behave when there are too many to fit on the screen:
• **Menu**: Overflow tabs in a dropdown.
• **Shrink**: Shrinks tabs to fit.
• **ShrinkWithMenu**: Shrinks tabs and uses a menu for overflow.
• **Scroll**: Adds scroll arrows.
• **ScrollWithMenu**: Combines scroll and menu. | Appearance | - | | Extension | Defines the file extension for text files imported as sessions (e.g., .csv, .txt, .asc). Must include the dot. | Text Session Format | - | | Total Header Line | Sets the number of header lines in the imported text file. Must be at least one; extra lines can contain metadata or comments. | Text Session Format | - | | Parameter Line | Specifies the line number where parameter definitions start in the text file. | Text Session Format | - | | Group Line | Indicates the line number for parameter groups. Set to 0 if not used. | Text Session Format | 0 if not used | | Date Exists | Specifies whether each data line includes a date column. If enabled, the first column is the date, second is the timestamp. | Text Session Format | - | | Time Format | Sets the format for timestamps in imported text files (e.g., DateTime, Milliseconds, Seconds, Time). | Text Session Format | - | | Delimiter | Sets the character used to separate columns in the text file (Tab, Space, Comma, Semicolon). | Text Session Format | - | | Unit | Specifies the line number where units are defined in the text file. | Text Session Format | - | | Lap Marker | Sets the parameter name used to identify laps in the text file. When the value changes, ATLAS creates a lap marker. | Text Session Format | - | | Mapping Mode | Lets you manage the mapping mode for the X-axis in Waveform Displays (e.g., Time or Distance). Plugins can add new modes. You can reorder, add, or remove mapping plugins. | X Axis Mapping | Controlled by plugins | | Crash Report Folder | Sets the directory where crash reports are saved. You can edit or browse to this folder. Used for diagnostics and troubleshooting. | Crash Reports | C:\Users\USERNAME\Documents\McLaren Applied Technologies\ATLAS 10\Crash Reports | | Number of crash reports to keep | Specifies how many crash reports are retained in the folder. Oldest reports are deleted when the limit is exceeded. | Crash Reports | 10 | | Watchdog TCP Port | Sets the TCP port for the ATLAS Watchdog, which monitors application health and can generate crash reports. | Crash Reports | 9092 | ## Diagnostics | Option Name | Description | Area | Default/Notes | |-------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------| | Application Insights (Allow collection of Metrics and Error data) | When enabled, this option allows ATLAS to share performance metrics and logged exceptions with McLaren Applied for product improvement and troubleshooting. No personally identifiable information (PII) is shared. This helps McLaren Applied monitor application health and usage trends, but preserves user privacy. | Metrics | Off by default. Toggle as needed for support or analytics. | | ATLAS Logging Level | Sets the level of detail for log messages written to ATLAS log files. Each level includes all levels below it:
• **Trace**: Extremely detailed logs, including code execution paths and algorithm steps. Intended for development or deep troubleshooting.
• **Debug**: Detailed logs for debugging, including variable states and configuration.
• **Info**: Logs significant events indicating normal operation (e.g., service start/stop).
• **Warn**: Logs events that could become issues if not addressed (e.g., high resource use).
• **Error**: Logs significant problems that affect operations but do not stop the application (e.g., failed API calls).
• **Off**: Disables logging entirely. | Logging | Default is Info. Trace is only recommended for support/development. | | Logfile Path | Specifies the folder where ATLAS log files are stored. You can set this to any accessible directory. Useful for organising logs or directing them to a monitored location for support. | Logging | User-defined. | | Max Archive Files | Sets the maximum number of archived log files to keep alongside the current active log file. When the limit is reached, the oldest archives are deleted. This helps manage disk space while retaining recent log history for troubleshooting. | Logging | Default is 9. | | Archive Above Size | Sets the file size threshold (in bytes) at which a log file is archived and a new log file is started. Prevents log files from growing too large. Default is 10MB (10,485,760 bytes). | Logging | Default is 10MB (10485760 bytes). | | SQL Race Logging Level | Sets the logging level specifically for SQL Race data, using the same levels as ATLAS Logging Level (Trace, Debug, Info, Warn, Error, Off). Allows for separate control of logging detail for SQL Race operations, which may be useful for diagnosing database-related issues. | Logging | Default is Info. Trace is only recommended for support/development. | ## Plugins | Option Name | Description | Area | Default / Limits / Notes | |-----------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Alarm library | Folder location of the alarm library used by ATLAS. | Alarms | Default: `C:\Users\[username]\Documents\McLaren Applied Technologies\ATLAS 10\Alarms`. Editable path. | | Additional Alarm File | Path to an extra alarm definition file to load alongside the main alarm library. | Alarms | Editable file path. | | Alarm Popup Mode | Controls how alarm pop‑ups are shown: **Per Alarm**, **Per Occurrence**, or **Shared**. | Alarms | Choose one of the three modes. | | Delayed Range | Sets the look‑back time window for alarms “behind the leading edge.” | Alarms | Default 3 s; min 2 s; max 5 min. **Note:** Increasing above default can degrade live telemetry performance. | | Should Auto‑Refresh | Enables/disables automatic refresh of the Session Browser’s session list. | Session Browser | Disabled by default. | | Auto Refresh Interval (seconds) | Interval, in seconds, for automatic Session Browser refresh when enabled. | Session Browser | Default 300 s. | | Filters Display Limit | Maximum number of filter results returned in Session Browser. | Session Browser | Default 100. | | Session Query Result Limit (Sessions) | Maximum number of sessions returned when databases/folders are selected in Session Browser. | Session Browser | Default 1000. | | Circuits Location | Directory containing circuit definition files (`*.ckt`/`*.cktx`). | Circuit Display | Editable directory; example shown `T:\Circuits`. | | X‑Axis Mapping | Which X‑axis mapping the Circuit Display uses (e.g., time/distance); mappings are configured under **Options → X Axis Mapping**. | Circuit Display | Example shown: `DistanceModeMapping (sLap)`. | | Circuit Location Parameter Name | Parameter used to define circuit location. | Circuit Display | Default: `NEventLocation`. | | End of Straight Parameter | Parameter used for “End of Straight” calculations. | Circuit Display | Default: `vCar:Chassis`. | | Global Circuit Definition | Choose a specific global circuit definition from the Circuits location, or allow automatic selection on session load. | Circuit Display | Auto/manual selection. | | Circuit Generator — Parameters | Parameters the generator uses to infer circuit size/shape. | Circuit Display | Defaults: `vCar:Chassis` (speed) and `gLat:Chassis` (lateral‑G). | | Circuit Generator — Shape/Invert/Rotation | Controls for the generated circuit’s shape (e.g., circle/figure eight/open), inversion, rotation. | Circuit Display | Editable controls. | | Circuit Generator — Generation Settings | Gain and Offset applied to generated circuits. | Circuit Display | Tunable numeric values. | | Bit Configuration File | File path to the Bit Display configuration file used by the Bit Display plugin. | Bit Display | Editable file path. | | Maximum Number of Parameters | Caps the number of parameters that can be added to a Scatterplot Display. | Scatterplot Display | Recommended: **24**. Increasing may cause performance issues. | | Session Details Tooltip Limit | Max number of session‑detail properties that can be selected/saved for the F1 Timeline’s Session Details tooltip. | F1 Timeline | Default 5; min 0; max 20. | | Snapshots Directory | Directory where Summary Display snapshot images (from the Snapshot generator) are stored. | Summary Display | Editable directory. | | Hide Windows cursor when dragging waveform | Hides the Windows cursor while left‑click + dragging the time/distance cursor on the Waveform Display. | Waveform Display | Toggle option. | | Enable Grid Scrolling | Waveform gridlines scroll with the leading edge during live or historical replay. | Waveform Display | Toggle option. | | Event Tracking Mask | Selects an event for use with Next/Previous event keyboard shortcuts on the Waveform. | Waveform Display | Choose event parameter. | | Maximum Number of Parameters | Caps parameters that can be added to a Waveform Display. | Waveform Display | Recommended: **100**. Increasing may cause performance issues. | | Date / Data Notation | For sessions spanning multiple days, controls how the date appears in the Waveform title bar (e.g., “Day 1” vs `DD/MM/YYYY`). | Waveform Display | Example default shown: `DayNumber`. | | Context Menu & Key Shortcut Statistic Level | Sets whether Right‑click and keyboard‑shortcut statistics apply at **Lap**, **Session**, or **Display** level. | Waveform Display | Example default shown: `Lap`. | | Override … for Reference Cursor | Allows the **Reference Cursor** level (applied with `R`) to override the chosen Context Menu/Shortcut statistics level. | Waveform Display | Toggle option. | | Raw Files Path | Folder path for RAW files used by the PCU Dash Display. | PCU Dash Display | Example shown: `T:\PCU8`. | | LED Status Light parameter name | Parameter used by PCU Dash for the LED Status Light. | PCU Dash Display | Example shown: `NLEDStatusAll`. | | Auto map CAN parameters (by name) | Automatically maps CAN parameters by matching names, simplifying setup. | PCU Dash Display | Toggle option. | | CAN Bus Identifier | Identifier number for the CAN bus used by the PCU Dash Display. | PCU Dash Display | Example shown: `7`. | ## Recorders | Option Name | Description | Area | Default / Notes | |-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|----------------------------------------------------------------------------------------------------------| | Server Listener IP | IP address for the SQLRace database server listener, used for remote access and live data recording. | SQLRace | Default: 127.0.0.1 | | Server Listener Port | Port number for the SQLRace database server listener. | SQLRace | Default: 7381 | | Data Server buffer size (MB) | Size of the storage buffer in the client for incoming data. Larger values help prevent data loss if network communications are poor. | Recording | Default: 20MB | | Data Server connection timeout (ms) | Time ATLAS waits for data from a data server before stopping recording. | Recording | Default: 20000ms | | Data Server retries enabled | Allows the client to retry loading data from the network if enabled. | Recording | Default: Unchecked | | Data Servers | List of ATLAS Data Server names and corresponding ports (format: {Server:Port}). | Recording | Default: Empty | | Server auto-refresh frequency (ms) | Interval between refreshing the list of current data servers (0 disables). | Recording | Default: 2000ms | | Server auto-refresh reply timeout (ms) | Time before a data server is deemed offline if no activity is detected. | Recording | Default: 100ms | | Server Reply Timeout (ms) | Time ATLAS waits for a reply from a data server. | Recording | Default: 3000ms | | Timeout for searching for Data Servers (ms)| Time allowed to find data servers. | Recording | Default: 3000ms | | Wide Band local address | Specifies which network card to use for sending/receiving data; shows all available IP addresses. | Recording | Default: ANY | | Auto-Dismiss Config Search Box | Automatically closes the config search box if checked. | General | Default: Checked | | Auto Increment Run Number | Automatically increments the session number for sequential recordings (e.g., R1, R2, ...). | General | Default: Unchecked | | Auto Record Connection Check Interval (ms)| Time ATLAS waits before stopping automatic recording. | General | Default: 1000ms | | Default disabled 'Still Processing Parameters' to continue | If enabled, ATLAS will auto-select continue/cancel if a recording is running when closing and the dialog is disabled. | General | Default: Unchecked | | DTV offload exclusion list | Comma-separated list of DTVs to exclude from offload at end of recording. Supports formats: application-name, unit-name.application-name, unit-name.* | General | Default: Blank | | DTV rename cached files | If checked, cached DTV files not uploaded will include "_Cached" in the filename. | General | Default: Unchecked | | Enable 'still processing parameters' dialog| Warns if parameter processing from config files is incomplete when recording ends prematurely. | General | Default: Checked | | Enable FTP Download | Enables offloading PGVs from FTP server as part of Ethernet Multi-Session Offload. | General | Default: Unchecked | | Enable Logging CAN Data | Enables CAN data logging in live recordings. | General | Default: Unchecked | | Enable Pre-build Configuration | Enables fast start by ignoring pre-cached PGVs. | General | Default: Unchecked | | Low Disk Space Warning Threshold (MB) | Warns when free disk space drops within this threshold of the minimum disk space. | General | Default: 100MB | | Minimum Disk Space (MB) | Minimum free disk space required during recording. Recording stops if below this value. | General | Default: 500MB (minimum allowed) | | PGV exclusion list | Comma-separated list of PGVs to exclude from processing at start of recording. Same formats as DTV exclusion. | General | Default: Blank | | PGV inclusion list | Comma-separated list of PGVs to include in processing at start of recording. Takes priority over exclusion list. | General | Default: Blank | | PGV stand-alone list (HEX) | List of stand-alone PGVs in hexadecimal format. | General | Default: Blank | | Polling Frequency to check available space (s)| How often to check available disk space. | General | Default: 5 seconds | | Quad sources excluded from lap trigger | Streams (comma-separated) to exclude from lap trigger detection. | General | Default: Blank | | Record only LIVE data | If checked, recording stops as soon as telemetry is uploaded. Requires "Enable Update Progress on clients" to be checked. | General | Default: Unchecked | | Record telemetry data from start | If checked, session includes data from start of Data Server recording; otherwise, only from when ATLAS recording starts. | General | Default: Unchecked | | Recording event script | Script executed on recording events (LIVE_STARTED, LIVE_STOPPED, OFFLOAD_STARTED, OFFLOAD_STOPPED). Format: 'script.exe EventId Argument'. | General | Default: Blank | | Separate vTAG Session | Splits incoming DST recording with master and VTS data into two sessions. | General | Default: Unchecked | | Temporary directory for recording files | Directory for temporary storage of session files during recording. If blank, files are written directly to data path. | General | Default: Blank | | Unload unused PGVs on start recording | Removes PGVs from cache when recording starts. | General | Default: Unchecked | | Virtual memory block size (KB) | Virtual memory slot size for memory mapping. | General | ATLAS: 5120KB, ADS: 1024KB | | Warn When Config is Missing | Warns if logging config is missing for loaded session. | General | Default: Checked | | Wideband minimum lap time | Minimum allowed time between lap triggers (>0s). | General | Default: 10 seconds | | Live data feed retrieval path | Directory for storing intermediate LDR files when using LDF. | Live Data Feed | Default: `:\Users\\Documents\My Documents\McLaren Electronic Systems\ATLAS 9\LDS\` | | PGV request timeout (s) | Max time to wait for requested PGV files via LDF. | Live Data Feed | Default: 30s | | Log file folder | Directory for log files. | Logging | Default: `:\Users\\Documents\McLaren Electronic Systems\ATLAS 9\log` | | Message log file size (MB) | Log file size limit; new file created when limit reached. 0MB = unlimited. | Logging | Default: 0MB | | Message log level (for debugging) | Logging level for developer/debugging diagnostics. | Logging | Default: 'Not Logging' | | Raw data logging | Saves raw data received during recording for telemetry diagnostics (.raw and .raw_tm files). | Logging | Default: Unchecked | | Raw data root folder | Default folder for raw data logging files. In addition, if the session folder path in the recorder is not blank then it will also append that session folder name on to the default raw data root folder. e.g. Raw data root folder = C:\Server\Data and the recorder's Session folder is C:\Recorder then the path used will be C:\Server\Data\Recorder | Logging | Default: Blank | | Sequence number anti-wrap time (hours) | Threshold for unwrapping sequence numbers if packet time is much earlier than latest. | Telemetry | Default: 0 (Disabled) | | Telemetry guard filter length | Synchronisation length for telemetry guard. | Telemetry | Default: 10 | | Telemetry guard time (s) | Rejects data if timestamp difference between packets exceeds this value. | Telemetry | Default: 0 seconds | | Unit Comms Default Timeout (ms) | Default timeout for unit communications. | Telemetry | Default: 200ms | | Ethernet buffer size (MB) | Local cache size for Direct Ethernet Wirelink. | Telemetry | Default: 5MB | | Ethernet offload connection retries | Number of times to retry reconnecting during offload if connection lost. | Telemetry | Default: 10 | | Ethernet receive timeout (ms) | Time to wait before retrying a receive message. | Telemetry | Default: 500ms | | Ethernet retries | Number of times to retry a message before aborting and displaying a warning. | Telemetry | Default: 1 | | Ethernet send timeout (ms) | Time to wait before retrying a send message. | Telemetry | Default: 500ms | | Ethernet upload config unique response | Must match client-specific BIOS configurations. | Telemetry | Default: Unchecked | | Ethernet upload message extended timeout (ms)| Time before Quad Ethernet Telemetry offload times out. | Telemetry | Default: 500ms | | Ethernet upload message interval timeout (ms)| Time before Quad Ethernet Telemetry data is re-requested during upload. Auto-extends if data is received outside this timeout. | Telemetry | Default: 3ms | | System Monitor base folder | Folder for System Monitor files. Multiple paths allowed (separated by semicolon). | System Monitor Folders | Editable | | Logging configuration folder | Folder for logging configuration files (must be ATLAS\Config\Logging\AutoConfig). Multiple paths allowed. | System Monitor Folders | Editable | | Car definitions file | Location and filename for car definitions, which define network communication for cars. | System Monitor Folders | Editable | | Ancillary parameters folder | Folder for Logging Configurations created in System Monitor. Controls which parameters are logged and how. | System Monitor Folders | Editable | | Parameter Unlock List folder | Folder where ATLAS searches for Parameter Unlock Lists. | System Monitor Folders | Editable | | Custom PGV folder | Folder where ATLAS stores received PGV files. | System Monitor Folders | Editable | | Configuration Folder | Folder for MAT.Atlas.TCPRecorder if using TCP listener recorder. | Folders | Editable | | Default Data Path | Default path where sessions are stored unless overridden. | Folders | Editable | | Decode DLLs (per stream) | Path to decode DLL for each of 16 streams (00–15) for processing generic data sources. Can browse or reset to default (AtlasApiDecoder64.dll). | Decode DLLs | Default: CustomDLLs\AtlasApiDecoder64.dll | | Debug Decode DLL | Allows decode DLL to crash and generate a memory dump for debugging. | Decode DLLs | Checkbox | | Stream Server Configurations | List of Stream Server configurations for the Stream Recorder. | Stream Recorder | Editable | | Friendly Name | Name for the Stream Server configuration. | Stream Recorder | User-defined | | Broker URL | URL and port of the broker for the Stream Server. | Stream Recorder | User-defined | | Stream Port | Port of the Stream Server. | Stream Recorder | User-defined or default | | Stream Creation Strategy | Strategy for stream creation in the broker: **TopicBased** (each stream is a topic) or **PartitionBased** (streams are partitions in one topic). Must match broker. | Stream Recorder | TopicBased or PartitionBased | | Partition Mapping | Maps streams to partition numbers (only for PartitionBased strategy). Must match broker mapping. | Stream Recorder | User-defined | --- # Workbooks Workbooks in ATLAS define your workspace layout and the arrangement of elements on the screen. Every ATLAS session contains at least one Workbook, which can hold multiple Pages, and each Page can display several [Displays](../displays/index.md). ![Workbook](assets/workbook.png) ## Saving and Opening Workbooks - **Save your workspace:** Workbooks are saved as `.wbkx` files, preserving the formatting, size, and position of each Display and Page. - **Open existing workspaces:** Use the File menu to open a Workbook. This replaces your current workspace. If you have unsaved changes, ATLAS will prompt you before proceeding. - **Save As:** To rename a Workbook, use the Save As command. The default name is assigned when first opened (e.g., Workbook1). !!! note Compare Sets, Session data, Bit Config Files, Functions files, Alarm Files, Circuit Display Definitions, SurfaceMap Display Definitions, and PCU Dash Display RAW CAN files are not stored in `.wbkx` files but are automatically opened by the Workbook. If these files are missing, Displays will be empty. ## Working with Pages - **Default setup:** When ATLAS starts or a new Workbook is opened, it includes one Page. - **Multiple Pages:** You can add, float, and arrange multiple Pages within a Workbook. Only one Page is selected at a time, but you can view several by floating them. - **Page Bar:** Located at the top of the Main Window, it displays tabs for each Page. Use horizontal scroll arrows if tabs exceed available space. ### Page Management Options Right-clicking a Page tab opens a menu with options to: - **Set:** Switch or remove Compare Sets associated with the Page. - **No Association:** Remove the current Compare Set link. - **Lock to Current Association:** Lock the Page to its current Compare Set. - **Title:** Edit the Page name. - **Page Colour:** Underline the tab with a chosen colour. - **New:** Add a new Page. - **Paste/Import Display:** Add copied or saved Displays. - **Duplicate:** Copy the Page and its Displays. - **Export:** Save the Page for use in other Workbooks. - **Close/Close Others/Close Tab Group:** Manage Page tabs. - **Float/Float Page Group:** Move Pages into separate windows. - **Pin Tab:** Toggle pin status. - **New Horizontal/Vertical Tab Group:** Organize tabs in groups. ![Pages](assets/pages.png) ## Selecting Pages - Click the desired tab on the Page Bar. - Or, press **Page Up** or **Page Down**. ## Adding Pages To add a new Page: - Select **New** from the File menu, then **Page**. - Click the **+** on the Page Bar. - Click the **New Page** icon on the Main Toolbar. - Right-click a Page tab and choose **New Page**. - Press **Ctrl + Q**, then select **New Page**. New Pages are named sequentially (e.g., Page2, Page3). ## Closing Pages To close a Page: - Click the **X** next to the Page name on the Page Bar. - Or, right-click the Page Bar and select **Close**. ## Status Bar The Status Bar appears at the bottom of the ATLAS Main Window and provides real-time feedback on system activity: - **Last Action:** Displays a brief description of the most recent significant event, such as "Recording started" or error notifications. - **Recorder Status:** Shows a symbol for each connected recorder, with initials indicating the recorder type. Status is highlighted by colour—yellow for idle, red for active recording. ![Recorder Status](assets/recorder-status.png) - **Error Count:** Indicates the number of error messages. Click to view details; right-click and select **Clear** to remove the messages. - **Alarms Warnings:** Shows the count of active alarms with warning status and lists the five most recent alarms. Refer to [Alarm Warnings](../alarms/index.md) for more information. ![Alarm Status](assets/alarm-status.png) ## Importing/Exporting Configuration It's possible to export and import configurations, a useful action when you want to send your ATLAS workbook setup to another and ensure that everything is setup and immediately setup without repeated setup. To export your configuration: 1. Go to `Tools > Import/Export Configuration` 2. Click `Refresh Current Configuration` 3. Select the source files you wish to include in the configuration export 4. Define a name for this profile configuration !!! tip Make sure the name is identifiable, as multiple profiles can be part of the same configuration bundle file. 5. Click `Export` to download your configuration bundle file. ![Export](assets/import-export-config.png) To import a configuration: 1. Go to `Tools > Import/Export Configuration` 2. Click the `Import ATLAS 10 Configuration Bundle` tab 3. Browse for the `.acgx` configuration bundle file 4. Select from the profile you wish to import and the source files. 5. Click `Import` to begin ![Import](assets/import-configuration.png) --- # Toolbar Toolbars provide quick access to various ATLAS commands. By default, they are docked along the top edge of the Main Window. Toolbar buttons are duplicates of Menu commands or other controls. Each item is described by its ToolTip text. Any command that is unavailable in a given context is greyed out but remains on the toolbar. Toolbars can be moved and rearranged by dragging the toolbar handle, found on the left-hand end of each toolbar: | Action | Icon | Description | Toolbar | |--------|------|-------------|---------| | New Workbook | ![icon](assets/new-workbook.png) | Create a brand‑new workbook. | Main Toolbar | | Open Workbook | ![icon](assets/open-workbook.png) | Open an existing workbook file. | Main Toolbar | | Save Workbook | ![icon](assets/save-workbook.png) | Save changes to the current workbook; prompts for a name if it hasn’t been saved before. | Main Toolbar | | Save Workbook As | ![icon](assets/save-as.png) | Save a copy of the current workbook under a different name. | Main Toolbar | | Add New Page | ![icon](assets/new-page.png) | Insert a new page into the active workbook. | Main Toolbar | | Bar Display | ![icon](assets/bar.png) | Insert a Bar Display into the active workbook. | Display Toolbar | | Bing Map Display | ![icon](assets/map.png) | Add a Bing Map‑based display to the workbook. | Display Toolbar | | Bit Display | ![icon](assets/bit.png) | Add a Bit Display for showing bit/boolean states. | Display Toolbar | | Block Bar Display | ![icon](assets/block.png) | Add a Block Bar Display for block‑style bar visuals. | Display Toolbar | | Circuit Display | ![icon](assets/circuit.png) | Add a circuit/track visualization display. | Display Toolbar | | Countdown Timer Display | ![icon](assets/countdown.png) | Add a display that counts down from a set time. | Display Toolbar | | Error Display | ![icon](assets/error.png) | Add a panel to show errors/alerts. | Display Toolbar | | Events Display | ![icon](assets/event.png) | Add a display that lists or plots events. | Display Toolbar | | Histogram Display | ![icon](assets/histogram.png) | Add a histogram to visualize distributions. | Display Toolbar | | Loadmap Display | ![icon](assets/loadmap.png) | Add a load‑map style visualization. | Display Toolbar | | Notepad Display | ![icon](assets/notepad.png) | Add a simple notes/notepad panel. | Display Toolbar | | Numeric Display | ![icon](assets/numeric.png) | Add a numeric readout for key values. | Display Toolbar | | PCU‑Dash Display | ![icon](assets/pcu.png) | Add a PCU dashboard‑style display. | Display Toolbar | | Scatterplot Display | ![icon](assets/scatter.png) | Add a scatter plot visualization. | Display Toolbar | | Summary Display | ![icon](assets/summary.png) | Add a summary panel aggregating key metrics. | Display Toolbar | | Surface Map Display | ![icon](assets/surface.png) | Add a surface‑map view. | Display Toolbar | | Video Display | ![icon](assets/video.png) | Add a player panel for video content. | Display Toolbar | | Waveform Display | ![icon](assets/waveform.png) | Add a waveform/trace plot area. | Display Toolbar | | Web Browser Display | ![icon](assets/web.png) | Embed a web browser view inside the workbook. | Display Toolbar | | Alarm Editor | ![icon](assets/alarm.png) | Open the Alarm Editor tool. | Browser Toolbar | | Session Browser | ![icon](assets/session.png) | Open the Session Browser. | Browser Toolbar | | Parameter Browser | ![icon](assets/parameter.png) | Open the Parameter Browser. | Browser Toolbar | | Function Editor | ![icon](assets/function.png) | Launch the Function Editor. | Browser Toolbar | | Value Cursor | ![icon](assets/value.png) | Show a vertical cursor to read values at a chosen point. | Cursor Toolbar | | Cross Hair Cursor | ![icon](assets/crosshair.png) | Show intersecting vertical and horizontal guides for precise value reads. | Cursor Toolbar | | Gradient Cursor | ![icon](assets/gradient.png) | Display a cursor that reports the gradient/slope at the selected point. | Cursor Toolbar | | Hide Cursor | ![icon](assets/hide-cursor.png) | Turn off the cursor overlay on the display. | Cursor Toolbar | | Create Zoom Box | ![icon](assets/create-zoom.png) | Drag to zoom into a region on Waveform or Scatterplot displays. | Display View Toolbar | | Undo All Zooms | ![icon](assets/undo-zoom.png) | Reset all zoom levels back to the default view. | Display View Toolbar | | Auto‑scale | ![icon](assets/auto-scale.png) | Automatically fit data to the available plotting area. | Display View Toolbar | | Reset Auto‑scale | ![icon](assets/reset-auto.png) | Exit auto‑scale and return to manual/normal scaling. | Display View Toolbar | | Distribute Parameters | ![icon](assets/distribute.png) | Arrange selected parameters in equally sized rows (Waveform). | Display View Toolbar | | Reset Distribute Parameters | ![icon](assets/reset-distribute.png) | Revert to the standard parameter layout (Waveform). | Display View Toolbar | | Toggle Display Title Bar | ![icon](assets/title.png) | Show or hide the title bar for the selected display. | Display Titles Toolbar | | Move to Previous Lap | ![icon](assets/previous.png) | Move the cursor back to the prior lap. | Laps Toolbar | | Move to Current Lap | ![icon](assets/current.png) | Zoom so only the currently selected lap is visible. | Laps Toolbar | | Move to Next Lap | ![icon](assets/next.png) | Advance the cursor to the next lap. | Laps Toolbar | | Laps & Markers Editor | ![icon](assets/lap-editor.png) | Open the editor to view or modify laps and markers in the primary session. | Laps Toolbar | | Move to Next Signal Change | ![icon](assets/next-gear.png) | Navigate to the adjacent signal change (next). | Laps Toolbar | | Move to Previous Signal Change | ![icon](assets/previous-gear.png) | Navigate to the adjacent signal change (previous). | Laps Toolbar | | Quick Access Assistant | ![icon](assets/quick-access.png) | Launch the assistant to quickly add parameters, functions, displays, or pages. | Quick Access Toolbar | | Go To | ![icon](assets/go-to.png) | Launch the Go To plugin to jump to a specific lap or marker in the primary session. | Go To Feature Toolbar | | Refresh Calculated Parameters | ![icon](assets/refresh-params.png) | Recompute all calculated parameters across the entire workbook. | Refresh Toolbar | | Reset Parameter Properties | ![icon](assets/reset-param.png) | Resets the settings for all parameters to their default settings. | Refresh Toolbar | --- # Overview ATLAS provides powerful tools for visualising data over time using Timebases and Graphical Timelines. Each Display has a Timebase, defined by a start and end time (duration) and a current position. Timebases can be set independently or linked across Displays, and are updated automatically when telemetry is received. The Graphical Timeline, shown at the top of the ATLAS Main Window, lets users control the Timebase using mouse interactions. It displays key session information, lap selection, markers, and navigation buttons for quickly moving between laps. The LapBar visually represents session duration, with laps and fastest lap highlighted. For Compare Sets with multiple sessions, the Timeline offers options to compare whole sessions, single laps, or fastest laps, and allows setting time offsets between sessions. This overview introduces the main concepts and controls for managing time and position in ATLAS, helping users navigate and analyse session data efficiently. ![Timeline Example](assets/timeline.png) --- # Timebase and Timeline Each Display in ATLAS has a **Timebase**, defined by: - **Start and End Point**: The time range covered. - **Duration**: The length of time shown. - **Position**: A specific point within the Timebase. **Default Settings**: Timebase duration and position are based on the absolute time in the active Session. Displays can have independent or linked Timebases—changes to a linked Timebase affect all associated Displays. ### Adjusting the Timebase You can change the Timebase in several ways: - **Position & Duration**: - Use the Graphical Timeline - Use Move controls - **Position Only**: - Move the cursor in Waveform, Circuit, or Scatterplot Displays - **Duration Only**: - Use Zoom controls in the Waveform Display When telemetry is received, the Timebase position updates automatically. ## Timeline Overview The **Timeline** is shown at the top of the ATLAS Main Window when a Session or Compare Set is loaded. It provides graphical control over the Timebase and lets you navigate, select laps, and compare data. ![Timeline Example](assets/timeline.png) ### Timeline Components - **Set Tag**: Indicates which Session or Compare Set the Timeline applies to. - **Set Name**: Shows driver and session description. - **Lap Selection**: Select laps to change the Timebase. - **Marker Selection**: Jump to specific points or ranges using markers. - **Navigation Buttons**: - Previous Lap - Next Lap - Fastest Lap - **Lap Bar**: Visualizes session duration and lap boundaries. - **Time Position**: Marked by a vertical red line. - **Compare Set Scope & Offset**: Choose what to compare (whole session, single lap, fastest lap) and set time offsets between sessions. ## Timeline Actions Interact with the Timeline using your mouse and keyboard: - **Double-click a lap**: Selects the whole lap as Timebase, centers Time Position. - **Triple-click**: Selects the entire session as Timebase. - **Click + Drag**: - Left to right: Select any duration - Right to left: Select whole laps - **Right-click**: Remove sessions or manage Compare Sets. - **Left-click a Set tag/number**: Show/hide Compare Sets. - **Right-click session identifier**: Promote to primary, delete, or view session details. - **Zoom**: Use `+` and `-` keys or mouse wheel to zoom in/out on the Timeline. ## Timebase vs Distance Mode The **Waveform Display** supports two modes: - **Time Mode** (default): Parameters plotted against time. - **Distance Mode**: Parameters plotted against distance. **When to use each mode:** - Use **Time Mode** for time-based analysis (e.g., event timing, telemetry trends). - Use **Distance Mode** for lap comparison, performance over track segments, or when assessing lap time improvements. ### Switching to Distance Mode 1. Select a valid Distance parameter in `Tools > Options > General > X Axis Mapping`. 2. Add and configure the Distance parameter (e.g., `sLap`). 3. In the Waveform Display, press `K` to toggle modes, or right-click and select the mode from the context menu. When Distance Mode is active, a Time Difference (TDiff) function is shown, helping you compare lap times and performance. ![Distance Mode Example](assets/waveform-mode.png) ## Timeline Visibility Timelines are visible if any display is associated with their Set. You can force visibility by right-clicking the Set marker and checking the Sets to always display. To show all Timelines, use `View > F1 Timeline > Show All Compare Sets`. ![Show All Timelines](assets/compare-set.png) ## Session Details Tooltip Hover over the session identifier in the Timeline to see session details. Customize which properties are shown by right-clicking and selecting `Session Details`. You can select up to 5 properties (default limit, configurable). ![Session Details](assets/session-details.png) ![Select Details](assets/add-more.png) --- # Go To The **Go To** feature in ATLAS 10 allows users to quickly navigate to specific laps, markers, or exact positions (by time or distance) within a session. This feature streamlines data exploration and analysis, making it easier to locate and review key moments. ### How to Open the Go To Window You can access the Go To window in any of the following ways: - Click the Go To icon on the ATLAS toolbar. - Select **View > Go to** from the Menu Bar. - Use the keyboard shortcut **Ctrl + G**. ### Navigation Options The Go To dialog provides three main navigation options, selectable via a drop-down list: 1. **Laps** – Jump to a specific lap in the session. 2. **Markers** – Move to a defined marker (point or range) within the session. 3. **Exact Setting** – Position the cursor at a precise time or distance. #### 1. Go To Feature - [Laps](../handling-data/laps-markers/laps.md) - The default option is **Laps**. - Select a lap from the drop-down list and click **Apply** to move the cursor to that lap. - **OK** applies the selection and closes the window; **Cancel** closes the window without changes. - For live or replay sessions, playback will pause and jump to the selected lap. ![Go To Laps](assets/go-to-laps.png) #### 2. Go To Feature - [Markers](../handling-data/laps-markers/markers.md) - Select **Marker** in the options drop-down to navigate to a session marker. - Choose a marker from the list: - **Range Marker**: Cursor moves to the center of the range and waveform zooms to the marker's time base. - **Point Marker**: Cursor moves to the marker point without changing zoom. - For live or replay sessions, playback will pause and jump to the selected marker. ![Go To Marker](assets/go-to-markers.png) #### 3. Go To Feature - Exact Setting Use the **Exact Setting** option to move the cursor to a specific time or distance. **Time Mode:** - Definitions available: - **Exact Time**: Enter a time value to move the cursor directly to that point. - **Time from start of lap**: Enter a time offset from the start of a selected lap. - **Time from end of lap**: Enter a time offset from the end of a selected lap. - If the session spans multiple days, a **Day** drop-down appears for selecting the day. - If the entered time is out of session range, an error dialog will prompt for a valid value. ![Error](assets/error.png) **Distance Mode:** - Definitions available: - **Exact Value**: Enter a distance value to move the cursor directly to that point. - **Value from start of lap**: Enter a distance offset from the start of a selected lap. - **Value from end of lap**: Enter a distance offset from the end of a selected lap. - If the entered distance is out of session range, an error dialog will prompt for a valid value. ![Go To Exact](assets/go-to-exact.png) --- # USB Recorder This guide provides a step-by-step overview of how to use a USB Logger system to collect, transfer, and visualise sensor data efficiently. It covers setup on the vehicle, data offloading procedures, and how to import and decode the data in ATLAS. Designed for flexibility, the system supports a range of operational environments. ## Prerequisites - Motion Applied USB Logger unit. - ATLAS Version 11.4.4.349-W47 or later. If you're interested in how Motion Applied can advance your data systems, contact [sales@motionapplied.com](mailto:sales@motionapplied.com). ## Setup ??? example "Prefer to watch a video demonstrating how to integrate the USB Logger with ATLAS?" **1. Connect the Logger to the ECU** - Use an Ethernet cable to connect the ECU to the USB Logger. - If you need quick access to the USB for session offloading, set the `log_destination` in `logger_config.json` to `usb_loom`. In this mode, the logger writes data through the loom to a USB stick placed in an easily accessible location for fast retrieval. - *(Optional)* You can adjust other logger configurations by editing the `logger_config.json` file. Each field includes comments explaining its purpose and valid values. !!! warning You can change the logger’s IP address in the `logger_config.jsonc` file under the `ethernet_settings` section.  The Logger IP address must be in the **same range** as the ECU IP address. **2. Configure Logging from the ECU** - Open **System Monitor**, connect to the TAG unit (e.g. TAG-320, TAG-510, or TAG-700), and in the **Explorer** tab navigate to: `BIOS > Data Acquisition > Telemetry > TLM-Px`. - Locate the parameter `TLMPxIPAddress` and set it to the logger’s IP address from the `logger_config.jsonc`. !!! tip While the parameter may appear as `NTAG320TLMPxIPAddress` on TAG-320 units, other models like TAG-510 or TAG-700 may use a slightly different prefix. To ensure compatibility, look for the `TLMPxIPAddress` field under the **Telemetry > TLM-Px** section in the BIOS tree. This structure is consistent across supported TAG units. - In ATLAS, go to **Tools > Options > System Monitor Folders** and set up the folders containing the required ECU configuration files (e.g., PGVs). !!! info When the vehicle is powered on, the ECU will automatically begin logging data to the USB drive. Depending on session length, multiple `.bin` files may be generated. **3. Swapping the USB** - When the vehicle enters the pit, remove the used USB and insert a new one. - The ECU will continue logging to the new USB without interruption. **4. Insert the USB into a PC** - Plug the USB into a PC using the adapter provided in the kit. - Copy the `.bin` files to a local folder on your PC. - After copying, you may delete the `.bin` files from the USB to prepare it for reuse. !!! info Even if you accidentally delete the `logger_config.jsonc` file from the USB, the logger will **continue recording** using the last configuration it stored locally. When you plug the USB back into the logger it will **automatically restore the latest version** of `logger_config.jsonc` onto the USB. You can then **review or update the configuration** as needed. **5. Add a USB Logger Recorder in ATLAS** - Launch **ATLAS**. - Open the **Session Browser** (`CTRL + L`) and create a new recorder. ![New Recorder](assets/usb-logger/usb_logger_1.png){: style="width:75%;"} Set the following: - **Database engine** (where the data will be stored) - **Auto Export SSN2** *(optional)* to automatically export the created sessions to SSN2 once processing is complete. Go to the USB Logger Bin File(s) tab and set: - **Folder path** to the `.bin` file(s) - **Select the bin files** you wish to import. ATLAS will automatically detect and import multiple `.bin` files found in the folder as separate sessions. !!! note When specifying the folder path, ensure that all required `bin` files are located directly within that directory. ATLAS will not process files located in subfolders. **6. Start the Recorder** - The recorder will process each binary file individually, creating a new session per `bin` file. If *Auto Export SSN2* was selected when configuring the recorder it will export the converted `.ssn` files to the export folder after processing is finished. - You can monitor progress via the status bar located in the bottom-right corner of ATLAS or in the recorder itself. ![New Recorder](assets/usb-logger/usb_logger_3.png){: style="width:75%;"} !!! warning The USB Logger Recorder does not record in the same way as other recorders. Behind the scenes it replays existing data and generates a historical session file from the replay. As a result you cannot drag the USB Logger recorder into a Set until it has finished converting the data. This recorder is **not** used for real-time replay. **7. Using the Data** - Once the session is fully read and marked as **Historical**, it will appear in the **Sources** panel. - You can import the session into a **Set** using **Append** or **Compare** mode, like any other historical data. ![New Recorder](assets/usb-logger/usb_logger_4.png){: style="width:75%;"} Depending on the selected database engine: - **SQLite**: Share the exported `.ssn` file via a network drive. - **SQL Race**: Team members with access to the database can view the session directly. ## Use Cases The USB Logger system is designed for flexibility and ease of use across a wide range of environments and industries. Example applications include: **Extreme Environments** Water-resistant design allows use in harsh weather or rugged terrain. **Limited Storage** Ideal for systems where the ECU has limited onboard storage capacity. **Endurance Operations** Supports long-duration sessions where quick data offloading is essential. **Difficult Access** Enables data retrieval without needing direct physical access to the ECU. **Limited Telemetry Bandwidth** Provides a fast alternative to telemetry for transferring session data. **Scalable for Small Teams** Runs directly on an engineer’s PC without requiring advanced setup. Can be integrated into larger systems later using tools like the SQL Race API. --- # NASCAR Telemetry Recorder ATLAS can be used for receiving and visualising Telemetric data from NASCAR’s Event Racing Data Platform. It is designed to connect to the data feed delivered live via JSON messages published from NATS.io message broker. The integration provides complete ATLAS data analysis functionality for use with the data from NASCAR. ATLAS currently supports all of the information collected to the CAN bus from Motion Applied’s TAG-400N. ## Prerequisites * Access to live NATS data feed from NASCAR. Request Subscriber Access from NASCAR by emailing [data.engineering@nascar.com](mailto:data.engineering@nascar.com) * ATLAS v11.3.1.540 or higher, which can be downloaded via our Motion Applied Portal [https://portal.motionapplied.com/](https://portal.motionapplied.com/) * ATLAS Licence with NASCAR Telemetry Recorder add-on. Please contact [sales@motionapplied.com](mailto:sales@motionapplied.com) ## Setup Once ATLAS has been installed and licensed: **Configure NASCAR Server connections** * Open ATLAS and click Tools | Options | Recorders | NASCAR Telemetry Recorder * Edit and configure Integration (Replay), Trackside and Cloud NASCAR Server connections ![nascar-server-connections.png](assets%2Fnascar-telemetry-recorder%2Fnascar-server-connections.png) !!! note * You will receive two sets of credentials from NASCAR: * Production - add these to your Trackside and Cloud server connection settings * Integration - add these to your Integration (Replay) server connection settings * Update the credentials / ca-file / cert-file / key-file paths so that they point to the subscriber credentials from NASCAR ![nascar-server-connection.png](assets%2Fnascar-telemetry-recorder%2Fnascar-server-connection.png) * Click OK on the NASCAR Server connection **Configure Wideband minimum lap time** * In the Recording option (Tools | Options | Recorders | Recording | General) set Wideband minimum lap time (secs) = **1** second ![wideband-minimum-lap-time.png](assets%2Fnascar-telemetry-recorder%2Fwideband-minimum-lap-time.png) * Click OK to save and close the Options screen **Add a NASCAR Telemetry Recorder** * In Session Browser, add a NASCAR Telemetry Recorder ![ntr-session-browser.png](assets%2Fnascar-telemetry-recorder%2Fntr-session-browser.png) * Configure the NTR Recorder Editor dialog ![ntr-recorder-editor.png](assets%2Fnascar-telemetry-recorder%2Fntr-recorder-editor.png) !!! info * Recorder Name - can be any friendly name e.g. Car 6 * Server: * Trackside - for use at track (recommended hard-wired) > Wi-Fi can be very spotty; if data has dropouts, then wired connection will be necessary. Dropouts have not been seen in remote recordings of data. * Cloud - for use at factory or other non-track locations * Integration (Replay) - for use after event, replays constant stream of data for testing purposes * File Telemetry - for replaying JSON files from previous events. These can be then be saved to an SSNDB or as SSN2 files. * Session Identifier - name of the session can be configured using tokens and information from the data, e.g. $SeriesId$_$RunType$_$RaceId$_$VehicleId$_%y%m%d%H%M%S * Car Number - Select a vehicle ID to get the relevant data for that car * Database engine - Select SQLite for storing data in a local cache. SQLServer option can be selected if there is a SQL Race database set up. * Database path - If SQLite option is selected, user can set a local path, folder and name the .ssndb for the local cache. For SQLServer option, a connection string for the database will be needed. > Do not use a folder location which is being used to sync the content of the folder to OneDrive. This doesn't work well with the Recorders as ATLAS has to constantly write and read from this location. * Auto Export SSN2 - If you check this box an SSN2 export for every recording will be saved such that it creates a file per start/stop of the recorder. * Select Close and add the recorder to a SET, then click start. If the NATS server is streaming data, the NTR recorder box will begin to flash red if a connection is achieved. If parameters are not present it is possible the car is not streaming. > If the recorder is not working, this might be down to a Firewall issue, so please check with your IT department !!! note * You can also access the previously recorded JSON files from [https://teamcontent.nascar.com/](https://teamcontent.nascar.com/) These file can be downloaded and played back into ATLAS and recorded as SSN2 files. > The JSON files must be copied into: C:\NASCAR\race_data ## Hints and Tips !!! tip * ATLAS performance is dependent on the hardware it is being run on and scales based on the processors and RAM available to the ATLAS application. We recommend using i7 or higher processor for running multiple recorders i.e. for streaming data from multiple cars. * Queue Size - we have included a diagnostic parameter called Queue Size, which tells how many packets are in the queue still to be processed. Low number (10's) denotes that the application is keeping up with the live stream, if not please check the recommended hardware requirements for ATLAS. * Numeric Displays and some parameters will need parameter Display Min and Max set. By default, they are way out of range and will load in very tiny text. Max 100 and Min 0 is a good starting point. * The provided Workbook has a Diagnostic Bit display. The APPStatus and BIOSStatus appear to be correct in the messages sent from server. The NLEDFlashFaults appear to be wrong, and NASCAR is aware. * Alarms and Functions will work with NTR data, but ATLAS cannot handle parameter names with spaces. So “Engine Speed” will not work in a function or alarm. NASCAR is aware of this, and we will investigate either getting the parameter names changed or updating ATLAS to accept name spacing. * All sessions by default will record into a SQLite database on your hard drive and export to SSN2 if you choose. SRMS (SQL Race Management Studio) can be used to edit or delete sessions from your SQLite databases after a weekend if they are not required. Please contact an FAE for further guidance on using SRMS. ## Known Limitations !!! warning * Data stream is UTC time and at 23:59.59 (7pm EST) ATLAS will stop recording. Restarting recorders will be necessary. This is known and our ATLAS team is looking at this. * Recorders - There is a known issue with ATLAS Telemetry Recorders (NTR included). If a recorder is added to a set and then removed, the recorder will disappear when ATLAS is restarted. This can be very inconvenient, so we recommend assigning a recorder to a set and leaving it. You can have up to 20 sets in a workbook and probably won’t have more than 20 recorders running at once. --- # Data Server Telemetry Recorder ## Purpose Connects to a live telemetry stream from an ATLAS Data Server over a network. ## Prerequisites 1. ATLAS Data Server must be running on the source machine. 2. The TAG channel must match between client and server. 3. The ATLAS client must be on the same network or have access to the server. ## Setup Instructions 1. Open Session Browser in ATLAS. 2. Add a Data Server Telemetry Recorder. 3. Configure the following: - Recorder Name: Friendly name for identification. - Data Server: Select from {servername}:{Data Stream name}. - Refresh: Click to update the list of available servers. - Session Identifier: Use tokens or manual entry. - Auto Record: Enable if you want recording to start automatically. 4. Click Start to begin manual recording. ## Notes The first client to connect sets the TAG channel. If the server list is outdated, click Refresh or restart ATLAS. Recording will not start unless the Data Server is running and the TAG channel matches --- # File Telemetry Recorder ## Purpose Replays previously recorded raw telemetry data for diagnostics or analysis. ## Setup Instructions !!! note .raw_tm file must exist. These are created during sessions where raw data logging is enabled. 1. Open Session Browser. 2. Add a File Telemetry Recorder. 3. Configure: - Recorder Name - Telemetry Timestamp File: Browse to .raw_tm file. - Session Identifier 4. Click Start to begin playback. ## Notes Useful for debugging or training without needing live data. The file must be located in the same directory as the session and named accordingly --- # Ethernet File Telemetry ## Purpose Streams live data from ECUs connected via Ethernet (e.g. TAG-310). ## Prerequisites Car Definition must be loaded. ECU must be connected via Ethernet. ## Setup Instructions 1. Open Session Browser. 2. Add an Ethernet Telemetry Recorder. 3. Configure: - Recorder Name - Car: Select from Car Definitions. - Unit: Choose a specific unit or “All Units”. - Data Versions: Enable to upload DTVs at start. - Events / Row Data: Enable to save .evt and .row files. - Export Config as Text - Session Identifier 4. Click Start to begin recording. ## Notes Ideal for continuous data streaming during track sessions. Does not require a radio link—Ethernet is sufficient --- # Ethernet Multi Session Offload ## Purpose Offloads multiple stored sessions from ECUs that support multi-session memory (e.g. TAG-310). ## Prerequisites ECU must support multi-session (BIOS B023+). ATLAS or ADS must be running. ## Setup Instructions 1. Open Session Browser. 2. Add an Ethernet Multi-Session Offload Recorder. 3. Use the Connection Tab to: - Select Car and Unit. - Choose SQL Server or SQLite. - Set Export Folder. - Define Session Identifier. - Use the Select Sessions Tab to: - View sessions stored on the ECU. - Select sessions to offload. - Use buttons: Select All, Clear All, Hide Offloaded, Query. 4. Click Start to offload selected sessions. ## Notes Sessions with overlapping timestamps are grouped into one SQL Race session. If the unit doesn’t support multi-session, use the standard Ethernet Telemetry Recorder --- # Ethernet Offload ## Purpose Offloads the most recent session from an Ethernet-connected ECU. ## Prerequisites ECU must be connected and powered. Car Definition must be loaded. ## Setup Instructions 1. Open Session Browser. 2. Add an Ethernet Offload Recorder. 3. Configure: - Recorder Name - Car / Unit - Data Versions - Export Config as Text / M Files - SQL Race Database Type - Export Folder - Session Identifier / Details 4. Click Start to begin offload. ## Notes Only the most recent session is offloaded. Ideal for quick post-session data capture --- # Stream Recorder ## Purpose Connects to a Kafka broker to record live telemetry via the [Open Data Streaming Architecture](https://atlas.motionapplied.com/developer-resources/secu4/docs/). ## Prerequisites Stream Server Configuration must be set up in Tools|Options|Recorders|Stream Recorder. image ## Stream Server Configuration Before any recording can be done, the Stream Server Configuration must be set. This tells the Stream Recorder important things about the Kafka broker it is recording from. ![New Configuration](assets/stream-recorder/blank-configuration.png) # First time Setup 1. Click on Tools|Options|Recorders|Stream Recorder 2. Click on Add to create new Stream Server Configuration. 3. Configure (Basic Settings): - **Friendly Name** - The name at which the configuration will be referred to (This should be unique). - **Broker URL** - The URL of the Kafka Broker. - **Domain Name** - The Domain Name of the Stream API which was setup for the broker. This is only needed if the Producer's Stream API has a domain name defined. - **Stream Creation Strategy** - The strategy at which new streams are made in Kafka. This needs to match the producer's Stream API settings. There are two modes: - **Topic Based** - Streams are divided into Kafka Topics. - **Partition Based** - Streams are divided into Partitions in a single Kafka topic. - **Partition Mapping** - Only required if Partition based Stream Creation Strategy was selected. Maps which stream goes to which parition number. This needs to match the producer's Stream API settings. 4. Click OK and the Stream Server Configuration is now available for all Stream Recorders in that ATLAS to use. # Advanced Settings (Available on ATLAS versions 11.5.2.972-W28 and later) There are Advanced Settings available for the Stream Server Configuration. This is used to setup Kafka Security and enable Kafka Tuning. ## Kafka Security Kafka Security is only needed if Kafka is setup with security enabled. For more info, look at [Kafka Security](https://atlas.motionapplied.com/developer-resources/secu4/stream_api/reference_docs/configuration/kafka-security/#overview). ## Kafka Tuning Kafka Tuning is available to fine tune how Stream Recorder consumes from Kafka. The file path are for the Kafka Tuning json. For more info, click [Kafka Tuning](https://atlas.motionapplied.com/developer-resources/secu4/stream_api/reference_docs/configuration/kafka-broker-tuning/). ## Recording Modes The Stream Recorder supports two operational modes for consuming OSAP data from Kafka. These modes determine where in the stream the Recorder begins reading data and how it handles historical backlog. 1. **Live Mode:** Live mode begins consuming data only from the moment the Recorder starts. The Recorder attaches to the Kafka topic at the current offset and continues forward. 2. **Live with Catchup Mode:** The Recorder starts at the live edge, ensuring it processes new data immediately. Simultaneously, it begins consuming all data from the start of the session. Catch‑up happens in the background, at a controlled rate, without disrupting live processing. Priority is always given to leading‑edge, real‑time data. ## Group Id The group id allows Stream Recorder to set the Kafka Consumer Group ID. This allows Stream Recorder to track the current offset in Kafka and continue reading from the last offset it read from. So if the ATLAS client disconnects and rejoins the stream, it will continue from the last offset it read from (assuming the topics has not been cleared down of messages). If left blank, Stream Recorder will always read from the leading edge of the session. ## Setup Instructions 1. Open Session Browser. 2. Add a Stream Recorder. 3. Configure: - **Recorder Name** - **Database Engine:** SQLite or SQL Race. - **Database Path** / **Connection String**. - **Delete Recorded Session on Close** - **Auto Export to SSN2** (If recording to SQLite) - **Export Folder** - **Stream Server** / **Data Source** - **Session Identifier Pattern** / **Source** - **Recording Mode** - **Group Id** - **Session Details Source** / **Details** 4. Choose between: - **Start:** Waits for a live session and records it. - **Auto Record:** Continuously records new sessions as they appear. ![Recorder Editor](assets/stream-recorder/stream-recorder-editor-page.png) --- # Overview ## What is a Session? A Session in ATLAS is a structured collection of data samples recorded from a vehicle. These samples are captured by a data logger and uploaded to ATLAS either: Live, via telemetry (wireless transmission), or Offline, via a wirelink (physical connection). Each Session is automatically created and stored in an SQL Race database as it is recorded. This database replaces the older .ssn file format used in previous versions of ATLAS. However, legacy .ssn files can still be loaded for analysis and comparison. ## Types of Sessions There are two primary types of Sessions in ATLAS: **Historical Sessions** These are Sessions that have already been recorded and stored. They can be loaded at any time for review and analysis. **Telemetry Sessions** These are live Sessions that are displayed in real-time as data is received from the vehicle. !!! warning Data visualisation and analysis in ATLAS can only begin once a Session is loaded into a Workbook. A Workbook is the environment where data is displayed using graphs, tables, and other visual tools. ## Why Sessions Matter Sessions are the foundation of all data analysis in ATLAS. They allow engineers and analysts to: - Monitor vehicle performance in real time - Compare data across different runs or configurations - Identify trends, anomalies, or failures - Share insights with other team members Sessions can be combined, compared, and manipulated in various ways to support these goals. The tools used to manage Sessions include: - Session Browser: The main interface for loading, recording, and editing Sessions. - Composite Sessions: Merged Sessions that combine multiple data recordings into one. - Compare Sets: Groups of Sessions loaded together for side-by-side analysis. ### Key Concepts | Term | Definition | |-------------------|----------------------------------------------------------------------------| | Session | A dataset recorded from a vehicle, stored in an SQL Race database. | | Historical Session | A previously recorded Session. | | Telemetry Session | A live Session streamed in real time. | | Workbook | The environment in ATLAS where Sessions are visualised and analysed. | | Session Browser | The tool used to manage Sessions. | | Composite Session | A combination of multiple Sessions treated as one. | | Compare Set | A group of Sessions loaded together for comparison. | --- # Session Browser ## Purpose of the Session Browser The Session Browser is the primary interface in ATLAS for managing telemetry data Sessions. It allows users to: - Load previously recorded (Historical) Sessions - Record new (Live) Sessions - Configure and manage data Recorders - Combine multiple Sessions into Composite Sessions - Group Sessions into Compare Sets for analysis - View and edit Session metadata - Connect to various data sources (e.g. databases, folders, services) This tool is essential for any user working with telemetry data in ATLAS, whether for real-time monitoring or post-session analysis. ## How to Open the Session Browser You can open the Session Browser in one of the following ways: **Toolbar Icon** Click the Session Browser icon on the ATLAS toolbar. **Menu Bar** Navigate to `View > Session Browser`. **Keyboard Shortcut** Press `Ctrl + L` on your keyboard. Once opened, the Session Browser will appear within the ATLAS interface. ## Layout of the Session Browser The Session Browser is divided into six main panels, each serving a specific function: ![Session Layout](assets/session-browser.png) ### 1. Session Sources This panel allows you to define where ATLAS should look for Sessions. Supported sources include: - Folders on your local machine or network - SQL Race databases - RTA service connections (requires an RTA licence) To add a new source: 1. Click the Add Data Source button or right-click the Sources panel. 2. Choose one of the following: - Connection Manager: For SQL Race databases - Folder Manager: For local or network folders - Service Connection Manager: For RTA hosts Once added, ATLAS will index the source and display available Sessions in the Session List. ### 2. Session Filters Filters help you narrow down the Sessions displayed in the Session List. By default, filters include: - Circuit - Driver - Run Type You can customise filters by: 1. Right-clicking the Filters title to see all available categories 2. Selecting or deselecting filter options 3. Saving filter combinations as Favourites for quick reuse ### 3. Session Recorders This panel is used to configure and manage data Recorders for capturing live telemetry. Supported Recorder types include: - Data Server Telemetry - Ethernet Telemetry - File Telemetry - Ethernet Multi-session Offload Data Recorder - TCP Listener Recorder - NASCAR Recorder (requires NASCAR licence) **Recorder Modes** When adding a Recorder to a Set, you can define its behaviour: - Create: Each new recording creates a new Session. - Replace: Each new recording replaces the previous Session in the Set. - Append: New recordings are appended to existing Sessions, forming a Composite Session. !!! note Starting a Recorder does not automatically load the Session into a Set. You must manually add it using the Session List or drag the Recorder into a Set. ### 4. Session List This panel displays all Sessions available from the selected source. Each entry includes metadata such as: - Session type (Historical or Live) - State - Driver - Circuit - Number of laps You can: - Search for Sessions using the search box - Right-click to add Sessions to new or existing Sets - Drag Sessions into Sets or onto other Sessions to create Composite Sessions - Select multiple Sessions using `Shift + Click` or `Ctrl + Click` ### 5. Compare Sets List Also known as the Set List, this panel shows which Sessions and Recorders are currently loaded into each Set. Features include: - Viewing Sessions within each Set - Expanding Composite Sessions to see individual components - Removing Sessions from Sets - Customising columns (changes apply to both this panel and the Session List) The active Set is highlighted with a coloured outline. ### 6. [Session Properties](properties.md) This panel displays detailed metadata for the selected Session. Properties are colour-coded: - White fields: Editable - Grey fields: Read-only Editable properties include: - Session Name and Number - Driver, Car, Engine, Circuit - Weather, Notes, and other descriptive fields - Lap Trigger Offset (used to align data across Sessions) --- # Loading Sessions ## [What is a Recorder?](../recorders/usb-logger.md) A Recorder in ATLAS is a tool that captures live telemetry data from a vehicle and stores it in an SQL Race database. Recorders are configured and managed through the Session Browser. Supported Recorder Types You can choose from several Recorder types depending on your data acquisition setup: - Data Server Telemetry - Ethernet Telemetry - File Telemetry - Ethernet Multi-session Offload Data Recorder - TCP Listener Recorder - NASCAR Recorder (requires a NASCAR licence) Each type corresponds to a specific method of receiving telemetry data. ### Adding and Configuring a Recorder 1. Open the Session Browser. 2. Click the Add Recorder button. 3. Select the appropriate Recorder type. 4. Configure the Recorder settings as required. Once added, the Recorder appears in the Recorders panel. ### Starting a Recording To begin recording Click Start on the Recorder. - Depending on the Recorder type, data may begin streaming immediately or wait for a trigger. !!! note Starting a Recorder does not automatically load the Session into a Set. You must manually add the Session to a Set to view and analyse it. When a Recorder is added to a Set, you must define its mode. This determines how new recordings are handled in relation to existing Sessions. ![Recorder Modes](assets/recorder.png) **Mode: Create** *Function:* Each new recording creates a new, separate Session. *Behaviour:* Previously recorded Sessions remain in the Set as Historical Sessions. *Use Case:* Ideal for capturing multiple independent runs or tests. **Mode: Replace** *Function:* Each new recording replaces the previous Session in the Set. *Behaviour:* Only the most recent Session is retained. *Use Case:* Useful for iterative testing where only the latest data is relevant. **Mode: Append** *Function:* Each new recording is appended to the existing Sessions in the Set. *Behaviour:* Multiple recordings are merged into a single Composite Session. *Use Case:* Best for continuous data collection across multiple runs or phases. Composite Sessions created via Append mode are displayed as a single timeline in ATLAS. Green line separators indicate where individual Sessions have been joined. ## What is a Historical Session? A Historical Session is a previously recorded Session stored in the SQL Race database. These Sessions can be loaded into ATLAS for review, comparison, or further analysis. Loading Methods You can load Sessions into a Set using the Session List in the Session Browser: 1. Double-click a Session to add it to the currently selected Set. 2. Drag and drop a Session into a Set. 3. Right-click a Session and choose: - Add to New Set: Creates a new Set and adds the Session. - Add to Set: Adds the Session to an existing Set. 4. Drag one Session onto another in the Set list to create a Composite Session. 5. Select multiple Sessions using Shift or `Ctrl + Click`, then: - Right-click and choose Append and Add to New Set. - Right-click and choose Append and Add to Set. ## What is a Compare Set? A Compare Set is a group of one or more Sessions (or Composite Sessions) loaded together for side-by-side analysis. Compare Sets are essential for: - Comparing different runs or configurations - Analysing performance trends - Identifying anomalies or inconsistencies ![Compare Set Example](assets/compare-set.png) You can create a Compare Set in several ways: 1. Double-click a Session in the Session List. 2. Drag and drop a Session into the Sets panel. 3. Right-click a Session and select Add to New Compare Set. 4. Select multiple Sessions and: - Right-click and choose Add to New Compare Set. - Drag them into the Sets panel. Adding to an Existing Compare Set: 1. Drag a Session onto an existing Set in the Sets panel. 2. Select multiple Sessions and drag them onto an existing Set. ### Viewing and Managing Compare Sets The Compare Sets List (also called the Set List) shows all active Sets. 1. Click a Set tab to expand and view its Sessions. 2. Click the arrow next to a Composite Session to view its components. 3. Right-click a Session to remove it from the Set or press Delete. Customising the Compare Sets View 1. Right-click column headers to add or remove metadata fields. 2. Changes apply to both the Compare Sets List and the Session List. ## What is a Composite Session? A Composite Session (also known as an Appended Session) is a single Session formed by appending multiple Sessions together. This is useful for: - Continuous data collection across multiple runs - Reducing clutter by merging related Sessions - Aligning data for comparative analysis ![Composite Session Example](assets/composite-set.png) Creating Composite Sessions - Historical Data: Load a Session into a Set, then drag additional Sessions onto it. - Live Data: Use a Recorder in Append mode. Each new recording is automatically appended to the previous one. Composite Sessions appear as a single timeline in ATLAS, with green separators marking the boundaries between original Sessions. ![timeline](assets/timeline.png) --- # Importing CSV and TXT Files ## Prerequisites - Your licence must include **CSV/ASCII Import**. - The source file must use a supported extension, such as `csv` or `txt`. - The file must be in a delimited text format that ATLAS can describe with the available settings. ## Configure a CSV or TXT format Open the ATLAS configuration/settings area and go to: **General > Text Session Format** This page maintains the text-file definitions, which is saved to: `%LOCALAPPDATA%\McLaren Applied Technologies\ATLAS 10\SQL Race\DataProviders.xml` ### Add a new file type 1. Click **New**. 2. Enter the file extension **without the leading dot**. Example: use `csv`, not `.csv`. 3. Fill in the format details for that file type. 4. Click **Set Definition** to save the definition. ATLAS only allows one definition per extension, so `csv` and `CSV` are treated as the same extension. ## Meaning of each format field | Field | Meaning | | --- | --- | | **Extension** | File extension to match, for example `csv` or `txt`. | | **Total Header Lines** | Total number of non-data lines at the top of the file. | | **Parameter Line** | Line number containing parameter/channel names. | | **Group Line** | Line number containing group names. Use `0` if the file does not have one. | | **Date Exists** | Whether the file includes date information. | | **Time Format** | One of `DateTime`, `Milliseconds`, `Seconds`, or `Time`. | | **Delimiter** | One of `Tab`, `Space`, `Comma`, or `Semicolon`. | | **Unit** | Line number containing units. Use `0` if units are not present. | | **Lap Marker** | Marker text used for lap markers in the file. | ## Load the file in ATLAS After the format definition is in place: 1. Open **Session Browser**. 2. Switch to the **Files** tab. 3. Navigate to the folder that contains the CSV or TXT file. 4. Double-click the file/session name to load it. If you use the same location frequently, you can add the folder to indexed sources or shortcuts from the **Files** tab context menu. ## Typical example For a comma-separated file with two header rows: | Field | Example value | | --- | --- | | Extension | `csv` | | Total Header Lines | `2` | | Parameter Line | `1` | | Group Line | `0` | | Date Exists | `True` or `False`, depending on the file | | Time Format | `Milliseconds` | | Delimiter | `Comma` | | Unit | `2` if units are on the second header row, otherwise `0` | | Lap Marker | `%` or whatever marker your file uses | ??? question "Troubleshooting" **The file does not appear or does not load** 1. **Is the licence correct?** Confirm that **CSV/ASCII Import** is included in your licence. 2. **Does the extension match?** The file extension must exactly match a definition in **Text Session Format**. 3. **Was the dot omitted?** The extension must be entered without a leading dot — `csv`, not `.csv`. 4. **Are you in the right tab?** The folder must be opened from **Session Browser > Files**. --- **Channels, units, or groups are wrong** 1. **Are the header line numbers correct?** Re-check **Total Header Lines**, **Parameter Line**, and **Group Line**. 2. **Is the delimiter right?** Confirm the delimiter matches the one used in your file. 3. **Is the time format right?** Confirm the chosen **Time Format** matches the data. 4. **Are optional lines set to zero?** Use `0` for any header line that does not exist in the file. --- **Save fails in Text Session Format** Each extension can only have one definition. Remove or rename any duplicate entries in the list before saving. --- # Changing Properties The Session Properties Panel in ATLAS provides detailed metadata for each Session. This information is essential for identifying, organising, and aligning Sessions for analysis. Some properties are editable, allowing users to customise or correct Session details, while others are read-only and automatically populated by the system. ## Session Identifier Macros Session Identifiers (aka _names_) help you identify recorded sessions. ATLAS provides formatting macros to fill in parts of the identifier automatically. You can include them in the identifier when configuring a recorder. !!! tip These macros can also be used in Session Details (metadata). ### Time Codes These codes format the session identifier from the time of recording: | Macro | Description | ------ | -------------------------------------------- | | `%y` | Year (Short form - 2 digits: 00 to 99) | | `%Y` | Year (Long form - 4 digits: 0000 to 9999) | | `%m` | Month (2 digits: 01 to 12) | | `%d` | Day of month (2 digits: 01 to 31) | | `%H` | Hour (2 digits: 00 to 23) | | `%M` | Minute (2 digits: 00 to 59) | | `%S` | Second (2 digits: 00 to 59) | !!! example `%y%m%d%H%M%S` = `210730140617` _at 2021-07-30 14:06:17 local time_ ### Session Details These codes incorporate standard session details (metadata): | Macro | Session Detail | ------ | -------------------------------------------- | | `%a` | _Car_ | | `%c` | _Circuit_ | | `%e` | _Engine_ | | `%r` | _Driver_ | | `%n` | _Session Number_ | | `%t` | _Race/Test_ | | `%O` | _Unit Data Source_ (populated automatically) | ### Parameter Data To extract parameter values, use `${param}$`, where `{param}` is the parameter identifier or name. This is commonly used together with text conversions — to spell out driver initials, for example. !!! example `$NDriverInitial1$$NDriverInitial2$$NDriverInitial3$_$NFOMRacingNo$` = `HAM_44` If the parameter is not present — for example, while configuration is being processed — the macro will not be expanded. ## Accessing the Session Properties Panel To view or edit Session properties: 1. Open the Session Browser. 2. Select a Session from the Session List. The Session Properties Panel will appear, displaying all available metadata for the selected Session. !!! tip Editable fields are shown in white. Read-only fields are shown in grey. ## Editable Session Properties These fields can be modified directly in the Session Properties Panel. Click on the value to edit it. **Property**|**Description** :-----:|:-----: Session Name|The name assigned to the Session. Automatically generated but can be renamed for clarity. Session Number|A numeric identifier. If the name ends in a number, it is incremented with each new recording. A blank entry is not incremented. Default is 1. Pit Lane Trigger|Defines the position of the pit lane trigger relative to the team pit box. Options: None, After Pit, Before Pit. Session Description|Free text field for describing the Session’s purpose or context. Driver|Name of the driver for the Session. Car|Identifier or name of the car used. Engine Description|Description of the engine configuration. Circuit|Name of the circuit or track. Race/Test|Indicates whether the Session was part of a race or a test. Chassis Number|Identifier for the chassis used. Note 1 / Note 2 / Extra Notes|Free text fields for additional comments or observations. Weather|Description of weather conditions during the Session. Version|Optional versioning field for tracking changes or configurations. Lap Trigger Offset|Distance (in metres) from the lap trigger to the Start/Finish line. Negative values = Start/Finish is ahead of the trigger. Positive values = behind. Used to align data across Sessions with different trigger positions. !!! tip Use consistent naming conventions and metadata values to make filtering and comparison easier across large datasets. ## Read-Only Session Properties These fields are automatically populated by ATLAS and cannot be edited. **Property**|**Description** :-----:|:-----: Unit Data Source|The Recorder that captured the Session data. Date of Recording|The date and time the Session was recorded. ECU Version|Version of the ECU (Engine Control Unit) used. ECU Serial Number|Serial number of the ECU. These values are critical for traceability and technical validation, especially when comparing Sessions across different hardware or configurations. **Special Field: Pit Lane Trigger** The Pit Lane Trigger setting affects how ATLAS calculates lap times and identifies the fastest lap. Options include: - None: Trigger is located at the team pit box. - After Pit: Trigger is after the pit lane. The last full lap may be the fastest and is included in analysis. - Before Pit: Trigger is before the pit lane. The in-lap is excluded from fastest lap calculations. This setting ensures accurate lap timing and should be configured based on the physical layout of the circuit. **Special Field: Lap Trigger Offset** The Lap Trigger Offset is used to align Sessions recorded with different lap trigger positions. It defines a consistent reference point on the track. - Negative values: Start/Finish line is ahead of the trigger. - Positive values: Start/Finish line is behind the trigger. This is especially useful when comparing data from multiple Sessions recorded at the same circuit but with different trigger setups. !!! note The reference point does not have to be the Start/Finish line. Any consistent track feature can be used, as long as it is applied uniformly across Sessions. ## Best Practices 1. Always review and update Session metadata after recording to ensure accuracy. 2. Use descriptive names and notes to make Sessions easier to identify later. 3. Align Sessions using Lap Trigger Offset when comparing data across different setups. 4. Use consistent values for fields like Driver, Circuit, and Car to improve filtering and searchability. --- # Database Connection Manager ## Database Connection Manager This guide explains how to manage database connections in ATLAS using the Database Connection Manager. You will learn how to add, remove, and view properties of database connections. For information on how to setup and configure SQL Server and SQL Race databases, please refer to the [setup guide](../../../../store/sqlrace/installation.md). ![Database Connection Manager](assets/database-connection-manager.png) ## Opening the Database Connection Manager You can open the Database Connection Manager in two ways: - Select **Tools > Database Connection Manager** from the Menu Bar. - Right-click **SQLRace Databases** in the Session Source panel within the Session Browser and select **Connection Manager**. ## Adding a Database Connection 1. Click the **Add** button. 2. Enter the required details: - **Server Name**: The server hosting the database. - **Database Name**: The name of the database. - **Friendly Name** (optional): A user-friendly name for the database. - **Root Path**: The folder location of the database file. 3. Click **Test Connection** to verify the details. 4. Click **OK** to save and enable the connection. ## Removing a Database Connection 1. Select the database connection you wish to remove. 2. Click the **Remove** button. 3. Confirm removal by clicking **Yes** or cancel by clicking **No**. ## Viewing Database Connection Properties 1. Select the database connection whose properties you want to view. 2. Click the **Properties** button to display the connection details. --- # Service Connection Manager !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Service Connection Manager The Service Connection Manager allows users with an RTA License to manage RTA hosts containing sessions in their indexed sources. This guide explains how to add, remove, edit, import, and export service connections. ## Opening the Service Connection Manager 1. Open the **Session Browser**. 2. Go to the **Indexed Sources** tab. 3. Click the **Add Data Source** button ( ). 4. Select **Service Connections**. ## Adding a Service Connection 1. Click the **Add** button. 2. Enter the following connection details: - **Friendly Name**: The name displayed in the Sources area of the Session Browser. - **URI**: The address of the RTA Session host. 3. If not using OAuth 2.0 Authorization: - Click **Test** to verify the connection. - If successful, click **OK** to save, or **Cancel** to discard. ![Service Connection Properties](assets/service-properties.png) 4. If using OAuth 2.0 Authorization: - Enable the OAuth 2.0 checkbox. - Fill in the required authentication fields. - Test the connection and click **OK** to save. Once the connection is set up, enable the **Indexed source** checkbox to retrieve sessions from the host. ## Importing and Exporting Service Connections - Use the **Import** and **Export** buttons in the Service Connection Properties to share connection files (`.svc`) with other users. ## Removing a Service Connection 1. Select the Service Connection to remove. 2. Click the **Remove** button. 3. Confirm by clicking **Yes**, or cancel by clicking **No**. ## Viewing and Editing Service Connection Properties 1. Select the Service Connection you want to view or edit. 2. Click the **Properties** button to access and modify the connection details. --- # Markers Markers define specific points or ranges of time within a session and are fully customizable. Each marker can include annotations and labels to describe its purpose. **Marker Properties:** - **Type:** Group/category of the marker. - **Label:** Short summary of the marker. - **Description:** Detailed explanation. - **Start Timestamp:** Optional start time (defaults to session start if omitted). - **End Timestamp:** Optional end time (defaults to session end if omitted). ![Markers](assets/markers.png) !!! tip To represent a single point in time, set both start and end timestamps to the same value. **Marker Labels:** Markers can have multiple labels for additional information. Label properties include: - **Name:** Computer-readable reference. - **Label:** Human-readable summary. - **Description:** Detailed explanation. - **Value:** Associated value. - **Format:** Display format (e.g., `%.3f` for three decimal places). - **Unit:** Measurement unit. Labels are optional and can be used in any combination. When reviewing markers in the timeline, labels appear in tooltips on hover. Example: ![Marker labels example](assets/markers-demo.png) ## Adding Markers Programmatically Markers can be added via code, such as in simulation models or analysis tools. Example using the SQL Race API: ```c# var marker = new Marker(markerStartTime, markerEndTime, "High Wind", "windSpeedData", "A period during which the wind was sustained above 9 m/s"); marker.AddLabel( new MarkerLabel { Label = "Average Speed", Description = "An average sustained wind speed during this time period", Value = "9.6382834", Format = "%.2f", Unit = "m/s", Name = "windSpeedDataHighVelocity01" }); marker.AddLabel( new MarkerLabel { Label = "Direction", Description = "The direction of the wind during this time period", Value = "323", Format = "%i", Unit = "deg", Name = "windSpeedDataHighDirection01" }); session.Markers.Add(marker); ``` > **Reference:** See the [Marker class documentation](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Marker.html). ## Using the Markers Tab The Markers tab in the Laps And Markers Editor displays markers for the active session. Users can: 1. **Add** a new marker 2. **Delete** the selected marker 3. **Go To** the selected marker 4. **Edit** the selected marker !!! note In LIVE mode, markers cannot be added, deleted, or edited. Editing is enabled once the session is HISTORICAL. ### Marker Types - **Point Marker:** Represents a single time point (only Start time set). - **Range Marker:** Represents a time range (both Start and End times set). ### Adding a Marker - Click **New** to enable editing. - By default, the Range checkbox is unticked (point marker). - Start time defaults to the current cursor time. - Edit the following fields: - **Label:** Marker name - **Description:** Optional - **Range:** Tick for range marker - **Start time:** Set as needed - **End time:** Enabled for range markers Edit times by typing, using arrows, or dropdown controls. Use the cursor button to set times to the cursor position. Reset button restores default/previous values. Click **Save** to add the marker to the list. ### Marker Selection and Management - **Delete:** Select a marker and press Delete to remove it. - **Go To:** Select a marker and press Go To to move the cursor (to the time point for point markers, or midpoint for range markers). - **Edit:** Select a marker, update details, and press Save to apply changes. Markers are also available in the Laps dropdown. Point markers show label and time; range markers show label and duration. !!! tip Transient markers can be added via Alarms (see Alarm Set Marker). --- # Laps In ATLAS, a Lap is a defined segment of time within a telemetry session—typically corresponding to a vehicle completing a circuit. But laps are not just for motorsport. They are a powerful structuring tool for any time-series data stream where repeated cycles or phases occur. Laps allow users to: - Segment sessions into meaningful intervals - Compare performance across repeated events - Anchor analysis to specific time windows - Drive automation in dashboards, alarms, and reports ![Laps](assets/laps.png) !!! tip In LIVE mode the dialog is showing the list of the laps recorded so far,but does not allow to make changes. The duration of the last lap will not be up to date in LIVE mode. As soon as the recording ends and session becomes HISTORICAL, the data will be up to date and editing will be enabled. **Timeline Session Details** From ATLAS 11.2.3.460 onwards, users can now add information from session details to the Timeline. The list of all session details is provided on the right-click context menu on the session in the Timeline. ![timeline-contextmenu.png](assets/timeline-contextmenu.png) The selected session details are visible as a tooltip by hovering over the specific session in the Timeline. ![timeline-tooltip.png](assets/timeline-tooltip.png) The number of session details that a user can add can be defined in Tools > Options > Plugins > F1 Timeline, the maximum number of session details allowed is 20. ## Why Use Laps? Laps are essential when: - You want to compare multiple runs of the same test or event - You need to isolate performance during specific phases (e.g. warm-up, cooldown, pit stops) - You’re building dashboards or reports that rely on consistent time windows - You want to automate workflows based on recurring patterns in the data **Key Capabilities** **1. Live Lap Tracking** During a live session, ATLAS can track laps in real time. This is especially useful in motorsport or test environments where engineers monitor performance lap-by-lap. - Laps are displayed as they are recorded - The most recent lap is updated continuously - Once the session ends, laps become editable !!! warning In LIVE mode, laps are read-only. Editing is only enabled once the session becomes HISTORICAL. **2. Manual Lap Definition** You can define laps manually using the Laps and Markers Editor. This is useful when: - The data source doesn’t emit lap markers - You want to redefine laps based on new criteria - You’re working with appended or stitched sessions Manual laps can be: - *Precise:* down to the millisecond - *Flexible: *placed anywhere in the session - *Named:* with custom labels like “Out Lap”, “Pit Entry”, “Cool Down” **3. Session Navigation** Laps act as bookmarks in your session. You can: - Jump to a lap instantly using the Go To button - Use laps to drive cursor synchronisation across displays - Filter displays or dashboards to show only data from a specific lap **4. Comparative Analysis** Laps are the foundation for overlaying and comparing telemetry data. For example: - Compare throttle traces across laps - Analyse tyre temperature evolution lap-by-lap - Identify anomalies or trends across repeated cycles ATLAS supports: - Multi-lap overlays - Delta comparisons - Statistical summaries across laps **5. Integration with Workbooks and Displays** Laps can be used to: - Drive conditional formatting (e.g. highlight fastest lap) - Trigger alarms or annotations in dashboards - Filter waveforms, histograms, or scatter plots to a specific lap - Laps are not just metadata—they are active drivers of visualisation and logic in ATLAS. **6. Out/In Lap Handling** ATLAS supports special lap types: - *Out Lap:* typically the first lap after leaving the pits - *In Lap:* the final lap before returning to the pits These are useful for: - Excluding non-representative data from analysis - Flagging laps with different operational contexts **7. Appended Sessions and Lap Continuity** When working with appended sessions (e.g. multiple test runs stitched together): - Laps are preserved per session - The first lap of each session is protected from deletion - You can still define new laps across the combined timeline This enables: - Longitudinal analysis across multiple runs - Consistent lap numbering even across session boundaries ## Advanced Use Cases for Laps in ATLAS Laps are not just a convenience for motorsport—they are a strategic structuring mechanism for any telemetry-driven workflow. Below are advanced scenarios where laps become critical to extracting value from ATLAS. ### Performance Benchmarking Across Sessions Laps allow engineers to: Compare performance across different test runs, even if they span multiple sessions or vehicles. Use lap overlays to visualise differences in throttle, brake, steering, or system behaviour. Identify repeatable patterns or outliers in performance. !!! example Compare Lap 3 from a baseline test with Lap 3 from a new configuration to quantify gains or regressions. ### Automated Report Generation Laps can be used as reporting units: - Generate lap-by-lap summaries of key metrics (e.g. max speed, average temperature, fuel usage). - Export lap segments for external analysis or compliance documentation. - Drive templated reports that automatically populate based on lap structure. !!! tip This is especially useful in endurance racing, fleet testing, or regulatory validation. ### Triggering Alarms and Events Laps can be used to: - Trigger alarms when a parameter exceeds a threshold during a specific lap. - Suppress alarms during out/in laps or cooldown phases. - Log events tied to lap transitions (e.g. pit entry, system reset, driver change). This enables context-aware alerting, reducing false positives and improving signal-to-noise ratio. ### Distance-Based Analysis Laps enable distance-domain visualisation: - Plot parameters against distance rather than time. - Compare how systems behave at the same physical location across laps. - Useful for track-based testing, proving grounds, or route-based validation. !!! example Compare suspension travel at the same corner across multiple laps. ### Session Stitching and Longitudinal Studies When sessions are appended: - Laps maintain continuity across stitched sessions. - Engineers can perform longitudinal analysis across days, drivers, or configurations. Useful for durability testing, regression analysis, or multi-driver comparisons. ### Custom Lap Logic via Functions Using ATLAS Functions (FDL), users can: - Define custom lap logic (e.g. laps based on temperature cycles, not distance). - Create virtual laps based on system states or external triggers. - Use laps as inputs to derived parameters or conditional displays. --- # Overview Parameters are physical properties that are measured during a Session. Typically, a parameter value is the conditioned signal output of a sensor. Each parameter generates data which is available for display. Parameters may be combined using Functions to give derived results. The output of a Function can be manipulated and displayed in the same way as a parameter. Each parameter has a set of properties associated with it that control how it is presented in a Display. These can be edited in the Parameter Properties panel. Functions also have Properties. Parameters are classified into Groups, aimed at simplifying their organization. Parameter Groups are defined in the data logging system and cannot be changed in ATLAS. Parameter groups are listed in the Parameter Browser. --- # Using Parameters ## Parameter Browser The Parameter Browser allows you to select and manage Parameters for the currently selected Display. The available Parameters depend on those recorded in the loaded Session. **Opening the Parameter Browser** - Click the Parameter Browser icon on the Browser Toolbar. - Select **View > Parameter Browser** from the Menu Bar. - Use the keyboard shortcut **P** with a display selected. - Right-click anywhere on a Display and choose **Parameter Selection** from the context menu. **Parameter Browser Layout** ![Parameter Browser](assets/parameter-layout.png) 1. **Application Groups Panel (Left)** - Browse and select Application groups to view their Parameters. - Expand/collapse groups using arrows or double-click. - Use "All Groups" if unsure where a Parameter is located. 2. **Search Criteria and Parameters List (Center)** - Search for Parameters by name or description. - Restrict search to Slow Row, Virtual Parameters, or Functions. - Add Parameters to the Display by double-clicking or dragging. - Select multiple Parameters with Ctrl/Shift + Click. - When comparing sessions, expand Parameters to see applicable sessions. 3. **Selected Parameters (Right)** - Shows Parameters added to the current Display. - Parameters with data are white; those without are greyed out. - Select a Parameter to highlight it on the Display. - Press **Delete** to remove a selected Parameter ## Changing Parameter Colour 1. Double-click the Parameter name on a Display to open the Parameter Properties window. 2. In the Appearance section, type a colour name or select from the dropdown. 3. Press **Return** or click to confirm the colour. ![Parameter Colour](assets/parameter-color.png) ## Parameter Aliasing (Changing Parameter Name) Parameter Aliasing lets you assign custom names to Parameters. - **Replace Mode:** Substitute original names with a new name in the same group. ``` vCar:Chassis Speed ``` - **Pick Mode:** Link a new name to one of several source names, with priority order. ``` Speed:Chassis CoverageSq:ATLAS GForce ``` Configure aliasing in `aliasing.config` saved at `%userprofile%\documents\McLaren Applied Technologies\`. !!! warning After making changes to `aliasing.config`, you must restart ATLAS for the new aliases to take effect. If a parameter (e.g., `pBrake`) is already present in your workbook and you create an alias (e.g., `BrakePressure`), `pBrake` will remain in the workbook but will no longer be searchable in the Parameter Browser. Only the alias name (`BrakePressure`) will be available for searching. To remove the old parameter name from your workbook, you need to delete it manually. ## Application Groups Applications organize Parameters into groups (e.g., Bios, Chassis, Engine) for easier navigation. Groups are shown in a tree format in the Parameter Browser. ## Number Formats Customize how Parameter values are displayed using format strings (similar to C language): - `[text]%[control][total digits].[decimal digits][format code]` - Control characters: `+` (show sign), `-` (left justify), `0` (leading zeros) - Format codes: `f` (float), `d` (integer), `x` (hex lower), `X` (hex upper) Set formats in the Appearance frame of the Parameter Properties panel. ## Parameter Properties Tool Window Manage Parameter appearance and behavior: - **Default:** Core properties from data acquisition. - **Changed:** Overrides for the current Display. - **Global:** Applies property value to all instances of the display type in the workbook. ![Parameter Properties](assets/parameter-tool-window.png) **Editing Properties** - Open the Parameter Properties tool window. - Click a property value to edit. - Right-click the icon next to a value for options: - Copy Name/Value - Paste Value - Show/Hide Description - Make Global - Clear Global Value - Reset to Default !!! tip Use the Parameter Browser and Properties tool window to fully customize how Parameters are displayed and behave in ATLAS. --- # Overview You can export sessions in multiple ways to suit your workflow. To export an entire session directly from SQL Race to an SSN2 file, simply right-click the session in the Session Browser and select **Export**. Make sure the session is not currently loaded into a Set before exporting. For more control—such as exporting specific parameters, laps, or using formats other than SSN2—use the [**Export Manager**](export-manager.md) found in the Tools menu. The Export Manager lets you create and manage export configurations (see example below). To export, select the desired session from the **Session** dropdown (showing sessions loaded via the Session Browser), choose an export configuration, and click **Export**. The **Status** field will display progress as a percentage and update to **Done** when the export is complete. ![Export Config](assets/export-config.png) --- # Export Manager The Export Manager helps you extract and save session data for analysis, reporting, or sharing. Use it to customize what data you export, how it's formatted, and where it's saved. ## Why Use the Export Manager? - **Flexible Data Export:** Choose exactly which session data and parameters you need. - **Multiple Formats:** Export to Text, Matlab, SSN2, or SQL Race Database. - **Reusable Configurations:** Save export settings for future use. - **Efficient Workflow:** Export directly from the Export Manager or the Export Data dialog. ![Export Manager 1](assets/export-manager-1.png) ## How to Use the Export Manager 1. **Open Export Data Dialog:** Start by creating a new Export Configuration. 2. **Select Session:** Pick the session to export from the drop-down list. Only sessions loaded via the Session Browser are available. 3. **Choose Parameters (Execution Tab):** - Select specific parameters or use the "All Parameters" checkbox. - Filter by application group, channel type, or search by name. - Selected parameters appear in a list below. ![Export Manager 2](assets/export-manager-2.png) 4. **Set Timing (Timing Tab):** - Choose laps to export or specify a custom time range. - Note: Custom time ranges work for text and Matlab exports. For composite sessions, only the first session is exported. ![Export Manager 3](assets/export-manager-3.png) 5. **Select Export Format:** - Pick your desired format (Text, Matlab 1D/2D, SSN2, SQL Race DB). - Format-specific options (e.g., column separators for Text) will be enabled as needed. 6. **Configure Destination:** - For SQL Race Database, select from available databases or add new connections via the Database Connection Manager, then set the **Session Identifier** (defaults to the selected session name). - For other formats, specify the file path. 7. **Auto Increment:** - All export types support automatic version numbering to the file name or session identifier for SQL Race. When enabled and an export with that name already exists, the system appends `_1`, and increments the number for subsequent duplicates. ![Export Manager 4](assets/export-manager-4.png) 7. **Save and Export:** - Save your configuration for future use. - Click "Export" to run the export immediately. Progress is shown next to the Export button. Use the Export Manager to streamline your data export process and ensure you get the data you need, in the format you want. ### Removing Team Security As of version 11.4.4.371, Export Manager now allows users to remove team security during export. This enables sharing sessions with trusted external parties (e.g., suppliers) using a generic ATLAS license. !!! info - Consortium selection only affects who can open the exported session. As a member of the original consortium, you can choose how data is restricted during export. - The exported session will only include parameters you have access to under your current license. - RDA restrictions always remain applied, regardless of consortium changes or “No Consortium” selection. - This feature is only available when exporting to SQLRace or SSN2 formats. ![Export Manager Consortium Options](assets/team-restrictions.png) **How to Remove Team Security During Export:** 1. Open Export Manager and create a new export configuration. 2. Select the session(s) you want to export. 3. Choose the export format: SSN2 or SQLRace. 4. Select consortium restriction options: - Unchanged – Keeps the current consortium setting from the selected session. This option will change based on the session chosen. - No Consortium – Removes all team restrictions. The exported session can be opened by any ATLAS license. - Specific Consortium – Locks the export configuration to one consortium only, even if you later change the session. 5. Confirm export options: - If “All Parameters” is selected, RDA-locked parameters will not be exported. 6. Click Export or Save. --- # Overview Functions in ATLAS are user-defined calculations that operate on Session data. They allow users to extend the capabilities of ATLAS by performing custom computations, automating data processing, and enhancing data visualization. **Key Features:** - **Custom Calculations:** Functions enable users to create tailored calculations beyond built-in parameters. - **Flexible Data Analysis:** By using functions, users can manipulate and analyze data according to specific requirements. - **Integration:** Functions can be used in Displays, similar to Parameters, and support Parameter Properties for dynamic data representation. - **C# and MATLAB Functions** As well as functions written in FDL, it's possible to use existing C# and MAT functions without translation. ## How do functions work? - Functions are written in the Function Definition Language (FDL) using the Function Editor. - They are stored in SQLRace for persistence and reuse. - Functions can incorporate expressions, statements, processors (pre-defined functions), DLLs (external code), constants, and comments for documentation. ## Why use functions? - To automate repetitive calculations. - To implement complex logic and data transformations. - To integrate external code via DLLs for advanced functionality. - To improve clarity and maintainability of data processing workflows. --- # Function Editor The Function Editor in ATLAS allows users to create, edit, and view Functions, which are used similarly to Parameters for data manipulation and display. Functions are written in the Function Definition Language (FDL) and can reference Parameters, Constants, other Functions, and Processors. ![Function Editor](assets/function-editor.png) ## Quick Start — Creating Your First Function 1. Open the Function Editor and click New. Default General and Function properties are prefilled. 2. Set General Properties (name, format, units, display min/max). These act like parameter properties for Displays. 3. Insert inputs from the Function Parameter Browser (use search; press Enter or double‑click to insert). You can include Parameters, Constants, Processors, or existing Functions. 4. Write FDL in the Function Definition panel (see sections below). 5. Build `(F6)` to check syntax. Success and errors appear in Function Build Output. 6. Save the Function; it becomes available like any Parameter in Displays. !!! tip If your environment does not have Resolve without App Name enabled (Options ▸ Data), use the full parameter identifier (e.g., vCar:Chassis) rather than a short name. ## Opening the Function Editor - Click the **Function Editor** button on the Browser Toolbar. - Select **View > Function Editor** from the Menu Bar. ## Function Editor Panels 1. **Function Browser**: Lists existing functions. Select to view properties; double-click to edit code. 2. **Properties**: Displays and allows editing of general and function-level properties. 3. **Function Definition**: Central text editor for writing function code. Each opened function appears in a new tab. 4. **Function Build Output**: Shows build progress, status, and errors. 5. **Function Parameter Browser**: Search and add Parameters, Constants, and Functions to your function. ## Editing Function Properties ### General Properties - **Parameter Name**: Required. Used as the function's identifier. - **Description**: Editable only in Function Editor. - **Identifier**: Unique, read-only, auto-updated. - **Format**: Output format (default `%5.2f`). - **Units**: Display units (optional). - **Display Minimum/Maximum**: Axis scaling (default 0.0/1000, required). ![General Properties](assets/general-properties.png) ### Function Properties (how and when a Function calculates) When a Function uses multiple inputs sampled at different times, choose how often and where to compute results: **Function Mode** - Instantaneous (On‑the‑fly) — evaluates at every sample point of every input; results are not stored. Default for non‑historical Functions. - Coverage cursor based — computes from Session start to the coverage cursor; results are stored. Default for historical Functions. - Leading edge — computes once at the leading edge, skipping gaps. **Calculation Mode** - Each sample point - Fixed frequency (set Calculation Frequency (Hz) explicitly) - Highest rate **Additional options** - Calculate Over Whole Session — compute across the entire Session. - Interpolate Between Samples — interpolate values off the current sample; if off, uses most recent sample. - Join Gaps Around Null — joins gaps for instantaneous Functions during recording (no effect on historical data). ![Function Properties](assets/function-properties.png) ## Using the Function Parameter Browser - Search for Parameters, Constants, or Functions from loaded sessions. - Filter results by All Groups, Processors, or Constants. - Add items to your function by double-clicking or pressing Enter. ![Parameter Browser](assets/parameter-browser.png) ## Function Build Output - Displays build success or error messages. - Syntax errors show line and position for troubleshooting. - Execution errors are logged and disable the function until resolved. ![Build Output](assets/build-output.png) ## Writing Functions in FDL **Parameters** Insert Parameters in code with the $ prefix (e.g., $Vcar). Type $ then Ctrl+Space to list or filter available items. You may also insert via the Function Parameter Browser. If Resolve without App Name is off, use full identifiers (e.g., $vCar:Chassis) **Variables and Arrays** Define variables by assignment—no declarations needed; scope is per Function. Array subscripts can be expressions. Names can include letters, digits, and underscores, and must not clash with a Constant name. ``` for (i = 0; i < 3; i++) { cyl[i] = 0; } ``` **Expressions and Operators** Expressions compute values using operators with a defined precedence (compatible with POSIX bc). Arithmetic (+ - * / % ^), relational (< <= > >= == !=), boolean (! && ||), bitwise (& ~ | >> <<), unary minus, and increment/decrement forms are supported. Parentheses () force evaluation order. Compound assignments (op=) evaluate the left side once (useful with arrays). **Statements and Control Flow** Statements control the sequence of evaluation of Expressions. | Statement | Description | |--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `if (expression) statement1 [else statement2]` | Evaluates `expression`. If non-zero, executes `statement1`; if zero and `else` is present, executes `statement2`. The `else` clause is optional. | | `while (expression) statement` | Executes `statement` repeatedly while `expression` is non-zero. Evaluates `expression` before each iteration. Loop ends if `expression` is zero or a `break` is executed. | | `for ([expression1]; [expression2]; [expression3]) statement` | Controls repeated execution of `statement`. `expression1` initializes the loop (first pass only). `expression2` is checked before each iteration; if non-zero, executes `statement`. After each iteration, `expression3` is evaluated. If `expression2` is missing, it defaults to `1`. | | `break` | Causes a forced exit from the most recent enclosing `while` or `for` loop. | | `continue` | (Extension) Causes the most recent enclosing `for` loop to start the next iteration. | | `return(expression)` | Returns the value of `expression` from a function. Always return a value from the first executed function, as Displays use it. | | `= null` | `null` is a special value meaning "not a number". Displays will not plot or calculate with `null` values. | > Parentheses `{ }` define a compound statement, allowing multiple statements to be grouped for execution within another statement. > Newlines (or `;`) separate statements; you can hide a newline with `\\`. !!! note "null means “not a number”" Displays don’t plot or compute with null. Always return a value in the first executed Function (because Displays use it). **Comments** Use `/* ... */`. Comments act as delimiters and can span lines; don’t split identifiers with comments. **Referencing other functions** You can call previously saved Functions by name followed by (), e.g., `FnSlip()`. Functions are pre‑calculated when a Display opens; Live inputs update incrementally. For more information on available expressions in FDL view the [Processing Guide](processing.md). ## Working with Constants Constants live outside Functions (global scope) in default_constants.txt: `C:\Users\[username]\Documents\McLaren Electronic Systems\ATLAS 9\Config` They can be numeric or string and are ideal for vehicle characteristics you reference across multiple Functions. If a Constant used by a Function is missing, the interpreter raises an execution error. **Constant Editor (open from Function Editor ▸ View ▸ Constants)** - Add — enter Name and Value (string/number) and click Add. - Update — select, edit Name or Value, then Update (changing Name creates a new Constant). - Delete — select and Delete (confirm when prompted). ![Constants](assets/constants.png) **Types of Constants** - Default — global, stored in default_constants.txt. - Session — stored within a specific Session; select Session type, then the Set and Session to view. ## Error Handling - **Syntax Errors**: Detected during build; must be resolved before use. - **Execution Errors**: Occur during function run; logged and disable the function. Errors can also originate from Constants (missing/invalid) and Processors (e.g., invalid interpolation files). --- # Processors Processors are compiled pre-defined routines. They provide functionality that would be hard or slow to implement in FDL. ## Trigonometric Calculated in _radians_: * `acos(x)` * `asin(x)` * `atan(x)` * `atan2(x,y)` _(arc tan of `y/x`)_ * `cos(x)` * `cosh(x)` * `sin(x)` * `sinh(x)` * `tan(x)` * `tanh(x)` ## Mathematical Simple scalar calculations: * `abs(x)` * `ceil(x)` * `exp(x)` * `floor(x)` * `ln(x)` _(natural logarithm)_ * `log(x)` _(base 10 logarithm)_ * `sqrt(x)` Calulated against parameters with respect to time (even in distance-mode): * `derivative(param)` * `integral(param)` ## Logical `not(x)` _or_ `!x` : Returns `TRUE` if `x` is 0, otherwise `FALSE`. `IsInRange(a, b, epsilon)` : Returns `1` if `a` is in range of `b`. `IsGT(a, b)` : Returns `1` if `a` is greater than `b`, otherwise returns `0`. `IsGE(a, b)` : Returns `1` if `a` is greater than or equal to `b`, otherwise returns `0`. `IsLT(a, b)` : Returns `1` if `a` if less than `b`, otherwise returns `0`. `IsLE(a, b)` : Returns `1` is `a` is less than or equal to `b`, otherwise returns `0`. ## Interpolation `BiLinearInterpolate(file, x, y)` : Returns a bi-linearly interpolated result corresponding to `x`, `y` from `file`. * `x` and `y` are parameter names (e.g. `$vCar`) or expressions * `file` is the path to a `.3d` file in the format specified below. ??? "Data file format" The filename must have the extension `.3d` The content is ASCII text, representing the values of data points in a grid. Comment lines are prefixed with `#`. Grid rules: * The grid must be fully populated and rectilinear but need not be regular i.e. there may be irregular intervals between ordinates but each intersection of an ordinate must have a data value; * There must be at least 4 data points in the grid; * Entries must be ordered within a row such that column range values increase to the right and within a column such that row range values increase to the bottom; * Entries on the same line must be separated with tabs or commas; * Both input values must lie within the grid. If any of these rules are broken, `BiLinearInterpolate(...)` returns a result of `0` and a warning is displayed when ATLAS first attempts to use the processor. | Line | Entry | Description | | ------- | ---------------------------------- | -----------------------------------------------------------------------| | 1 | _blank , X1, X2, …, Xi_ | _Xi_ = value of column _i_ ordinate | | 2 | _Y1, Z1,1, Z2,1, … , Zi,1_ | 1st row of table (_i_ columns × _j_ rows),
_Yi_ = value of row _j_ ordinate
_Zi,j_ = data value at intersection of column _i_ and row _j_ | | 3 | _Y2, Z1,2, Z2,2, … , Zi,2_ | 2nd row of table | | … | … | 3rd and successive rows of table | | _n_ | _Yj, Z1,j, Z2,j,... , Zi,j_ | last row of table | ???+ example This file… ``` # Example 3d grid ,5,7.5,10 0,1,2,3 10,4,5,6 25,7,8,9 30,10,11,12 ``` … creates this table:
Input X Range
5 7.5 10
Input Y Range 0 1 2 3
10 4 5 6
25 7 8 9
30 10 11 12
???+ info "Formulae" If _Xi <= `x` <= Xi+1_ and _Yj <= `y` <= Yj+1_ then _Za = Zi,j + ( (`x` - Xj) * (Zi+1,j - Zi,j) / (Xi+1 - Xi) )_ _Zb = Zi,j+1 + ( (`x` - Xj) * (Zi+1,j+1 - Zi,j+1) / (Xi+1 - Xi) )_ _result = Za + ( (`y` - Yj) * (Zb - Za) / (Yj+1 - Yj) )_ `LinearInterpolate(file, x)` : Returns a linearly-interpolated result corresponding to `x` from `file`. * `x` is a parameter name (e.g. `$vCar`) or expression * `file` is the path to a `.3d` file in the format specified below. If `x` under or over the limits of the data set then a result extrapolated from the lowest or highest two data points is returned. ??? "Data file format" The filename must have the extension `.2d` The content is ASCII text, representing the values of data points in a grid. Comment lines are prefixed with `#`. Grid rules: * There must be at least two pairs of data but there can be as many as required; * Entries must be ordered with lowest input, first; * Entries on the same line must be separated with tabs or commas; * No consecutive pair of inputs may be the same (to prevent divide by zero errors when calculating _m_). If any of these rules are broken, `LinearInterpolate(...)` returns a result of `0` and a warning is displayed when ATLAS first attempts to use the processor. | Line | Entry | Description | | ------- | --------- | -----------------------------------------------------------------------| | 1 | _Y1, Z1_ | 1st pair (_Y_ = input value, _Z_ = resulting data value); required | | 2 | _Y2, Z2_ | 2nd pair; required | | … | … | 3rd and successive pairs; optional | | _n_ | _Yn, Zn_ | last pair | ???+ example This file… ``` # Example 2d grid 0,1 10,4 25,7 30,10 ``` … creates this table: | Input range | Data values | | ------------- | --------------| | **0** | 1 | | **10** | 4 | | **25** | 7 | | **30** | 10 | ???+ info "Formulae" * If `x` is between Yi and Yi+1 then _result = mi*(`x` – Yi) + Zi_ * If `x` is below Y1 then _result = m1*(`x` - Yi) + Z1_ * If `x` is above Yn then _result = mn-1*(`x` - Yn) + Zn_ Where mi = (Zi+1 - Zi)/(Yi+1 - Yi) ## Filtering !!! note `param` must be a parameter from a data channel or another function, not the result of a calculation: `Filter(param,freq)` : Calculates the smoothed value of the parameter. This is a low-pass filter with a cut-off frequency of `freq`. `HighPassFilter(param,freq)` : Calculates the smoothed value of the parameter. This is a high-pass Filter with a cut-off frequency of `freq`. `DigitalFilter1(param, B1, B2, A1, A2)` : Implements a 1st order digital filter using the difference equation _`A1`*y(n) = `B1`*x(n) + `B2`*x(n-1) - `A2`*y(n-1)_ `DigitalFilter2(param, B1, B2, B3, A1, A2, A3)` : Implements a 2nd order digital filter using the difference equation _`A1`*y(n) = `B1`*x(n) + `B2`*x(n-1) + `B3`*x(n-2) – `A2`*y(n-1) - `A3`*y(n-2)_ `DigitalFilter3(param, B1, B2, B3, B4, A1, A2, A3, A4)` : Implements a 3rd order digital filter using the difference equation _`A1`*y(n) = `B1`*x(n) + `B2`*x(n-1) + `B3`*x(n-2) + `B4`*x(n-3) - `A2`*y(n-1) - `A3`*y(n-2) - `A4`*y(n-3)_ !!! caution For all 3 types of digital filter, it is your responsibility to ensure the function is evaluated at the appropriate frequency for the filter coefficients. The _Fixed Frequency_ calculation mode is recommended to ensure that changes to the parameters sampling rates do not affect filter results. `MedianFilter(param,n)` : Calculates the median filter value in a data window of _n/2_ samples either side of the current data point. `MedianFilter9(param)` : Calculates the median filter value of a data window 4 points either side of the current data point. The effect is to remove spikes from the data, where their width is less than 4 data-points. `MedianFilter17(param)` : As above, but the data window is 17 points wide, enabling spikes to be filtered out up to 8 data-points wide. `MedianFilter25(param)` : As above, but the data window is 25 points wide, enabling spikes to be filtered out up to 12 data-points wide. `PhaseCompFilter(param, freq)` : Calculates a phase-compensated filter which does not lag behind its input parameter. The filter has a cut-off frequency of `freq`. The phase-compensated filter does not respond well at very low cut-off frequencies, especially for pre-calculated functions. In this case, the data is filtered at a higher cut-off frequency. ## Special These perform special operations on data — such as returning information about a recorded parameter — or special actions — such as writing log output. !!! note `param` must be a parameter from a data channel (i.e. not the result of a calculation). `BitOfWord(param, bit)` : Returns `1` if bit number bit in the parameter `param` is `1`, otherwise returns `0`. Bits count from `0` and from least significant to most significant. `ClearLog(logname)` : Clears the contents of the file path `logname`. `CurrentLapNumber()` : Returns the current Lap number. `IsFirstPass()` : Returns `TRUE` the first time that the function is used to evaluate a term and returns `FALSE` for successive passes. Reset when the function is completed. `IsInError(param)` : Returns `TRUE` if the parameter value is in error for that parameter (iIndicates that the sensor may be out of range). `IndexBySample(param, n)` : Returns the value of parameter `param`, `n` samples forward or backward (`n` –ve). `IndexByTime(param, t)` : Returns the value of parameter `param`, `t` seconds forward or backward (`t` –ve). `LapBestFit(x, y, Or, Os)` : Returns the best fit line of parameters `x` and `y` with nth order of `Or` and coefficient index of `Os` (i.e. `0` = offset, `1` = slope etc.) Returns coefficients during current lap. `LapMax(param)` : Returns the maximum value of parameter `param` during the current lap. `LapMean(param)` : Returns the mean value of parameter `param` during the current lap. `LapMin(param)` : Returns the minimum value of parameter `param` during the current lap. `LapStdDev(param)` : Returns the standard deviation of parameter `param` during the current lap. `LapTime(lap)` : Returns the lap time in seconds of the given lap `lap`. `LinearRegressionOffset(x, y)` : Returns Linear Regression Offset for parameters `x` and `y` during current lap. `LinearRegressionR2(x, y)` : Returns Linear Regression R2 for parameters `x` and `y` during current lap. `LinearRegressionSlope(x, y)` : Returns Linear Regression Slope for parameters `x` and `y` during current lap. `SamplesMax(param, n)` : Returns the rolling maximum value of parameter `param` over the previous n samples. `SamplesMin(param, n)` : Returns the rolling minimum value of parameter `param` over the previous n samples. `SessionMax(param)` : Returns the maximum value of parameter `param` during the whole session. `SessionMean(param)` : Returns the mean value of parameter `param` during the whole session. `SessionMin(param)` : Returns the minimum value of parameter `param` during the whole session. `SessionStdDev(param)` : Returns the standard deviation of parameter `param` during the whole session. `SetEvent(n, d, g, p, v)` : Sets an Event at a given time. * `n` — Name * `d` — Description * `g` — Group * `p` — Priority (0 -2) * `v` — Volatility (0,1) `SamplesPerSec(param)` : Returns the sample rate in Hz being used to sample the parameter. `TimeIntoLap()` : Returns the time in seconds from the start of the current lap in the Active Layer. `TimeIntoSession()` : Returns the time in seconds from the start of the current session in the Active Layer. `TimeOfDay()` : Returns the time in seconds from midnight for the Active Layer Time Position. `UwLapMean(param)` : Returns the unweighted Mean value of parameter `param` during the current Lap. `UwLapStdDev(param)` : Returns the unweighted Standard Deviation value of parameter `param` during the current Lap. `UwSessionMean(param)` : Returns the unweighted Mean value of parameter `param` during the Session. `UwSessionStdDev(param)` : Returns the unweighted Standard Deviation value of parameter `param` during the Session. `WriteToLog (‘logname’, entry)` : Appends a line to the file `logname` with the text: `FunctionName` & ++tab++ & `entry` * `FunctionName` = the name of the function in which the processor was executed. * `entry` = a string enclosed in quotes or an expression which can be evaluated to a numeric value which is converted to a string representing the value to 8 or more digits. ## Operator Order of Precedence * `||` boolean operator _(left associative)_ * `&&` boolean operator _(left associative)_ * `!` boolean operator _(non-associative)_ * `<`, `<=`, `>`, `>=`, `==`, `!=` relational operators _(left associative)_ * `=` assignment operator _(right associative)_ * `+` and `-` operators _(left associative)_ * `*`, `/` and `%` operators _(left associative)_ * `^` operator _(right associative)_ * unary `-` operator _(non-associative)_ * `++` and `--` operators _(non-associative)_ ## Mathematical Operators `-` _expr_ : The result is the negation of the expression. _expr_ `+` _expr_ : The result of the expression is the sum of the two expressions. _expr_ `–` _expr_ : The result of the expression is the difference between the two expressions. _expr_ `*` _expr_ : The result of the expression is the product of the two expressions. _expr_ `/` _expr_ : The result of the expression is the quotient of the two expressions. _expr_ `%` _expr_ : The result of the expression is the remainder of the division. `^` _expr_ : The result of the expression is the value of the first expression raised to the second expression. The second expression must be an integer. `(` _expr_ `)` : Alters the standard precedence to force the evaluation of the expression, e.g. a + b / c is not the same as (a + b) / c. ## Variable Operators `var` refers to either a simple or an array variable. A simple variable is just a name; an array variable is specified as `name[expr]` (array subscript may be an expression). `++`_var_ : The variable is incremented by one and the new value is the result of the expression. `--`_var_ : The variable is decremented by one and the new value is the result of the expression. _var_`++` : The result of the expression is the value of the variable and the variable is then incremented by one. _var_`--` : The result of the expression is the value of the variable and the variable is then decremented by one. _var_ `=` _expr_ : The variable is assigned the value of the expression. _var_ **_op_**`=` _expr_ : This is equivalent to _var_ `=` _var_ **_op_** _expr_ with the exception that the _var_ part is evaluated only once. This can make a difference if var is an array. **_op_** is one of the mathematical operators `+`, `-`, `*`, `/`, `%` or `^`. ## Relational Operators Relational expressions evaluate to either `0` (if the relation is false) or `1` (if the relation is true). These may appear in any legal expression. _expr1_ `<` _expr2_ : The result is `1` if _expr1_ is strictly less than _expr2_. _expr1_ `<=` _expr2_ : The result is `1` if _expr1_ is less than or equal to _expr2_. _expr1_ `>` _expr2_ : The result is `1` if _expr1_ is strictly greater than _expr2_. _expr1_ `>=` _expr2_ : The result is `1` if _expr1_ is greater than or equal to _expr2_. _expr1_ `==` _expr2_ : The result is `1` if _expr1_ is equal to _expr2_. _expr1_ `!=` _expr2_ : The result is `1` if _expr1_ is not equal to _expr2_. ## Boolean Operators The result of all boolean operators is either `0` or `1` (for `false` and `true`), as in Relational expressions. Any non-zero value is treated as `1`. `!` _expr_ : The result is `1` if _expr_ is `0`, otherwise `0`. (`NOT`) _expr_ `&&` _expr_ : The result is `1` if both expressions are non⌀zero, otherwise `0`. (`AND`) _expr_ `||` _expr_ : The result is `1` if either expression is non⌀zero otherwise `0`. (`OR`) ## Bitwise Operators `~`_expr_ : Bitwise `NOT` _expr_ `&` _expr_ : Bitwise `AND` _expr_ `|` _expr_ : Bitwise `OR` _expr_ `>>` _expr_ : Shift right _expr_ `<<` _expr_ : Shift left --- # Using DLLs ## Advanced: Using DLLs For complex logic, you can write functions in **.NET languages** like C# and import them as DLLs. ### Key Requirements - Implement `IDotNetFunction` interface. - Place DLLs in: `%USERPROFILE%\Documents\McLaren Electronic Systems\SQL Race\Functions` - Restart ATLAS to register. ## Constants Constants are reusable values defined outside functions. - Stored in `default_constants.txt` - Managed via the **Constants Editor** - Types: **Default (global)** and **Session-specific** Use them to avoid hardcoding values like gear ratios, thresholds, or vehicle dimensions. ## Testing & Debugging ### Building Functions - Press `F6` or click **Build Function**. - Errors appear in the **Build Output** panel. ### Refreshing Calculated Parameters - Go to `Tools > Refresh Calculated Parameters`. - Choose **Workbook** or **Page** scope. !!! warning Avoid full refreshes on live data to reduce CPU load. ## Best Practices - Use **Fixed Frequency** mode for digital filters. - Use **comments** (`/* ... */`) to document logic. - Use **SessionMean()**, **LapMax()**, etc. for performance analytics. - Avoid overloading live sessions with too many functions. --- # Overview Alarms in ATLAS let you define conditions over live or historical data; when a condition is met, ATLAS takes one or more actions (e.g., log, warn, pop‑up, or set a marker). Alarms evaluate on every incoming sample of the parameter(s)/event(s) you specify. Alarms can also be defined in the Alarms.xal file ## Key Concepts Alarm groups contain related alarms, and all alarms in a group can be disabled in one go at the group level. An alarm is defined where a condition becomes true. Alarms can be run on: - Live: Runs continuously as telemetry is received. - Manual: Runs on-demand over a historical session. - Load Session: Execute the alarm automatically on session load !!! note Alarms triggered on `load session` will not execute if sessions have been loaded as part of a workbook. An alarm must be part of an alarm group. --- # Using Alarms ## Creating a new alarm On the alarm itself you can define behaviour, conditions and actions. To create a new alarm go to `Tools -> Alarm Editor` and create or select an alarm group and then click `New Alarm`. ![Alarm Editor](assets/alarm-summary.png) You can double click an alarm group to change which sets are active for the alarms in that group. This means that even if the condition of an alarm is met, it will not trigger from the parameters in that set if the group is not active for that set. ![Alarm Sets](assets/groups.png) ## Alarm Properties ### Alarm Conditions A parameter condition has a primary (required) and a secondary (optional) condition. The primary condition is the main condition that must be met for the alarm to trigger. The secondary condition can be used to add additional criteria that can also be satisfied for the alarm to activate. An event condition is based on an event ID and does not have primary or secondary conditions. **Primary Conditions** `Parameter + Operator + Value` **Secondary Conditions** `Logical Operator + Parameter + Operator + Value` **Event Condition** `Event ID` ![Alarm Editor](assets/alarm-editor.png) ### Alarm Actions Actions are what happens when an alarm is triggered. **Warn** If Warn is selected as an action, the Status Bar shows a running count and the last five raised alarms. ![Alarm Warning](assets/warning.png) **Audio Warning** If Audio Warning is selected as an action, a sound plays when the alarm triggers. You can configure the sound as None, Text-to-Speech, or Sound effect. **Pop-up** If PopUp is an action, a dialog displays when the alarm triggers. Pop‑ups are configured in Options → Plugins with three modes: Per Alarm (reuse per alarm), Per Occurrence (new for each trigger), Shared (one shared dialog for subsequent triggers). ![Alarm Pop-up](assets/popup.png) **Log** If Log is an action, entries are written to Alarm.log. **Set Marker** If Set Marker is an action, ATLAS places a transient point marker at the trigger time/distance, labeled with the Alarm Name and using the Comment as description. These markers are not persisted and are removed when the session unloads; you can view them via the Graphical Timeline dropdown and in [Laps and Markers Editor](../handling-data/laps-markers/markers.md). ![Alarm Marker](assets/markers.png) ## Monitoring Each triggered alarm will log in the Summary, listing priority, time, name, set, session and lap. Selecting an alarm in the Summary will highlight the trigger time in the session. ![Alarm Summary](assets/summary.png) --- # Overview ATLAS Viewer displays are visual components that show telemetry data from your sessions. Each display specialises in presenting data in a specific format — from time-series waveforms to numeric readouts, bar gauges, scatter plots, and more. You can add multiple displays to a workbook page, dock or float them, and save layouts in workbooks. ![Multiple Displays on a Page](assets/workbook.png) !!! info "20+ Display Types" ATLAS includes over 20 display types, each optimised for specific analysis tasks. Choose the right display for your data visualisation needs. ## Adding a Display === "Quick Access" Press `Ctrl+Q` twice, type the display name, press `Enter` :zap: **Fastest** — searchable, works from anywhere === "Display Toolbar" Click the display icon on the toolbar. Hover to see display names. :mouse: **One click** when you know the icon === "Menu" **File** > **New** > **Display** and select the display type :books: **Browse** all available displays !!! tip "Pro Tip" Use `Ctrl+Q` twice for fastest access. It's searchable and works from anywhere! ## Display Comparison | Display Type | Purpose | Key Features | Max Parameters | |--------------|---------|--------------|----------------| | [**Waveform**](waveform.md) | Multi-parameter time/distance traces | Interactive legend, reference cursor, autoscale, event markers, live telemetry | 100 (default, adjustable) | | [**Scatterplot**](scatterplot.md) | 2D plot of Y vs X, optional Z colour mapping | Best-fit/reference lines, up to 5 parameter sets, custom draw styles | 24 | | [**Histogram**](histogram.md) | Distribution of a single parameter | Spectral/cumulative modes, adjustable bins, auto-refresh | 1 | | [**FFT**](fft.md) | Frequency spectrum of a single parameter | Window functions, dB scale, peak markers, zoom | 1 | | [**Loadmap**](loadmap.md) | Pseudo-3D heatmap of time in parameter ranges | Adjustable buckets, colour bar, lap refresh | 2 | | [**Surface Map**](surface.md) | Interactive 3D mesh graph | Rotate/zoom, axis/grid customisation, requires .3d file | 2 | | [**Numeric**](numeric.md) | Instant numeric readouts | Auto-sizing text, colour thresholds, grid layout | 150 | | [**Bar**](bar.md) | Vertical gauge bars with numeric values | Real-time updates, custom refresh/background | 16 | | [**Block Bar**](block.md) | Minimalistic single-parameter bar | Conditional/gradient colouring, orientation options | 1 | | [**Bit**](bit.md) | Status bits with custom colours/labels | Boolean/status flags, .bcg file support, layout options | 20 | | [**Summary**](summary.md) | Lap-based statistics table or plot | Min/Max/Mean per lap, resizable columns, snapshot export | Multiple | | [**Circuit**](circuit.md) | Track map with car position | Sectors/segments, shaded timebase, circuit editor | 1 | | [**PCU Dash**](pcu.md) | Steering wheel display simulator | Multiple dash types, auto-load config | Default only | | [**Event**](event.md) | ECU events with priority and snapshots | Priority filtering, masking, row colouring | 100 | | [**Error**](error.md) | Live ECU error monitoring | Status tracking, filtering, double-click to jump | N/A | | [**Notepad**](notepad.md) | Free-text session notes | Custom fonts/colours, snapshot/copy | N/A | | [**Bing Map**](map.md) | Satellite map of GPS location | Zoom/scale, GPS formats, lock map centre | N/A | | [**Web Browser**](web.md) | Embedded browser | Live feeds/web content, show/hide controls | N/A | | [**Countdown Timer**](countdown.md) | Countdown clock or time offset | Custom text, font/colour options | N/A | | [**Video**](video.md) | Synchronised video playback | In-car camera, event correlation | N/A | ## Switching Between Sessions If you have multiple sessions loaded (Compare Sets): | Action | Shortcut | Result | |--------|----------|--------| | Switch selected display | `Shift+` | Changes active display only | | Switch entire page | `Ctrl+` | All displays on page switch together | | Click coloured tag | Mouse click | Menu shows available sets | When a whole page shows the same set, a coloured line appears across the page header. ## Display Tools ### Display Properties Every display has a **Display Properties** panel (press `D` or right-click > **Display Properties**) for formatting options specific to that display type. Properties can be: - **◻️ Square**: Default value. Reset to default with the reset button. - **♦️ Diamond**: Changed for this instance only. - **⚪ Circle**: Global setting — affects all future instances of that display type. ![Display Properties example](assets/properties.png) Right-click any property to make it global, clear global assignment, or reset to default. The meaning of each property is shown at the bottom of the Display Properties window when selected. !!! warning "Global Properties" Global properties affect ALL future displays of that type. Use carefully! ### Zoom There are two types of zoom operation: - **X only**: Zoom the X-axis only. Changes the Duration on the Timebase (Waveform Display only). - **X and Y**: Click and drag a Zoom Box to zoom both axes in Waveform or Scatterplot displays. In a Waveform Display, drawing from top-to-bottom creates an X+Y zoom box; drawing bottom-to-top creates an X-only zoom strip. In a Scatterplot Display, the zoom region is always a box regardless of direction. ### Columns Some displays (e.g., Event, Error, Summary) have resizable/hideable columns. Hover at a header edge to resize; double-click to auto-fit. Fully hiding a column by shrinking width to zero can be reversed by grabbing the split-line cursor and dragging open. ### Toolbar The display toolbar has buttons for adding all displays quickly to the current workbook page. Hover over an icon to see its name; click to add. ![Display Toolbar](assets/display-toolbar.png) ### Context Menus Right-click on most displays to access: - Copy to clipboard - Export snapshot - Display properties - Parameter properties - Reset zoom/scale - Display-specific tools ## Adding Parameters After opening a display, use the **Parameter Browser** to add telemetry channels: 1. Open Parameter Browser (`Ctrl+P` or View menu) 2. Browse or search for parameters 3. Drag parameters onto the display, or select and click "Add to Display" Many displays allow double-clicking an item (bar, axis, legend row, etc.) to open **Parameter Properties** for fast formatting. ## Live Telemetry When connected to live telemetry: - Displays update in real-time - Waveform scrolls as data arrives - Numeric/Bar displays show current values - Pause scrolling by clicking the plot area ## Workbooks Displays are saved in workbooks which remember: - Display types and positions - Parameter selections - Display properties - Layout configuration Save your workbook (`Ctrl+S`) to preserve your display setup for future sessions. ## Quick Reference | Key | Action | |-----|--------| | `Ctrl+Q, Ctrl+Q` | Quick Access Assistant | | `Ctrl+P` | Parameter Browser | | `D` | Display Properties | | `Shift+` | Switch display to Compare Set n | | `Ctrl+` | Switch page to Compare Set n | | `Ctrl+S` | Save workbook | --- For in-depth guidance on parameters, sessions, cursors, workbooks, and common workflows, see the [Working with Displays](essentials.md) guide. --- # Working with Displays This guide covers the core concepts and workflows that apply across all display types in ATLAS Viewer. Understanding these will help you work efficiently with any display. !!! tip "Quick Start" New to ATLAS displays? Start here to learn the fundamentals, then explore individual display guides for specifics. ## Display Windows Every display is a window that can be: - **Docked**: Attached to the main window in a tabbed layout - **Floating**: Separate window that can move independently (great for multi-monitor setups) - **Resized**: Drag edges or corners to resize - **Closed**: Click X button or right-click > Close Only one display is "selected" at a time (click inside it). ## Display Properties ### Opening Display Properties - Press `D` when display is focused - Right-click display > Display Properties - Toolbar button when display is selected ### Understanding Property Icons !!! info "Property Status Icons" **◻️ Square Icon**: Default value — reset with the reset button **♦️ Diamond Icon**: Changed for this display instance only — other displays unaffected **⚪ Circle Icon**: Global setting — applies to ALL displays of this type, including future ones ### Making Properties Global 1. Right-click the property name 2. Select "Make Global" — property becomes global (⚪ icon) 3. All future displays of that type will use this value To remove global status: right-click the property > "Clear Global". !!! warning Global properties affect ALL future displays of that type. Use carefully! The meaning of each property is shown at the bottom of the Display Properties window when selected. ## Working with Parameters ### Parameter Browser The Parameter Browser is your main tool for finding and adding parameters. **Opening**: - Press `Ctrl+P` - View menu > Parameter Browser - Usually docked on the left side **Features**: - :mag: **Search**: Type in search box to filter the parameter list (supports partial matches) - :star: **Favourites**: Star frequently used parameters for quick access from the Favourites tab ### Adding Parameters to Displays === "Drag and Drop" Find parameter in Parameter Browser, click and drag onto display, release to add. :mouse: **Fastest** for single parameters === "Add Button" Select parameters in Parameter Browser, click "Add to Display" button. :heavy_check_mark: **Best** for multiple parameters === "Double-Click" Double-click parameter in browser to add to active display (some displays only). :zap: **Quick** single-parameter add ### Parameter Properties Edit individual parameter settings by double-clicking the parameter in the display, or right-click > Properties. | Setting | Description | |---------|-------------| | **Colour** | Parameter trace or text colour | | **Format** | Decimal places, hex, scientific | | **Units** | Display units | | **Scale** | Y-axis range (waveforms) | | **Thresholds** | Warning and critical levels | | **Post-Processing** | Smoothing, filtering, differences | ### Removing Parameters - Select parameter in display - Press `Delete`, or right-click > Remove Parameter !!! note Some displays allow hiding parameters instead of deleting — hidden parameters can be restored. ## Display Layout and Organisation ### Pages Workbooks contain pages; pages contain displays. - **Create pages**: Right-click page tabs > New Page (or duplicate an existing page) - **Name pages**: Right-click page tab > Rename — use descriptive names like "Overview", "Sector 1", "Gearbox" - **Organise**: Group related displays on the same page; use separate pages for different analysis areas ### Docking and Floating **Docking**: - Drag display by title bar - Blue target indicators show dock locations - Drop on target to dock **Floating**: - Drag display outside dock area to create an independent window - Perfect for multi-monitor setups - Drag back to dock area to re-dock ### Resizing - Drag edges to resize - Drag corners to resize proportionally - Double-click title bar to maximise/restore - Docked displays share space with neighbours ## Session Management ### Loading Sessions **Single Session**: 1. File > Open Session 2. Browse to session file 3. Session loads into Compare Set 1 **Multiple Sessions** (Compare Sets): 1. File > Add to Compare Set 2. Choose set number (1-8) 3. Session loads into specified set 4. All displays can switch between sets ### Switching Between Sessions | Action | Shortcut | Result | |--------|----------|--------| | Switch selected display | `Shift+` | Changes active display only | | Switch entire page | `Ctrl+` | All displays on page switch together | | Click coloured tag | Mouse click on title bar | Menu shows available sets | When a whole page shows the same set, a coloured line appears across the page header. ### Compare Sets Compare Sets let you analyse multiple sessions: - Load up to 8 sessions simultaneously - Compare different laps, runs, or drivers - Each display can show a different set - Switch quickly with keyboard shortcuts ## Cursor Control ### Moving the Cursor === "Mouse" Click in display to position cursor. Drag cursor line to move. :mouse: **Quick** positioning === "Keyboard" | Keys | Action | |------|--------| | `←` / `→` | Move by one sample | | `Shift+←` / `Shift+→` | Move faster | | `Home` | Jump to session start | | `End` | Jump to session end | | `Page Up` / `Page Down` | Larger steps | :keyboard: **Precise** control === "Direct Entry" Right-click cursor area, enter specific time or distance, press `Enter` to jump. :dart: **Exact** positioning ### Cursor Synchronisation !!! info "Synchronised Cursors" The cursor is synchronised across displays: - Moving cursor in one display moves all displays - All displays show data at the same time point - **Exception**: Displays showing different sessions ## Workbooks ### What are Workbooks? Workbooks save your entire workspace: - All pages and displays - Display positions and sizes - Display properties - Parameter selections - Compare set configuration ### Saving and Opening - **Save**: `Ctrl+S` or File > Save - **Save As**: File > Save As (useful for creating templates) - **Open**: File > Open Workbook (restores exact workspace; prompts if sessions not found) ### Workbook Templates Create reusable templates: 1. Set up displays and layout 2. Configure all properties 3. Save As with a descriptive name 4. Use as starting point for similar analysis Keep templates in a dedicated folder for easy access. ## Common Workflows === "Quick Session Review" 1. Open workbook template 2. Load session 3. Cursor through interesting areas 4. Make notes in Notepad display :fast_forward: **Goal**: Rapid session inspection === "Multi-Lap Comparison" 1. Load laps into compare sets 2. Switch displays to each lap 3. Use Reference Cursor to compare points 4. Note differences in Notepad :left_right_arrow: **Goal**: Identify lap differences === "Event Investigation" 1. Open Event Display 2. Select event of interest 3. Cursor jumps to event time 4. Check parameters in other displays :mag: **Goal**: Diagnose issues === "Dashboard Monitoring" 1. Create Numeric Displays for key parameters 2. Set warning/critical thresholds 3. Position displays for visibility 4. Save as monitoring workbook template :bar_chart: **Goal**: Live monitoring station ## Performance Tips For smooth operation: - Limit parameters per display to a reasonable number - Use appropriate refresh rates - Close unused displays - Disable features you don't need (e.g., interpolation) - For large sessions, zoom to area of interest and enable sub-sampling where available - Use Summary Display instead of Waveform for statistics over large ranges ## Best Practices !!! tip "Organisation" :file_folder: **Organise with pages**: Group related displays on separate pages :floppy_disk: **Use templates**: Create workbooks for common analysis tasks :bookmark: **Name sessions clearly**: Easy to identify later !!! tip "Efficiency" :keyboard: **Learn shortcuts**: Faster than mouse for frequent actions :arrows_counterclockwise: **Save often**: Don't lose your workspace :broom: **Clean up**: Close displays you're not using !!! tip "Analysis" :memo: **Document findings**: Use Notepad display for notes :rocket: **Start simple**: Add complexity as needed :chart_with_upwards_trend: **Work methodically**: Systematic approach beats random exploration ## Troubleshooting ??? question "Display not updating?" - Check session is loaded - Verify parameter is available in session - Check refresh rate in display properties - Ensure display is focused (click it) - Check you're not paused ??? question "Can't add more parameters?" - Each display has a maximum parameter limit (see [Display Comparison](index.md#display-comparison)) - Check parameter exists in loaded session - Verify display supports that parameter type ??? question "Display shows 'No Data'?" - Check cursor is in valid data range - Verify parameters have data at cursor position - Ensure session is properly loaded - Check time/distance mode is correct ??? question "Properties won't save?" - Save workbook after changes (`Ctrl+S`) - Check workbook is not read-only - Verify write permissions to workbook location ??? question "Display performance is poor?" - Reduce number of visible parameters - Lower refresh rate - Disable expensive features (interpolation, etc.) - Close other displays temporarily - Check system resources ??? question "Display looks cluttered?" - Hide parameters temporarily (don't delete) - Use Distribute function (Waveform) - Adjust font sizes and layout - Reduce number of visible parameters ??? question "Cursor not moving?" - Click to focus the display - Check if display supports cursors - Verify session is active --- # Bar Instantaneous value of selected parameters as vertical bars with numeric readouts. Updates in real time with telemetry. Up to 12 parameters. Double‑click a bar to edit that parameter’s properties. ![Bar Display](assets/bar-display.png) ## Adding a Bar Display To add a Bar Display to a Page do one of the following: - Click the Bar Display button on the Display Toolbar. - Click `File > New > Display` and select Bar Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Bar Display. You can now use the Parameter Browser to add some Parameters to set up the display as required. Any number of bar graph displays can be opened at the same time. The display updates in real time with the reception of Telemetry data. --- # Bit Status bits of one or more parameters, with configurable On/Off colours, labels, and captions per bit. You can pick which bits are active for each parameter. ![Bit Display](assets/bit-display.png) ## Adding a Bit Display To add a Bit Display to a Page do one of the following: - Click the Bit Display button on the DisplayToolbar. - Click `File > New > Display` and select Bit Display. - Press `Ctrl + Q` twice to use the QuickAccess Assistant and select New Bit Display. To select which Bits you want to show for each Parameter: 1. Right click the column on the Display for the required Parameter. 2. Select Show Bit Options or Double Click a Parameter name on the Display headers 3. Choose from the following options for each Bit: | Option | Description | Default Value | |-------------|--------------------------------------------------------|--------------------| | **Active** | Whether the Bit is shown or not | — | | **Caption** | Text caption for that Bit | — | | **OffColour** | Colour displayed when that Bit is off | Red | | **OffLabel** | Text label displayed when that Bit is off | — | | **OnColour** | Colour displayed when that Bit is on | Green | | **OnLabel** | Text label displayed when that Bit is on | — | You can also quickly set up your Bit Display by loading a `.bcg` (Bit Configuration) file. ### Using a Bit Configuration File - The location of the `.bcg` file can be set in **Tools > Options > Plugins > Bit Display**. - Each Bit Display can be configured using the same `.bcg` file, with settings based on the parameter shown. #### File Format Each line in the `.bcg` file configures bits for a specific parameter. All bit definitions for a parameter must be on the same line. **Format:** ``` ParameterName,BitNumber,Show(1|0),Caption,OffColor(decimal),OnColor(decimal),OffLabel,OnLabel,BitNumber,Show(1|0),Caption,OffColor(decimal),OnColor(decimal),OffLabel,OnLabel,... ``` - `ParameterName`: Name of the parameter. - `BitNumber`: Zero-based index of the bit. - `Show`: `1` to show, `0` to hide. - `Caption`: Text caption for the bit. - `OffColor` / `OnColor`: Color when bit is off/on, as a decimal value of the C++ `COLORREF` format (`0x00BBGGRR` reordered to `0x00RRGGBB` and converted to decimal). - `OffLabel` / `OnLabel`: Text label when bit is off/on. **Example:** ``` Status,0,1,Power,255,65280,Off,On,1,1,Error,255,65280,No,Yes ``` For custom colors, convert the hex value to decimal after reordering. For example, `0x00877D7C` becomes `0x007C7D87` and is written as `8158599`. !!! note The Bit Display will use the `.bcg` file only after you double-click the display and check the **Use bit config file?** option. The display will then refresh with the new configuration. --- # Block Minimal single‑parameter bar—quick, graphic visualization without numeric clutter. Optional gradient colour and conditional colouring using parameter warning/critical thresholds. ![Block Bar Display](assets/block.png) ## Adding a Block Bar Display To add a Block Bar Display to a Page do one of the following: - Click the Block Bar Display button on the Display Toolbar. - Click `File > New > Display` and select Block Bar Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Block Bar Display. --- # Circuit Track map from a Circuit Definition file, with cursor‑synchronised car position indicator and optional sectors/segments; a shaded Timebase Region on the map mirrors the Waveform timebase. ![Circuit Display](assets/circuit.png) ## Adding a Circuit Display To add a Circuit Display to a Page do one of the following: - Click the Circuit Display button on the Display Toolbar. - Click `File > New > Display` and select Circuit Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Circuit Display. You can now use the Parameter Browser to add some Parameters to set up the display as required. The current cursor position is shown on the circuit if you add the sLap Parameter. ## Circuit Editor The Circuit Editor is a tool within ATLAS that allows users to create and modify Circuit Definition files. These files control how a racing track is visually represented and subdivided into Segments (Corners and Straights) and Sectors, ensuring consistency across displays such as Circuit Display and Waveform. - **Segments**: Engineering-relevant divisions of the lap, typically representing corners (green) and straights (yellow). - **Sectors**: User-defined partitions of the lap, often used for timing and analysis, with customizable colors. ![Example Circuit with Segments and Sectors](assets/circuit-editor.png) ### File Management - Circuit definitions are stored in a user-configurable folder (set in Options ▸ Plugins). - The Circuit Display’s dropdown menu is populated from files in this folder. - Supported formats: - `.cktx`: ATLAS Circuit Definition (preferred; always written on save). - `.ckt`: Atlas 9 legacy format (can be opened and converted to `.cktx`). ### Using the Editor You can open the Circuit Editor in two ways: 1. Via **Tools ▸ Circuit Editor**, then select a `.cktx` or `.ckt` file. 2. By right-clicking an existing Circuit Display and choosing **Edit**. - **Save**: Writes the current definition in `.cktx` format. Legacy `.ckt` files are converted. - **Save As**: Allows duplication under a new name, useful for creating event-specific variants. The editor is organized into three tabs, each with a live preview: 1. **General Tab** - Set the circuit’s name, add documentation notes, and specify total length. - Preview shows the circuit as it will appear in displays. 2. **Segments Tab** - Define the sequence of corners and straights. - Add, rename, or delete segments; adjust boundaries by dragging or entering values. - Segments are color-coded and labeled for clarity. 3. **Sectors Tab** - Partition the lap for timing/analysis. - Add, rename, recolor, or delete sectors; adjust boundaries interactively. - Sectors are independent of segments. ## Workflows - **Create Segments**: Open a circuit, switch to Segments, insert boundaries, set types and names, adjust distances, and save. - **Define Sectors**: Switch to Sectors, insert splits, name and color sectors, adjust boundaries, and save. - **Correct Mistakes**: Use Add/Delete without moving the cursor to revert changes. - **Document**: Update the circuit’s name and notes for clarity and team standards. !!! tip "Tips & Best Practices" - Keep segment/sector boundaries monotonic and within adjacent limits. - The Start/Finish line is fixed; plan boundaries accordingly. - Use **Save As** for alternate layouts to avoid overwriting base definitions. - Undo is limited; use Add→Delete cycles for safe reversion. ## Integration Checklist - Ensure Circuit Display shows the correct circuit, segments, sectors, and orientation. - Waveform pages should use the intended circuit definition and enable Circuit Stripe as needed. - Confirm the Options ▸ Plugins folder points to the correct repository for shared access. --- # Countdown A configurable countdown clock (or Time Offset from now) with custom text pre‑ and post‑expiry. ![Countdown Timer Display](assets/countdown-timer.png) | Option | Description | |----------------|---------------------------------------------------------------------------------------------| | **Timer Mode** | Choose between Countdown Timer (counts down to set time) or Time Offset (shows offset time).| | **ExpiryHour** | Hour of expiry for the countdown timer. | | **ExpiryMinute** | Minute of expiry for the countdown timer. | | **OffsetHours** | Hours offset from current time for Time Offset mode. | | **OffsetMinutes** | Minutes offset from current time for Time Offset mode. | ## Adding a Countdown Timer Display To add a Countdown Timer Display to a Page do one of the following: - Click the Countdown Timer Display button on the Display Toolbar. - Click `File > New > Display` and select Countdown Timer Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Countdown Timer Display. --- # Error ECU errors during live connection. Columns include Status (Forthcoming/Current/Logged, with colours), Name, Description, Timestamp. Double‑clicking an error jumps the timeline cursor to first occurrence. Logged errors persist until ECU reset based on your choice (per Lap or Session). ![Error Display](assets/error-display.png) ## Adding an Error Display You can add an Error Display to a Page using any of the following methods: - **Display Toolbar:** Click the Error Display button. - **Menu:** Go to `File > New > Display` and select Error Display. - **Quick Access Assistant:** Press `Ctrl + Q` twice, then select New Error Display. ## Error Row Details Each error is displayed as a row with the following columns: - **Status Indicator:** A coloured symbol representing the error status: - *Forthcoming*: Error expected to occur later in the session. - *Current*: Error is actively occurring (displayed in red). - *Logged*: Error occurred previously but is no longer active (displayed in white). *Note:* The error text colour matches the status indicator. - **Name:** Displays the error code and name. - **Description:** Provides a brief description of the error. - **Status:** Indicates the error's relation to the session timeline: - *Forthcoming*: Will occur later in the session. - *Current*: Actively occurring error. - *Logged*: Previously occurred, now inactive. - **Timestamp:** Shows the time when the error was first detected. ## Error Status Behaviour - When an error is detected, it appears in red text with a *Current* status. - If the sensor returns to normal, the error switches to *Logged* status and white text. - Logged errors remain visible and will revert to *Current* if the same error is detected again. --- # Event ECU Events (from program version files defined in Options), with Priority (High/Medium/Low), Timestamp, Group, Id, Description, and parameter snapshots at the event time. Tabs allow quick filtering by priority or “Masked”. Clicking an event moves the timeline cursor to that time/distance. ![Event Display](assets/event.png) ## Adding an Event Display - Use the Event Display button on the DisplayToolbar. - Navigate to File > New > Display and select Event Display. - Press Ctrl + Q twice to open the QuickAccess Assistant and select New Event Display. **Event Display Overview** Each event appears as a row with the following columns: - **Priority Indicator**: Symbol showing event priority (High, Medium, Low). - **Timestamp**: Time when the event occurred. - **Group**: Parameter group name related to the event. - **Id**: Unique integer for diagnostics. - **Description**: Text describing the event. - **Parameter Name**: Value of selected parameters at event time. **Filtering and Customization** - Tabs at the top allow filtering by priority: All, High, Medium, Low, Masked. - Column widths are adjustable (see "Columns in Displays"). - Warning ranges for parameters can be set in the Appearance section of Parameter Properties. **Display Options** - By default, text uses the foreground colour. - If "Show Warning Colours" is enabled in Display Options, parameter text changes colour when in the warning range. - If "Critical" is checked: Critical colour (default Red) is used. - If "Critical" is unchecked: Non-Critical colour (default Blue) is used. --- # FFT !!! danger "Preview Functionality" The FFT Display is currently a preview feature. As such, the content in this page is subject to change based on feedback and roadmap changes. This display is available from ATLAS [v11.5.4.230](../../../../blog/posts/viewer/11.5.4.230-W37.md) onwards. # FFT Display Frequency content of a single parameter. The FFT Display takes the samples in the current session timebase and transforms them into a spectrum — magnitude against frequency — so you can see *which* frequencies are present in a signal rather than how it varies over time. ![FFT Display](assets/fft.png) ## When to Use FFT Display :vibration_mode: **Vibration analysis**: Identify the dominant frequencies in an accelerometer or strain channel. :gear: **Rotational orders**: Relate spectral peaks to shaft, wheel or engine speeds. :radio: **Noise and interference**: Spot mains hum, sensor noise or aliasing artefacts in a channel. :chart_with_upwards_trend: **Filter checking**: Confirm that a filtered signal has actually lost the frequencies you expected. ## Enabling the FFT Display !!! note The FFT Display is off by default. Enable it in **Tools > Options > General > Preview Features** by ticking **Allow FFT display**, then restart ATLAS. Until you do, the display does not appear on the toolbar, in the **File > New > Display** menu, or in the Quick Access Assistant. ## Adding an FFT Display To add an FFT Display to a page, use one of the following methods: - **Display Toolbar:** Click the **FFT Display** button ![FFT Display toolbar icon](assets/fft-icon.png). - **Menu:** Go to **File > New > Display** and select **FFT Display**. - **Quick Access Assistant:** Press `Ctrl + Q` twice, then select **New FFT Display**. ## Adding a Parameter The FFT Display shows **one parameter at a time**. - Press `P` to open the Parameter Browser and select a parameter, or - Drag a parameter onto the display from the Parameter Browser or another display. Dropping a parameter onto a display that already has one **replaces** it — you do not need to remove the old parameter first. The spectrum trace, the fill and the peak markers all take the parameter's colour, and the parameter name is shown at the top of the plot. ## Display Anatomy - **Header readouts**: `Fs` — the sample rate used for the transform, and `df` — the frequency resolution (the width of one bin). Both appear at the top left once a spectrum has been computed. - **X-Axis**: Frequency in Hz, from 0 up to the Nyquist frequency (half the sample rate). - **Y-Axis**: Magnitude, either linear or in dB. - **Spectrum trace**: The parameter's spectrum, optionally filled below the curve. - **Peak markers**: Circular markers on the most prominent peaks. - **Status message**: Replaces the plot when there is not enough data to transform. ## Display Properties Press `D` or right-click > **Display Properties** to open the property grid. | Category | Property | Default | Description | |---|---|---|---| | General | **Background Colour** | Black | The background colour of the display. | | FFT | **Logarithmic Scale (dB)** | Off | Switches the magnitude axis between linear and dB. | | FFT | **Window Function** | Hann | The window applied before the transform: None, Hann, Hamming or Blackman. | | FFT | **FFT Size** | 1024 | The maximum number of samples used for the transform. | | FFT | **Show Fill** | On | Shows or hides the shaded area under the spectrum curve. | | FFT | **Show Peaks** | On | Shows or hides the peak markers. | ### FFT Size FFT Size sets the *maximum* number of samples fed into the transform. It is limited to the range 64 to 16384 and is always snapped to a power of two: entering a larger value rounds up to the next power of two, entering a smaller one rounds down to the previous one. So typing `1000` gives 1024, and typing `900` when the size was 1024 gives 512. The size you choose trades frequency resolution against the amount of data summarised: | FFT Size | Frequency resolution | Use when | |---|---|---| | Smaller (64–512) | Coarser | You only need the broad shape of the spectrum. | | Default (1024) | Balanced | General-purpose analysis. | | Larger (2048–16384) | Finer | You need to separate peaks that are close together in frequency. | If the parameter has fewer samples than the FFT Size, the transform still runs at the next power of two above the sample count, and the remainder is zero-padded. ### Window Function A window function tapers the ends of the sample block before the transform, which reduces the spectral leakage that would otherwise smear a peak across neighbouring bins. | Window | Character | |---|---| | **None** | No tapering. Sharpest possible peak, most leakage. | | **Hann** | Good general-purpose compromise between peak width and leakage. Default. | | **Hamming** | Slightly narrower main lobe than Hann, with a higher first side lobe. | | **Blackman** | Widest main lobe, lowest leakage — best for spotting a small peak next to a large one. | ### Logarithmic Scale With **Logarithmic Scale (dB)** off, the Y-axis shows linear magnitude and is titled *Magnitude*. With it on, magnitudes are converted to decibels (20 × log₁₀ of the linear magnitude) and the axis is titled *Magnitude (dB)*. The dB view compresses the range, which makes small peaks visible alongside a dominant one. !!! note Switching the scale clears and recomputes the spectrum, so the plot blanks briefly. ## Reading the Spectrum Hover anywhere on the trace to get a tooltip giving the frequency and the magnitude of the nearest point — the frequency in Hz to two decimal places, and the magnitude to four (in dB when the logarithmic scale is on). The two header readouts tell you what the spectrum is capable of showing: - **Fs** is estimated from the timestamps of the samples returned for the session, so it reflects the actual logging rate of the parameter. The highest frequency the plot can show is half of it. - **df** is the spacing between adjacent points on the frequency axis — the sample rate divided by the transform length. Two components closer together than `df` cannot be resolved into separate peaks. In the screenshot above, a parameter logged at 5000 Hz and the default 1024-point transform give a resolution of 4.88 Hz per bin, over a frequency axis reaching 2500 Hz. ### Peak Markers With **Show Peaks** on, the display marks the peaks in the spectrum — up to ten of them, chosen by magnitude — with a circular marker. Hover over a marker for its exact frequency and magnitude. Turn **Show Peaks** off for a cleaner trace when you are only interested in the overall shape. ## Zoom and Navigation | Action | How | |---|---| | **Zoom to a region** | Hold the right mouse button and drag a box over the area of interest. | | **Zoom with the wheel** | Scroll up to zoom in, down to zoom out. The zoom centres on the pointer. | | **Left-drag zoom** | Right-click > **Zoom Box**, then drag once with the left button. | | **Undo Zoom** | Right-click > **Undo Zoom**. Steps back one zoom level. | | **Undo All Zooms** | Right-click > **Undo All Zooms**. Returns to the full spectrum. | Both axes auto-range until you zoom. Undoing past the first zoom level restores auto-ranging, so the display resumes fitting the whole spectrum as new data arrives. ## Refreshing The display recomputes when the data it depends on changes — a new session, a new parameter, or a change to any of the FFT properties. Requests are throttled to five a second, and the spectrum is computed over the whole of the current session timebase rather than a cursor window. Changing or removing the session clears the display. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | The FFT Display is not on the toolbar or in the display menu | The preview feature is not enabled, or ATLAS has not been restarted since enabling it | Tick **Allow FFT display** in **Tools > Options > General > Preview Features** and restart ATLAS. | | "Insufficient data, please select a wider timeline" | The parameter returned too few samples over the session timebase to compute a transform | Widen the timeline, choose a parameter logged at a higher rate, or open a session with a longer timebase. | | The plot is empty and there are no readouts | No parameter has been added, or the session has no data for it | Press `P` and select a parameter, or drag one onto the display. | | The peaks you expect are missing or merged | The frequency resolution is too coarse to separate them | Increase **FFT Size**, or switch **Window Function** to Blackman if a large peak is masking a small one. | | A small peak is invisible next to a large one | Linear magnitude scaling | Turn on **Logarithmic Scale (dB)**. | ## Quick Reference | Key | Action | |---|---| | `D` | Display Properties | | `P` | Parameter Browser | | `Ctrl + Q, Ctrl + Q` | Quick Access Assistant | --- # Histogram Distribution of a single parameter against time or distance, either Spectral (time/distance in each bin) or Cumulative (≤ each bin). X‑axis range comes from the parameter’s Display Min/Max. Optional % labels on bars and Y‑axis as % or absolute. Refresh per lap in Live Mode. ![Histogram Display](assets/histogram.png) ## Adding a Histogram Display To add a Histogram Display to a Page do one of the following: - Click the Histogram Display button on the Display Toolbar. - Click `File > New > Display` and select Histogram Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Histogram Display. --- # Loadmap Pseudo‑3D heatmap of time spent in ranges of one parameter vs. another (e.g., gear vs. speed). Grid squares show in‑range value to nearest 1%; bucket counts control resolution. Refresh per lap in Live Mode. ![Loadmap Display](assets/loadmap.png) ## Adding a Loadmap Display To add a Loadmap Display to a Page do one of the following: - Click the Loadmap Display button on the DisplayToolbar. - Click `File > New > Display` and select Loadmap Display. - Press `Ctrl + Q` twice to use the QuickAccess Assistant and select NewLoadmap Display. --- # Map Satellite map centered on the current location (with zoom control and scale). ![Bing Map Display](assets/map.png) ## Adding a Bing Map Display To add a Bing Map Display to a Page do one of the following: - Click the Bing Map Display button on the Display Toolbar. - Click `File > New > Display` and select Bing Map Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Bing Map Display. In the bottom right hand corner of the display is the scale. This changes when using zoom in and zoom out. To zoom in and out of the display use the mouse wheel or press Ctrl + or Ctrl -. ## Parameter Selection The Bing Map requires **Latitude** and **Longitude** parameters to be added to the display **respectively**. To show the parameter selection dialog do one of the following: - Press **P** in the Bing Map display. - Right click the display and select **Parameter Selection**. ## Display Properties To show the display properties do one of the following: - Press **D** in the Bing Map display. - Right click the display and select **Display Properties**. **Note:** Clicking on a property provides a description of the property at the bottom of the tool window. ![Bing Map Display Properties](assets/map-properties.png) The options for the Bing Map Display are as follows: | Group | Name | Description | |-------------|-----------|-----------| | General | Show Title Bar | Shows or hides the Display's title bar | | General | Title | Title of the Display | | Data | Auto Refresh Rate (Hz) | Sets the display updates rate (Hz) when automatically panning data. | | Data | GPS Data Format | Format of GPS data in the session. It can be Decimal Degrees, F1MS, or Nascar. | | Map | Lock Center to Data | Lock the center of the display to the current data location. | --- # Notepad Free text notes for checklists and session details; useful to park instructions alongside data. ![Notepad Display](assets/notepad.png) ## Adding a Notepad Display To add a Notepad Display to a Page do one of the following: - Click the Notepad Display button on the Display Toolbar. - Click `File > New > Display` and select Notepad Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Notepad Display. ## Using the Notepad Display The notepad display can be placed inside of another display's frame, in the legend area, or as a standalone display. A Snapshot of the Notepad can be saved to a file. This can be saved as text or as an image. 1. Right click in the display. 2. Select either Snapshot to Data File or Snapshot to Image File. A copy of the Notepad text can be saved to the Clipboard. 1. Right click in the display. 2. Select Copy to Clipboard. Alternatively, press Ctrl + C to copy the Notepad text to the Clipboard. --- # Numeric !!! abstract "At a Glance" **Parameters**: Up to 150 **Layout**: Auto-sizing grid **Key Features**: Colour thresholds, hex formatting, status indicators **Best For**: Dashboards, live monitoring, quick inspection The Numeric Display shows instant numeric readouts for up to 150 parameters at the cursor position. Text auto-sizes to fit the display window, and colours can reflect warning/critical thresholds. Double-click a value to edit that parameter's properties. ![Numeric Display](assets/numeric.png) ## When to Use Numeric Display :satellite: **Live monitoring**: Watch current values during recording :bar_chart: **Dashboard creation**: Build custom parameter dashboards :mag: **Quick inspection**: Check exact values at any cursor position :traffic_light: **Status monitoring**: Use colour thresholds to highlight issues :left_right_arrow: **Parameter comparison**: View multiple related parameters together ## Adding a Numeric Display To add a Numeric Display to a page, choose one of the following methods: - **Toolbar:** Click the **Numeric Display** button on the Display Toolbar. - **Menu:** Go to **File > New > Display** and select **Numeric Display**. - **Shortcut:** Press `Ctrl+Q` twice to open the Quick Access Assistant, type "Numeric", and select **New Numeric Display**. Once added, use the **Parameter Browser** to add parameters and configure the display as needed. ## Display Anatomy The Numeric Display shows a grid of parameters with: - **Parameter name** (optional) - **Current value** (at cursor position) - **Units** (optional) - **Colour highlighting** based on thresholds Text size automatically adjusts to fit the available space. ## Key Features ### Auto-Sizing Layout The display automatically: - Calculates optimal text size based on window size - Arranges parameters in a grid - Adjusts when you resize the window - Fills available space efficiently You can override auto-sizing with fixed font size if preferred. ### Colour Thresholds Parameters can show different colours based on their values: !!! success "Normal" Default text colour (typically white) !!! warning "Warning" Text colour changes when a value enters its warning range !!! danger "Critical" Uses critical colour (default red) for critical values !!! info "Non-Critical" If critical is disabled, non-critical values use the selected colour (default blue) !!! note "Configuration" Set warning and critical ranges for each parameter in the **Appearance** section of the Parameter Properties panel. ### Flexible Formatting Control what's displayed for each parameter: - Show/hide parameter name - Show/hide description - Show/hide numeric value - Show/hide units - Show/hide sign (+ or -) - Show/hide status indicator ### Status Indicators Parameters can display status symbols: - **Up arrow**: Value increasing - **Down arrow**: Value decreasing - **Warning symbol**: In warning range - **Critical symbol**: In critical range ### Hex Formatting For parameters displayed in hexadecimal: - Option to show "0x" prefix - Configurable uppercase/lowercase - Full hex formatting control ### Grid Overlay Optional grid lines between parameters: - Show/hide grid - Custom grid colour - Adjustable padding around items - Column spacing control ## Display Properties ### Text Formatting **Font Family**: Choose from available system fonts **Font Size**: - **Auto**: Automatically scales to fit window (recommended) - **Fixed**: Specify exact point size **Alignment**: - Name alignment (Left, Center, Right) - Value alignment (Left, Center, Right) **Visibility**: - Show parameter name - Show parameter description - Show value - Show units - Show sign (+ or -) - Show status indicator ### Colour Options **Text Colouring**: - **Single colour for all**: All parameters use same text colour - **Individual colours**: Each parameter uses its own colour - **Apply critical/warning to text**: Threshold colours affect text colour **Background Colouring**: - **Display background**: Overall display background colour - **Apply critical/warning to background**: Threshold colours affect background - **Selected item colours**: Foreground and background for selected parameters **Note**: Critical/warning colours can apply to text OR background, not both simultaneously. ### Grid and Layout - **Show grid**: Enable/disable grid lines between parameters - **Grid colour**: Colour of grid lines - **Column padding**: Space between columns - **Item padding**: Space around each parameter ### Data Update - **Refresh rate**: Update frequency in Hz (1-1000 Hz) - **Auto-refresh**: Enable continuous updates ## Adding Parameters After creating a Numeric Display: 1. Open Parameter Browser (`Ctrl+P`) 2. Find the parameters you want to monitor 3. Drag parameters onto the display, or 4. Select parameters and Add to Display You can add up to 150 parameters. ## Editing Parameter Properties Double-click any parameter value to open its Parameter Properties panel where you can: - Change parameter colour - Set warning/critical thresholds - Adjust decimal places - Change display format (decimal, hex, scientific) - Set custom units - Enable/disable features per parameter ## Working with Parameters ### Selecting Parameters - Click a parameter to select it - Selected parameters show highlighted background - `Ctrl+Click`: Add to selection - `Shift+Click`: Select range ### Reordering Parameters Drag parameters to reorder them within the grid. ### Removing Parameters - Select parameter(s) - Press `Delete` key, or - Right-click > Remove Parameter ### Copying Values **Copy to Clipboard**: - Right-click > Copy (for selected parameters) - Right-click > Copy All Values copied as tab-delimited text, suitable for pasting into Excel or other tools. ### Snapshot Export Export current display values to file: 1. Right-click display 2. Select "Snapshot to File" 3. Choose format and location ## Keyboard Shortcuts | Key | Action | |-----|--------| | `D` | Display Properties | | `Delete` | Remove selected parameters | | `Ctrl+C` | Copy selected values | | `Ctrl+A` | Select all parameters | | `↑/↓/←/→` | Navigate selection | ## Example Configurations === "Dashboard Layout" Create a clean monitoring dashboard: - [x] Set **Show parameter name**: ON - [x] Set **Show value**: ON - [x] Set **Show units**: ON - [ ] Set **Show description**: OFF - [x] Enable **Apply critical/warning to text** - [x] Set appropriate thresholds :bar_chart: **Perfect for**: Live monitoring stations === "Minimal Value Display" Show just the numbers: - [ ] Set **Show parameter name**: OFF - [x] Set **Show value**: ON - [ ] Set **Show units**: OFF - [x] Use **Individual colours** for differentiation - [ ] Disable **Show status indicator** - [ ] Hide grid for cleaner look :1234: **Perfect for**: Dense parameter grids === "Status Monitor" Highlight issues immediately: - [x] Enable **Apply critical/warning to background** - [x] Set **Display background**: Dark colour - [x] Configure warning/critical thresholds - [x] Enable **Show status indicator** - [x] Use **Auto font size** for visibility :traffic_light: **Perfect for**: Alert dashboards ## Tips & Tricks !!! tip "Layout Optimization" :white_check_mark: **Dashboard creation**: Use Numeric Display as main dashboard component :art: **Colour coding**: Set up threshold colours to instantly spot issues :arrows_counterclockwise: **Live monitoring**: Keep refresh rate reasonable (20-60 Hz) to avoid flicker !!! tip "Data Display" :1234: **Hex values**: Great for displaying status words and bit fields :mag: **Quick check**: Add Numeric Display temporarily to inspect specific values :desktop_computer: **Multi-monitor**: Float Numeric Display on second monitor for live monitoring :bookmark_tabs: **Parameter grouping**: Group related parameters for easier monitoring ## Troubleshooting ??? question "Text too small to read?" - Reduce number of parameters - Increase window size - Use fixed font size instead of auto ??? question "Text too large?" - Add more parameters to fill space - Use fixed font size and reduce point size ??? question "Colours not showing?" - Check warning/critical thresholds are configured in Parameter Properties - Verify "Apply critical/warning" option is enabled - Check text vs background colour application setting ??? question "Values not updating?" - Check refresh rate is appropriate - Verify session is loaded and active - Check cursor is moving (if not live) ??? question "Parameter disappeared?" May have been deleted instead of hidden. Re-add from Parameter Browser. --- # Dash A simulator of the Motion Applied PCU Dash (as fitted to steering wheels). Supports multiple dash product types; auto‑loads a raw file if present (per Options). Positioning/alignment controls help fit the simulator to the display window. ![PCU Dash Display](assets/pcu-dash.png) ## Adding a PCU Dash Display To add a PCU Dash Display to a Page do one of the following: - Click the PCU Dash Display button on the Display Toolbar. - Click `File > New > Display` and select PCU Dash Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New PCU Dash Display. --- # Scatterplot !!! abstract "At a Glance" **Parameters**: Up to 5 XYZ sets **Modes**: 2D (XY) or 3D (XYZ with colour mapping) **Key Features**: Best-fit lines, zoom box, colour mapping **Best For**: Correlation analysis, efficiency maps, calibration maps The Scatterplot is a two-dimensional (X-Y) plot of one parameter against another, optionally colour-mapped by a third Z parameter. It's designed to reveal relationships and trends across sessions, with controls for sampling density, draw style, best-fit curves, and engineering reference lines. Up to five parameter sets can be shown in one display. ![Scatterplot Display](assets/scatterplot.png) ## When to Use Scatterplot Display :chart_with_upwards_trend: **Correlation analysis**: See how two parameters relate to each other :world_map: **Map creation**: Create torque maps, efficiency maps, calibration maps :mag: **Pattern identification**: Spot trends and clusters in data :art: **Multi-parameter relationships**: Visualise three parameters simultaneously with colour mapping :left_right_arrow: **Comparison**: Plot multiple parameter sets to compare behaviour ## Adding a Scatterplot Display To add a Scatterplot Display to a page, use one of the following methods: - **Toolbar:** Click the **Scatterplot Display** button on the Display Toolbar. - **Menu:** Go to **File > New > Display** and select **Scatterplot Display**. - **Shortcut:** Press `Ctrl+Q` twice to open the QuickAccess Assistant, type "Scatterplot", and select **New Scatterplot Display**. - **From Waveform:** Select two parameters in a Waveform legend and use **View Scatterplot of Selected Parameters** (`Ctrl+T`) to create a Scatterplot pre-wired to those signals. A third "Z" parameter can be selected to create a three-dimensional scatter plot. ### Adding the Z-axis 1. Press `P` on the display to open parameter configuration. 2. Click the **Use Z Axis** box. 3. Select the parameter from the list. 4. The Z-axis is scaled using a colour bar on the Scatterplot display. ## Display Anatomy - **Plot Area**: Shows data points - **X-Axis**: Horizontal axis (first selected parameter) - **Y-Axis**: Vertical axis (second selected parameter) - **Colour Bar**: Shows Z-axis mapping (if 3D mode) - **Grid**: Optional grid overlay - **Best-Fit Lines**: Optional trend lines - **Zero Axes**: Optional reference lines at X=0 and Y=0 ## Key Features ### 2D and 3D Plotting **2D Mode (X vs Y)**: - Plot one parameter against another - Points coloured by parameter or fixed colour - Simple correlation analysis **3D Mode (X vs Y vs Z)**: - Third parameter mapped to point colours - Colour bar shows Z-value scale - Visualise three dimensions simultaneously ### Multiple Parameter Sets Add up to 5 different XY or XYZ parameter sets: - Each set plots as separate series - Different colours per set - Compare different parameter relationships - Independent best-fit lines per set ### Colour Mapping When using Z-axis (3D mode): - Points coloured by Z-value - Colour bar shows value-to-colour mapping - Customise colour scale - Adjust colour range ### Draw Styles Choose how data points appear: - Points (dots) - Lines connecting points - Points with connecting lines - Filled shapes ### Grid Configuration **Grid Size Options**: - Small: Fine grid for detailed work - Medium: Balanced grid spacing - Large: Coarse grid for overview - Custom: Specify exact spacing **Grid Display**: - Show/hide major gridlines - Show/hide minor gridlines - Custom grid colours - Adjust grid opacity ### Zoom and Navigation **Zoom Box**: - Click and drag to create zoom region - Automatically zooms to selected area - Undo/redo zoom with toolbar buttons **Zoom History**: - Multiple zoom levels remembered - Navigate back/forward through zoom history - Reset to full view **Manual Scaling**: - Set exact X and Y axis ranges - Auto-scale to data - Lock aspect ratio (optional) ### Cursor Modes **Value Mode**: Standard cursor **Crosshair Mode**: Adds horizontal and vertical reference lines **In-Place Reference Line**: Fixed reference marker ## Best Fit & Reference Lines ### Opening the Editor - Right-click the Scatterplot and choose **Edit Best Fit and Reference Lines...**. - The editor lists all current lines with X/Y parameter binding, type (Best Fit or Reference), formula (for Reference Lines), and specified order (for Best Fit). ### Best Fit Lines - Polynomial fit up to 5th order. - **Order:** - *Unspecified*: ATLAS automatically selects the most accurate polynomial order (up to 5). - *Specified (1-5)*: Forces that exact order (e.g., 1 = linear). - **Limits:** Define which samples are used for the fit (do not clip the drawn line). The calculated line spans the entire plot area. - **Colour & Sessions:** Line colour defaults to the set's colour. In compare mode, each Best Fit line is recalculated per session and uses the session colour. Custom colours apply only in single-session mode. **Fit Types Available:** | Type | Use Case | Formula | |------|----------|---------| | Linear | Straight relationships | y = mx + b | | Polynomial | Curved relationships | y = aₓxⁿ + ... + a₁x + a₀ | | Power | Power law | y = axᵇ | | Exponential | Growth/decay | y = aeᵇˣ | | Logarithmic | Log relationships | y = a ln(x) + b | !!! tip "Formula Display" Enable **Show Formula** to display the equation with coefficients near the line. Updates when data changes! ### Reference Lines - Explicit formula: Supply coefficients for `Y = f(X)`. Use **Auto Calculate** to fill coefficients from the current sample set. - **Limits:** Define where the line is drawn (limits do clip the rendered extent). - **Colour:** Reference line colour is retained across sessions — one line per scatterplot. ### Placing a Linear Reference Line (Interactive) - Choose **Place Linear Reference Line** from the right-click menu. - A temporary white line with drag handles appears: - Drag centre to move; drag ends to pivot and stretch. - Confirm to persist the line, then select which scatterplot set to apply it to. - X/Y axis limits are inferred from handle positions. Changing the selected set updates coefficients and limits; manual tweaks are lost if you switch sets again. ### Line Lifecycle & Visibility - **Automatic deletion:** Lines are tied to specific X/Y parameter pairs. Deleting X or Y, swapping axes, or replacing Z with X or Y deletes associated lines without warning. Undo does not restore them. - **Global toggles:** - *Show Best Fit and Reference Lines* (on/off for all lines). - *Show Best Fit Line Formulas* (shows formulas under the X-axis). - Per-line or per-set visibility toggles are not available. !!! warning "Lines and Parameter Changes" If lines disappear after changing parameters, remember that changing X/Y (or swapping with Z) auto-deletes associated lines. Recreate them after the change. ## Scatterplot Sampling Modes In the ATLAS Scatterplot, there is a selection of different _Data Point Modes_. The different _Data Point Mode_ selection alters the way in which the X, Y (and Z) samples are calculated for plotting co-ordinates on the scatterplot. These can be accessed via the _Display Properties_ Window for the Scatterplot. ![Scatterplot Display Properties — modes](assets/display-properties.png) There are 6 different modes, each illustrated below with a worked example: * All Samples * Fixed Frequency * Fixed Sample Count * X Parameter * Y Parameter * Z Parameter This mode setting applies at a _display_ level, so any plots within that display inherit these properties. ### All Samples Mode _ATLAS 10.4.1 and later_ ![Selecting All Samples mode](assets/all-samples-mode-select.png) !!! note In this mode, the _Frequency (Hz)_ and _Sample Count_ fields are ignored and read only. _All Samples_ mode operates by taking each sample of the highest rate parameter, and referencing the sample and hold values of the other parameter(s). There is no averaging of samples, and only real values will be plotted. If the X Parameter is logged at 100Hz, and the Y at 10Hz, the Y parameter will effectively be super-sampled at 100Hz in order to plot a point for each X parameter sample. If super-sampling is not desired, we recommend creating a function(s) which down-samples parameters where required, such that X, Y (and Z) are all at the same rate. **Example** ![All Samples mode example](assets/all-samples-mode-example.png) | Time | X Parameter Value
(Blue) 2Hz | Y Parameter Value
(Red) 1Hz | X Co-ordinate
plotted | Y Co-ordinate
plotted | | ----- | --------------------------------- | --------------------------------- | ------------------------- | ------------------------- | | 0.0 | 0 | 4 | 0 | 4 | | 0.5 | 1 | | 1 | 4 | | 1.0 | 2 | 2 | 2 | 2 | | 1.5 | 3 | | 3 | 2 | | 2.0 | 4 | 0 | 4 | 0 | | 2.5 | 5 | | 5 | 0 | ### X, Y, Z Parameter _ATLAS 10.4.3 and later_ ATLAS 10.4.3 introduced 3 new modes in the scatterplot: X, Y and Z Parameter. These work in a similar manner to _All Samples_ mode — however instead of inferring the highest rate parameter and using that to look up corresponding values on the other axes, the user can specify explicitly whether the X, Y or Z parameter is used as the "master". **Example: X Parameter** For every sample of the X Parameter, a corresponding value of Y (or Z) is referenced regardless of the rate of these parameters: ![X-Parameter example](assets/x-parameter-example.png) | Time | X Parameter Value
(Blue) 2Hz | Y Parameter Value
(Red) 1Hz | X Co-ordinate
plotted | Y Co-ordinate
plotted | | ----- | --------------------------------- | --------------------------------- | ------------------------- | ------------------------- | | 0.0 | 0 | 4 | 0 | 4 | | 0.5 | 1 | | 1 | 4 | | 1.0 | 2 | 2 | 2 | 2 | | 1.5 | 3 | | 3 | 2 | | 2.0 | 4 | 0 | 4 | 0 | | 2.5 | 5 | | 5 | 0 | **Example: Y Parameter** For every sample of the Y Parameter, corresponding samples of X (or Z) are referenced: ![Y-Parameter example](assets/y-parameter-example.png) | Time | X Parameter Value
(Blue) 2Hz | Y Parameter Value
(Red) 1Hz | X Co-ordinate
plotted | Y Co-ordinate
plotted | | ----- | --------------------------------- | --------------------------------- | ------------------------- | ------------------------- | | 0.0 | 0 | 4 | 0 | 4 | | 0.5 | 1 | | | | | 1.0 | 2 | 2 | 2 | 2 | | 1.5 | 3 | | | | | 2.0 | 4 | 0 | 4 | 0 | | 2.5 | 5 | | | | ### Fixed Frequency and Fixed Sample Count Modes ![Selecting Fixed Frequency mode](assets/fixed-freq-select.png) _Fixed Frequency_ mode reads the _Frequency (Hz)_ field, but ignores _Sample Count_. ![Selecting Fixed Sample Count mode](assets/fixed-sample-count-select.png) _Fixed Sample Count_ mode reads the _Sample Count_ field, but ignores _Frequency (Hz)_. _Fixed Frequency_ & _Fixed Sample Count_ modes calculate samples to plot in a very similar way. Both of these modes calculate a Time Period and then plot **averages** of X, Y or Z parameters over that Period. * If a Period only spans a single sample, then the real sample value will be used; * If the Period spans multiple samples, then those samples will be averaged. The only difference between the modes is how the Time Period is calculated. **Fixed Frequency Period Calculation** Fixed Frequency Sample Time Period is set by a frequency. _Time Period = 1/Frequency (Hz)_ If the Frequency is set to _0.25Hz_, then the Time period will be _4 seconds_ _(1/0.25)_. **Fixed Sample Count Period Calculation** Fixed Sample Count Time Period is set by splitting up the Display Time range, into a number of samples. _Time Period = Display Time Range / Number of Samples_ If the Display Time range is displaying _20 seconds_ of data (can be changed by Zoom Level), and the Number of Samples is set to _5_, then the Time Period will be _4 seconds_ _(20/5)_. **Example: Fixed Frequency & Fixed Sample Counts** ![Fixed Frequency & Fixed Sample Count mode example](assets/fixed-example.png)
Time X Parameter
Value (Blue) 2Hz		 Y Parameter
Value (Red) 1Hz		 X Co-ordinate
plotted		 Y Co-ordinate
plotted
Sample Period 1
0-1.5 Seconds		 0 4 (0+1+2) / 2 = 1.5 (4+2) / 2 = 3
1
2 2
Sample Period 2
1.5-3.0 Seconds		 3 (3+4+5) / 2 = 6 0
4 0
5
In the example above, Sample Period of 1.5 seconds could be achieved in both _Fixed Frequency_ or _Fixed Sample Count_: _Time Period (Fixed Frequency) = 1 / **0.667Hz**_ _Time Period (Number of Samples) = 6 (seconds on display) / **2 (samples)**_ ## Display Properties ### Data Configuration - Fixed frequency (Hz) - Fixed sample count - Data point mode (see [Sampling Modes](#scatterplot-sampling-modes)) - Show Z-axis colour mapping ### Grid Settings - Grid size (Small/Medium/Large/Custom) - Custom grid spacing - Show major gridlines - Show minor gridlines - Grid colour ### Visual Options - Primary draw style - Show boundary lines (best-fit) - Show zero axes - Show best-fit formulas - Show colour bar (3D mode) - Background colour ### Axes - X-axis label and range - Y-axis label and range - Z-axis label and range (3D) - Auto-scale options - Lock aspect ratio ## Working with Parameters ### Adding Parameter Sets 1. Add X-axis parameter (first) 2. Add Y-axis parameter (second) 3. (Optional) Add Z-axis parameter (third) 4. Repeat for additional parameter sets (up to 5) ### Editing Parameter Properties Double-click axis label or parameter to configure: - Parameter selection - Colour assignment - Scale and range - Units display - Decimal precision ### Removing Parameters - Select parameter set - Press `Delete` or right-click > Remove ## Keyboard Shortcuts | Key | Action | |-----|--------| | `D` | Display Properties | | `P` | Parameter configuration | | `Z` | Zoom box mode | | `Ctrl+Z` | Undo zoom | | `Ctrl+Y` | Redo zoom | | `Home` | Reset zoom | | `Ctrl+T` | View Scatterplot of Selected Parameters (from Waveform) | | `Delete` | Remove selected parameter set | ## Example Use Cases === "Torque Map" Create an engine torque map: 1. Add Scatterplot Display 2. **X-axis**: Engine RPM 3. **Y-axis**: Throttle Position 4. **Z-axis**: Engine Torque (colour mapped) 5. Enable colour bar 6. Adjust grid for readability 7. Set appropriate axis ranges :gear: **Result**: Visual map of torque across RPM and throttle === "Correlation Analysis" Analyse parameter relationships: 1. Add parameters to analyse (X vs Y) 2. Add linear best-fit line 3. Enable "Show Formula" 4. Look for strong correlation (tight to line) 5. R² value indicates correlation strength :mag: **Result**: Quantify relationship between parameters === "Efficiency Mapping" Find efficient operating zones: 1. **X-axis**: Speed 2. **Y-axis**: Power 3. **Z-axis**: Fuel consumption (colour) 4. Identify efficient zones (cooler colours) 5. Spot inefficiencies (warmer colours) :zap: **Result**: Optimise performance and efficiency ## Tips & Tricks !!! tip "Getting Started" :white_check_mark: **Start simple**: Begin with 2D, add Z-axis when needed :grid: **Grid sizing**: Use Custom grid for specific map resolutions :art: **Colour scales**: Adjust Z-axis range to enhance colour contrast !!! tip "Analysis" :mag: **Zoom in**: Work at appropriate detail level for analysis :chart_with_upwards_trend: **Best-fit lines**: Use to quantify relationships numerically :left_right_arrow: **Multiple sets**: Compare different sessions or laps :floppy_disk: **Save templates**: Save scatterplot configurations for repeated use !!! tip "Performance" If the plot feels cluttered, reduce density via Fixed Sample Count or switch Draw Style to Small Point or Cross. If Z-colouring appears sparse, check the Z limits (Parameter Properties > Appearance > Limits). Points with Z outside limits are not drawn. To focus on part of the lap/run, set a Reference Cursor on a Waveform and use the Scatterplot's solid/faint point effect to isolate that window. ## Troubleshooting ??? question "No data points showing?" - Check parameters are added correctly - Verify session is loaded - Check axis ranges include data - Try auto-scale ??? question "Points too dense to see pattern?" - Use fixed frequency mode to thin data - Zoom into specific regions - Adjust point draw style - Enable best-fit line to see trend ??? question "Colour bar not showing?" - Verify Z-axis parameter is added - Check "Show colour bar" is enabled - Check Z-parameter has valid data ??? question "Best-fit line incorrect?" - Check fit type is appropriate for data - Try different polynomial degree - Verify sufficient data points - Check for outliers affecting fit ??? question "Zoom stuck?" - Press `Home` to reset zoom - Use undo zoom (`Ctrl+Z`) - Check manual axis ranges ??? question "Lines disappeared after parameter change?" - Lines are tied to specific X/Y parameter pairs - Deleting X or Y, swapping axes, or replacing Z with X or Y deletes associated lines without warning - Undo does not restore them — recreate the lines after the change --- # Summary Lap‑based statistics for selected parameters as a table or as a plot. First columns are fixed: Lap #, Lap Name, Lap Time. For each parameter, you can add stats columns such as Min, Max, Mean, Start, End, Change, Range, Absolute Mean, Geometric Mean, Std Dev, Slope, Slope Intercept, R². ![Summary Display](assets/summary.png) ## Adding a Summary Display To add a Summary Display to a page, use one of the following methods: - Click the **Summary Display** button on the Display Toolbar. - Go to **File > New > Display** and select **Summary Display**. - Press **Ctrl + Q** twice to open the Quick Access Assistant, then select **New Summary Display**. Once added, use the **Parameter Browser** to choose parameters and configure the display as needed. The first three columns—**Lap #**, **Lap Name**, and **Lap Time**—are always shown. For each parameter, you can display one or more statistics over the relevant data range. Available statistics include: - **Min**: The minimum value recorded for the parameter within the lap. - **Max**: The maximum value recorded for the parameter within the lap. - **Mean**: The arithmetic average of all parameter values within the lap. - **Start**: The parameter value at the beginning of the lap. - **End**: The parameter value at the end of the lap. - **Change**: The difference between the end and start values of the parameter for the lap. - **Range**: The difference between the maximum and minimum values for the parameter within the lap. - **Absolute Mean**: The mean of the absolute values of the parameter within the lap. - **Geometric Mean**: The geometric average of the parameter values within the lap. - **Standard Deviation**: A measure of the variability or dispersion of the parameter values within the lap. - **Slope**: The rate of change of the parameter values over the lap, calculated using linear regression. - **Slope Intercept**: The intercept value from the linear regression of the parameter values over the lap. - **Slope R Squared**: The coefficient of determination (R²) indicating how well the linear regression fits the parameter data within the lap. You can resize columns individually, or right-click and select **Auto Resize Columns** to automatically adjust all column widths to fit the data. To switch between table and plot views, right-click the display and select **Show As Plot** or **Show As DataGrid**. --- # Surface Map The Surface Map Display provides an interactive 3D visualization using a `.3d` map definition. It pairs two parameters (such as throttle and engine speed) and interpolates a third value. The display shows the mesh from the `.3d` file, data points colored by the X‑axis parameter, and highlights the current cursor point. You can rotate the map with the mouse, zoom with the mouse wheel, and double‑click to reset the view. ![Surface Map Display](assets/surface.png) ## Adding a Surface Map Display To add a Surface Map Display to a page, use one of the following methods: - Click the **Surface Map Display** button on the Display Toolbar. - Go to **File > New > Display** and select **Surface Map Display**. - Press **Ctrl + Q** twice to open the Quick Access Assistant and select **New Surface Map Display**. ## Setting Up the Display After adding the display, specify the location of the `.3d` file: 1. Press **D** to open the Display Properties tool window. 2. Enter the file location in the **Map Definition** option. Next, use the **Parameter Browser** to add the two parameters for which the map file is configured. ## Display Features - The map (grid and mesh) appears once parameters are set. - The mesh (from the `.3d` file) is shown in white by default. - Data points are colored according to the X-axis parameter (blue in the example). - The current cursor point is highlighted as an enlarged point (red in the example). - All display properties can be adjusted in the Display Properties tool window. You can interact with the map by rotating it with the mouse, zooming in or out with the mouse wheel, and resetting its position by double-clicking the display. --- # Video !!! danger "Preview Functionality" Video display is currently a preview feature. As such, the content in this page is subject to change based on feedback and roadmap changes. # Video Display Related video (e.g., in‑car camera) played alongside session data. ![Video Display](assets/video.png) ## Adding a Video Display !!! note You must enable preview features in Tools > Options > General > Preview Features to use the Video Display. To add a Video Display to a page, use one of the following methods: - **Display Toolbar:** Click the **Video Display** button. - **Menu:** Go to **File > New > Display** and select **Video Display**. - **Quick Access Assistant:** Press `Ctrl + Q` twice, then select **New Video Display**. ## Synchronising Video with a Session To synchronise a video with session data: - **Play Video:** Click **Play** on the video. The waveform cursor will automatically synchronise to the current video frame. - **Navigate by Cursor:** If the video is stopped, click on the waveform cursor. The video will display the frame corresponding to the timestamp under the cursor. --- # Waveform !!! abstract "At a Glance" **Parameters**: Unlimited **X-Axis**: Time or Distance **Key Features**: Multiple cursors, statistics, events, zoom **Best For**: Time-series analysis, multi-session comparison The Waveform Display plots multiple parameters as traces against time or distance. It's the most versatile display in ATLAS, supporting live telemetry and historical sessions, fast zooming, and rich formatting. Use it for: - Braking/throttle application vs. speed, per corner or lap - Multi-session comparisons with Time Difference (TDiff) in Distance mode - Event-driven triage (ECU events/errors) aligned to timestamps or distances ![Waveform Display](assets/waveform.png) ## When to Use Waveform Display :racing_car: **Driver analysis**: Brake/throttle application vs. speed per corner :bar_chart: **Multi-session comparison**: Compare different laps or sessions side-by-side :mag: **Event investigation**: See telemetry context around ECU events :chart_with_upwards_trend: **Trend analysis**: Identify patterns over time or distance :satellite: **Live monitoring**: Watch real-time telemetry streams ## Adding a Waveform Display === "Quick Access" Press `Ctrl+Q` twice, type "Waveform", press `Enter` :zap: **Fastest method** === "Toolbar" Click the Waveform icon on the Display Toolbar :mouse: **One click** when you know the icon === "Menu" **File** > **New** > **Display** > **Waveform Display** :books: **Browse all options** ### Parameter Selection - Use Parameter Browser (default max 100 parameters, configurable) - Double-click legend items for Parameter Properties (supports single/multi-session editing) ### Compare Sets - Load multiple sessions into a set - Switch sets: `Shift+` (selected display), `Ctrl+` (whole page), or click coloured tag on title bar ## Display Anatomy ### Plot Area - One coloured trace per parameter - Grid (vertical/horizontal/both) with adjustable spacing (shown at top of plot) - Cursor line showing current position - Cursor text at top-left (shows Time, and Distance in Distance mode) - X-axis showing time or distance - Event markers below X-axis (coloured squares) ### Legend - One row per parameter with cursor value and statistics - Colour bar matches trace colour - Click rows to select parameters - Reposition, resize, grid, show units, toggle headers - Select rows (`Ctrl`/`Shift`) to highlight traces and perform actions (hide/delete/distribute) ### Axes - **X-Axis**: Time (default) or Distance (press `K` to switch) - **Y-Axis**: Per-parameter scales (enable in Parameter Properties) ## Key Features ### Cursors **Cursor Modes** (View menu or shortcuts): | Mode | Key | Description | |------|-----|-------------| | Value | `V` | Vertical line, shows sample values (default) | | Crosshair | `C` | Adds horizontal sight line | | Gradient | `G` | Diagonal line with three handles; legend shows per-signal gradient | | Hide | `N` | Turn off cursor display | !!! tip "Reference Cursor" Press `R` to add a red reference line for comparison. **Features:** - Red reference line; grey window between reference and current cursor defines analysis range - Legend shows ΔTime/ΔDistance and computes statistics for that window - Drag either cursor to adjust window size ![Waveform Display with Reference Cursor](assets/ref-cursor.png) ### Statistics Enable statistics to see parameter behaviour over ranges: !!! note "Setup Steps" 1. Check **Show Statistics** in Display Properties 2. Choose your **Statistics Level** 3. Toggle metrics with keyboard shortcuts **Statistics Levels:** | Level | Scope | Prefix | |-------|-------|--------| | Display | Visible timebase | D | | Lap | Current lap only | L | | Session | Entire session | S | | All | All loaded sessions | A | Reference Cursor can have its own level or swap with the chosen level. **Available Metrics:** | Metric | Key | Description | |--------|-----|-------------| | Min | `M` | Minimum value | | Max | `X` | Maximum value | | Mean | `N` | Average value | | Delta | `E` | Difference between cursors | | Std Dev | `Q` | Standard deviation | !!! example "Reading Statistics" Statistics appear in legend with prefixes: **LMin** (Lap Min), **SMax** (Session Max), etc. ### Selecting, Highlighting, Hiding Parameters **Select Parameters**: - Click legend row to select one parameter - `Ctrl+Click`: Add to selection - `Shift+Click`: Select range - Selected traces flash (by default) **Hide Selected Parameters**: - Hides traces but keeps legend entries - Names shown with strikethrough - Values shown in italics - Unhide by selecting and pressing Hide again **Delete Selected Parameters**: - Removes parameters completely from display **Distribute**: - Stacks selected traces into separate bands - Useful for avoiding overlapping traces - **Reset Distribute**: Returns to normal view **Autoscale**: - Fits selected parameters to available vertical space - **Reset Auto-Scale**: Returns to original scaling ### Events and Markers **Event Indicators:** !!! info "Event Colours" Coloured squares below X-axis indicate event severity: - :red_square: **Red**: High priority events - :blue_square: **Blue**: Medium priority events - :green_square: **Green**: Low priority events **Features:** - Hover for event details - Multiple events at same position show highest priority colour - Mask/unmask events by hovering and right-clicking - View masked events via context menu **Other Markers:** | Marker Type | Description | Toggle | |-------------|-------------|--------| | Lap Markers | Vertical dashed lines at lap boundaries | Display Properties | | Date/Time Markers | Clock time or lap time on X-axis | Time mode only | | Circuit Stripe | Track sectors/segments across top of plot; toggle labels; choose global/overridden Circuit Definition | Display Properties | ### Zoom and Navigation **Zoom Box**: - Click and drag to create zoom region - Top-to-bottom: Zoom X and Y axes - Bottom-to-top: Zoom X-axis only (zoom strip) - Mouse wheel: Zoom X-axis (time/distance) - Toolbar zoom buttons **Autoscroll** (Live Sessions): - Middle mouse button (or both buttons) - Pointer position controls scroll speed - Click to exit autoscroll mode **Pause/Inspect**: - Single-click plot to pause scrolling and position cursor - Double-click to resume scrolling ### Time vs. Distance Mode Press `K` to cycle between X-axis mappings: - **Time Mode**: X-axis shows elapsed time - **Distance Mode**: X-axis shows distance travelled **Time Difference (TDiff)** in Distance Mode: - Plots time difference between sessions - Shows which session is ahead/behind at each distance - Enable **Show Time Diff** in Display Properties - Choose TDiff start point: - Start of lap - Current cursor position ### Sampling and Interpolation **Sub-Sampling** (when cursor spans multiple samples): Choose how to display data: - **First**: Show first sample value - **Min**: Show minimum in range - **Max**: Show maximum in range - **Mean**: Show average - **Min→Max**: Draw vertical line from min to max - **Max→Min**: Draw vertical line from max to min Legend shows min/max values; trace shows vertical min-max line. **Interpolation**: - Enable **Interpolate between samples** (Parameter Properties > Post Processing) to smooth traces and generate interpolated legend values - Makes stepped traces appear smoother ### Session Comparison **Show Differences**: - Displays value difference between sessions at cursor - Enable in Parameter Properties > Post Processing - Pause live scrolling for readability - Not available for: - Boolean or Text parameters - Functions - When using Filters or Trends ### Left-to-Right Mode (Strip Chart) For live telemetry analysis: 1. Enable **Draw left to right** in Display Properties 2. Traces overwrite from left as laps progress 3. Set **Gap %** to add gap ahead of cursor 4. Choose **Left to Right Overlay** lap for reference Useful for comparing current lap against previous laps in real-time. ### Live Telemetry & Historical Replay - **Live:** Waveform scrolls/updates if a live session is in the set - **Historical Replay:** Enable **Allow Historical Replay**; double-click plot to replay ## Keyboard Shortcuts ### Cursor Modes | Key | Action | |-----|--------| | `V` | Value cursor mode | | `C` | Crosshair cursor mode | | `G` | Gradient cursor mode | | `N` | Hide cursor | | `R` | Toggle reference cursor | ### Statistics | Key | Action | |-----|--------| | `M` | Toggle Min statistic | | `X` | Toggle Max statistic | | `N` | Toggle Mean statistic | | `E` | Toggle Delta statistic | | `Q` | Toggle Std Dev statistic | ### Navigation | Key | Action | |-----|--------| | `K` | Cycle X-axis mode (Time/Distance) | | `D` | Display Properties | | `←` / `→` | Move cursor | | `Shift+←` / `Shift+→` | Move cursor faster | | `Home` | Cursor to start | | `End` | Cursor to end | ### Compare Sets | Key | Action | |-----|--------| | `Shift+` | Switch set (selected display) | | `Ctrl+` | Switch set (whole page) | ## Display Properties Key properties you can configure: ### General - Background colour - Show grid (horizontal, vertical, both) - Grid spacing - Show title bar - Allow historical replay ### Legend - Position (top, bottom, left, right) - Font size - Show units - Show warning indicators - Value alignment ### Cursor & Statistics - Cursor mode (Value, Crosshair, Gradient) - Show statistics - Statistics level - Reference cursor level ### Time/Distance - X-axis mode - Show time difference - TDiff start point - Time notation format ### Events - Show event indicators - Show lap markers - Show date/time markers - Circuit stripe visibility ### Sub-Sampling - Drawing mode (First, Min, Max, Mean, etc.) - Interpolate between samples ## Example Configurations === "Driver Analysis" Analyse brake/throttle inputs: 1. Add Waveform Display 2. Add Brake, Throttle, Speed parameters 3. Use **Distribute** to stack traces 4. Switch to **Distance** mode (`K`) 5. Add **Reference Cursor** (`R`) to compare corners :racing_car: **Perfect for**: Corner-by-corner driver analysis === "Multi-Session Comparison" Compare laps side-by-side: 1. Load multiple sessions 2. Switch to **Distance** mode (`K`) 3. Enable **Show Time Diff** 4. Use **Show Differences** for value deltas :bar_chart: **Perfect for**: Lap comparison and performance progression === "Live Monitoring" Real-time telemetry view: 1. Connect live session 2. Enable **Draw left to right** (strip chart mode) 3. Set **Gap %** ahead of cursor 4. Choose overlay lap for reference :satellite: **Perfect for**: Pit wall and live engineering ## Tips & Tricks !!! tip "Parameter Management" :heavy_check_mark: **Too many parameters?** Use Distribute to stack traces vertically :mag_right: **Can't see parameter?** Use Autoscale to fit to screen !!! tip "Analysis" :left_right_arrow: **Comparing laps?** Use Reference Cursor to compare two points :triangular_flag_on_post: **Analysing events?** Event markers show when things happened !!! tip "Visualisation" :chart_with_upwards_trend: **Stepped signals?** Enable interpolation for smoother traces :mag: **Large dataset?** Zoom in to see detail, zoom out for overview :arrows_counterclockwise: **Multi-session comparison?** Use Time Diff in Distance mode :world_map: **Track context?** Enable Circuit Stripe to see where on track ## Troubleshooting ??? question "No Y-axis scales showing?" Enable per-parameter Y-Scale in Parameter Properties. ??? question "Traces overlapping and hard to read?" - Use Distribute function to stack traces into bands - Reduce number of visible parameters - Hide less important parameters temporarily - Use Autoscale to fit data to vertical space ??? question "Event markers not showing?" - Check **Show Event Indicators** is enabled in Display Properties - Verify events aren't masked in Event Filter Display - Check event priorities are visible ??? question "Differences not showing?" "Show Differences" not available with: - Filters, Trends, or Functions - Boolean/Text parameters - During live scrolling (pause to read values) ??? question "Cursor not moving?" - Click to focus the waveform display - Check keyboard focus is on the display ??? question "Stepped signals look wrong?" Consider enabling **Interpolate between samples** in Parameter Properties > Post Processing if interpolation is appropriate for the signal type. ??? question "Missing event markers?" Ensure **Show Event Indicators** is on; check Event Filters for masked priorities. --- # Web Display An embedded browser—useful for live camera feeds, weather, or other web‑based information. ![Web Browser Display](assets/web.png) ## Adding a Web Browser Display To add a Web Browser Display to a Page do one of the following: - Click the Web Browser Display button on the Display Toolbar. - Click `File > New > Display` and select Web Browser Display. - Press `Ctrl + Q` twice to use the Quick Access Assistant and select New Web Browser Display. --- # Overview Extend ATLAS capabilities by connecting with third‑party systems and custom workflows. ## [Remote Telemetry Architecture (RTA)](rta/index.md) RTA provides low‑latency access to live and historical telemetry using your existing databases. It’s designed for teams building integrated data pipelines without complex setup. ### Key Functionality - Database integration - Live & historical session access - OAuth2.0 with MFA - Microservice‑based data delivery ## [APIs](apis/index.md) ATLAS APIs provide direct programmatic access for automation, custom tooling, and deeper analysis. **Includes:** - SQL Race API — query and analyse telemetry datasets - Automation API — streamline repetitive or operational tasks - Display API — create specialised custom visual displays --- # Overview !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Introduction to RTA ATLAS _Remote Telemetry Architecture_ (RTA) defines web services to let you load telemetry directly from your existing storage: This can work with a wide range of technologies, such as: * [InfluxDB](https://www.influxdata.com/products/influxdb/), [TimescaleDB](https://www.timescale.com/) and other time-series stores * [MongoDB](https://www.mongodb.com/), [Couchbase](https://www.couchbase.com/) and other document databases * [Parquet](https://parquet.apache.org/), [HDF5](https://www.hdfgroup.org/solutions/hdf5/) and other file formats * [SQL Server](https://www.microsoft.com/en-gb/sql-server), [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/)... ## Live Monitoring The RTA specification includes [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) streaming for live telemetry monitoring. Our reference architecture recommends implementing this by taking a feed from your acquisition pipeline, and buffering it through [Redis](https://redis.io/) — which decouples the client activity and allows users to join the session at any point. This enables ATLAS to display data in real-time even if your acquisition pipeline cannot write to storage in real-time, or if the storage is not available until acquisition is complete — common with file formats. When users open live sessions, ATLAS selectively subscribes to the parameters on display using a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) connection, and the [Redis](https://redis.io/) buffer covers the gap until data is flushed to storage. Users can get a complete and real-time view of the data even when joining hours into a session. ## Toolkit Services The [RTA API Specification](../../../developer-resources/rta/api/index.md) provides all the nececessary information to create a slick integration between ATLAS and your data infrastructure, with detail including: * [OpenAPI schema](https://www.openapis.org/faq) describing the REST API * [JSON schemas](https://json-schema.org/) for the data model and query dialect * [Protobuf schemas](https://developers.google.com/protocol-buffers) for low-level data transport However, you can save a lot of development effort by deploying [services](../../../developer-resources/rta/services/index.md) from our implementation toolkit. By using these services to cover all the data management functionality, you won't need to implement any of the browsing, search or configuration management features. All you'll need to write is a simple hook into your ingest pipeline, and a [Data Adapter Service](introduction/data-services.md): These Toolkit Services provide [gRPC](https://grpc.io/) services for integration with the rest of your data infrastructure. [gRPC](https://grpc.io/) is a modern, high performance framework for making service calls. It supports all popular programming languages, and the services are fully-described with schemas — so you get a high-quality client library that will work with your existing software stack, and high-quality developer tools to guide use of the API. You can deploy these services on [Windows](../../../developer-resources/rta/downloads.md#binaries) or [Linux](../../../developer-resources/rta/downloads.md#binaries), and on-premises or in the Cloud using our [Docker images](../../../developer-resources/rta/downloads.md#docker). Our services are specifically designed to work well with [Kubernetes](https://kubernetes.io/) and [AWS](https://aws.amazon.com/). ## Standalone Options If you do not have any central telemetry database or file archive, RTA may not be the right architecture for you — but ATLAS is also very capable operating standalone: * ATLAS can browse and mount files directly. This is a great option if your use case involves a lot of travel and in-field laptops. * ATLAS can display and record live telemetry directly from equipment — e.g. CAN bus — or the local network. This option is popular in motorsport and eSports. Some plugin development will be required for both these capabilities. [Contact us](mailto:sales@motionapplied.com) for a more in-depth technical discussion. --- # Architectures !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Architecture Patterns The [RTA API Specification](../../../../developer-resources/rta/api/index.md) does not require any specific architecture or technology stack — but there are some common patterns. This page illustrates some architectures supported by [services from our implementation toolkit](../../../../developer-resources/rta/services/index.md). !!! info "Legend" ## Development ### RTA Server [RTA Server](../../../../developer-resources/rta/services/rta-server/README.md) provides a simple, single-process deployment to experiment to explore the [RTA API Specification](../../../../developer-resources/rta/api/index.md) and gain familiarity with [Sessions](sessions.md), [Configuration](configuration.md) and the data types. ### Microservices The [Toolkit Services](../../../../developer-resources/rta/services/index.md) are designed be to run as microservices within a [container orchestrator](https://docs.docker.com/get-started/orchestration/), like [Kubernetes](https://kubernetes.io/). Here is the same capability, split into individual components: All these services are horizontally-scalable, facilitating high-availability deployments. ### OAuth 2.0 Security ATLAS and the [Toolkit Services](../../../../developer-resources/rta/services/index.md) can use [OAuth 2.0](https://oauth.net/2/) with multi-factor authentication: The [Gateway Service](../../../../developer-resources/rta/services/rta-gatewaysvc/README.md) proxies the authorization bearer token through to the other microservices. ## Data Adapter Service RTA does not require data format-shifting. Instead of copying the data with a loader, you can index it in place and deploy a [Data Adapter Service](data-services.md) to read the store: === "InfluxDB" _McLaren provide an [InfluxDB Data Service](../../../../developer-resources/rta/services/rta-influxdatasvc/README.md) in the implementation toolkit:_ === "Files" _Custom file formats can be served using exactly the same pattern:_ === "Data Federation" _Hetrogeneous formats can all be handled in one deployment:_ ## Continuous Ingest and Live Streaming The diagrams above show data being indexed after it has arrived. But this can be integrated into existing ingest pipelines. Calls to the [Toolkit Services](../../../../developer-resources/rta/services/index.md) are made using [gRPC](https://grpc.io/), which is supported in a wide range of languages — so it is easy to publish [configuration](configuration.md) and [session](sessions.md) metadata from an existing framework. This is not limited to metadata. You can live-stream data to ATLAS: === "Live Streaming" _A copy of the data is streamed across to [Redis](https://redis.io/), and connected to clients via [Stream Service](../../../../developer-resources/rta/services/rta-streamsvc/README.md) web sockets:_ === "Live Streaming with Files" _This streaming strategy can provide a fast bypass when the store has high latency — for example, file storage:_ ## Remote Access The web services are specifically designed for high-latency operation across the WAN or Internet. === "Remote Access" _Connections are secured with TLS and OAuth 2.0, so users can connect to a data center or cloud deployment without a VPN:_ === "Remote Access across Company Boundaries" _Many OAuth 2.0 providers offer identity federation, so subsets of data can be shared securely across company boundaries:_ --- # Data Services !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Data Services Data Services read samples and events from a telemetry store, and serve them to ATLAS. The telemetry store could be a database, or a data lake, or an object store, or a set of files... The service itself can be a simple read-only _data adapter_. It can extend an existing application, or it could be a standalone service sharing the existing data store. For high performance, it should access the data store as directly as possible rather than proxying a web service. [Services from the toolkit](../../../../developer-resources/rta/services/index.md) can cover the other key requirements, which are: * Data management — _[Sessions](sessions.md)_ * Describing the data model — _[Configuration](configuration.md)_ _Example deployment, with supporting [Gateway Service](../../../../developer-resources/rta/services/rta-gatewaysvc/README.md) (proxy) and [Schema Mapping Service](../../../../developer-resources/rta/services/rta-schemamappingsvc/README.md):_ The [Developer Guide](../../../../developer-resources/rta/worked-guide/index.md) includes a walkthrough demonstrating the process of [writing a Data Adapter Service](../../../../developer-resources/rta/worked-guide/tutorials/data-adapter/index.md), which can be written from scratch in any framework using the [API Specification](../../../../developer-resources/r../../../developer-resources/rta/api/index.md), or built in [ASP.NET Core](https://docs.microsoft.com/en-us/aspnet/core) taking advantage of Motion Applied's [library support](../../../../developer-resources/rta/downloads.md). Development time can be as little as a few hours for a prototype, up to a few weeks for an efficient, production-grade service. ## Toolkit Services Motion Applied currently provide two data services in the [services toolkit](../../../../developer-resources/rta/services/index.md): [RTA Data Service](../../../../developer-resources/rta/services/rta-datasvc/README.md) : Read/write storage optimized for RTA, backed by either a file system, or [Amazon S3](https://aws.amazon.com/s3/) and [DynamoDB](https://aws.amazon.com/dynamodb/). Cost-efficient and very fast — but provides no data query or summarization capabilities. [Influx Data Service](../../../../developer-resources/rta/services/rta-influxdatasvc/README.md) : Data Adapter Service for [InfluxDB](https://www.influxdata.com/products/influxdb/). --- # Sessions !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Sessions _Sessions_ are a logical way to organise telemetry — like a file system. Each session is labelled with user-defined metadata, including: * unique identity — GUID, database key, or query string * human-readable name and properties * timestamp and start/end time of the data * relationships to other sessions ## Browse and Search ATLAS leverages this metadata to make sessions easy to browse, filter and sort. ATLAS Session Browser This system provides excellent scalability and can be tightly integrated with the existing data environment. ## Modelling and Simulation ATLAS provides traceability and version control features to support _Virtual Product Development_ through models and simulations. !!! example Models are used extensively in _Formula One_ development. For example, an aerodynamics model predicts the air-flow over the car as it races around a track, depending on hundreds of variables — such as the ride height, wing angle and cooling setup. On-track, air-pressure sensors and pitot tubes measure real downforce, and engineers use ATLAS to compare this telemetry with predicated parameters to find the correlation with the model. Both the telemetry and models may go through multiple versions to cleanup the data — for example, removing pressure traces from sensors that are contaminated with debris. Simulated telemetry sessions are often used to add new parameters based on a physics model, signal processing pipeline, or machine learning algorithm. RTA can describe the relationship between the parent telemetry and child models — and ATLAS will leverage this to keep the data together. ## Alternate Versions You can also offer users multiple versions of data. For example, data collected during a test session might need some trimming and cleanup. ATLAS lets you switch out the original data for the clean version — yet keep the original available for easy reference. ## Applying this concept to your environment Sessions typically represent a source of data — such as a set of equipment — and cover a bounded time period: * If you are using ATLAS to monitor hardware and embedded systems, this may naturally encompass a test session. * If you are integrating with a continuously-stored time series, you'll need to find a logical split. For example, if you were collecting vehicle telemetry, you might define a session for each vehicle and journey. You can build this catalogue in real-time, or as a batch process. Here are a few example scenarios: === "Files" _If you have an existing collection of telemetry files:_ * Map each file to a session, using the file name and metadata * If several files belong together, you can treat them as one session — or use a parent-child relationship === "Databases" _If you are using a time-series database like [InfluxDB](https://www.influxdata.com/products/influxdb/) or [TimescaleDB](https://www.timescale.com/):_ * Use the database tagging and query facilities * Split the sessions are points of interest, or by schedule === "UDP Stream" _If you are acquiring data from a real-time UDP stream:_ * You can open a session even before data starts flowing * Metadata can be updated as you go — or later, to reflect test results === "Models" _If you are processing each telemetry session through models:_ * You can associate the sessions in ATLAS so the data is loaded together * Use the _type_, _quality_, _group_ and _version_ metadata to organise the model output --- # Configuration !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Configuration ATLAS is data-driven: it uses _Configuration_ to describe the telemetry. The core data types are: * _Parameters_ — representing sampled data points, each with a time and value * _Events_ — representing a moment in time, with associated text or values Most time-series databases can do these, too — but ATLAS has some additional capabilities built from its heritage in Engineering, rather than infrastructure monitoring. ## Designed for Scale ATLAS is designed for large sets of parameters, and many samples. A typical session from a [Motion Applied TAG-320B ECU](https://motionapplied.com/products/TAG-320B) on a Formula One car contains 50,000 - 60,000 parameters. Of these, around 4,000 are sampled at up to 1 KHz, with the remainder being densely-packed snapshots of embedded memory captured every few seconds. A race-length session contains about 2,000,000,000 samples. ## Apps and Parameter Trees ATLAS is designed so that a stream of telemetry might contain data from multiple devices and models, assembled by different teams — so configuration is divided into _Apps_. Each App can have 10,000+ parameters, so they are organised into a tree, which users can browse and search: ATLAS Parameter Browser Parameters aren't just a name — they have descriptions, units, formatting and expected ranges. ## Calibration and Conversions ATLAS can apply gain and offset calculations to correct parameter data. This is useful when working with electronics and embedded systems. The list of parameters and their valid ranges will be known when software is flashed onto a unit, but it may be necessary to zero some sensor readings after startup. ## Dense Data ATLAS and RTA can handle data efficiently, by preserving the data width — from single-byte to double-precision. There is also a facility to take a buffer — representing a section of memory from an embedded controller — and pick it apart into parameters. This is called _Row Data_ in the [API Specification](../../../../developer-resources/r../../../developer-resources/rta/api/index.md). ## Text As in [Grafana](https://grafana.com/), parameters that represent enumerations can be mapped onto text. Events can also carry text — or values that can then be formatted to text. !!! note ATLAS is not a logging platform: it is designed primarily for numerical telemetry. Think [Prometheus](https://prometheus.io/), not [Fluentd](https://www.fluentd.org/). --- # Overview ATLAS APIs are designed with an API-first strategy, enabling seamless integration with third-party tools and cloud-native platforms. ATLAS APIs are a suite of extensibility interfaces designed to integrate, automate, and extend the capabilities of the ATLAS telemetry and data analytics platform. They support real-time and historical data workflows across motorsport, automotive, aerospace, and simulation environments. ## Key APIs and Usage #### [SQLRace API](../../../developer-resources/atlas/sql-race/index.md) Provides direct access to telemetry databases. Used for in-process data extraction and session creation. #### [Display API](../../../developer-resources/atlas/display-api/index.md) Enables creation of custom visualisations with native ATLAS look and feel. #### [Stream API](../../../developer-resources/secu4/stream_api/index.md) Built on gRPC and Protobuf for real-time telemetry streaming. Supports both reading and writing of telemetry packets. #### [Configuration API](../../../developer-resources/system-monitor/configuration_api/index.md) The Configuration API offers a cross-platform gRPC interface for ECU programming and monitoring, supporting integration with clients in multiple languages. --- # Overview Welcome to the Developer Resources section! This area provides everything you need to get started and succeed when integrating with the ATLAS platform. Here you'll find: - **Getting Started Guides:** Step-by-step instructions for setting up your development environment. - **Tools & Utilities:** Recommended tools and extensions to streamline your workflow. - **Best Practices:** Coding standards, review guidelines, and tips for effective collaboration. - **Reference Materials:** API documentation, architecture overviews, and troubleshooting resources. ## Developer Community Looking for community examples and projects? Check out the [ATLAS Developer Community](https://github.com/atlas-dev-hub) on GitHub. This is the home for community-contributed examples, custom implementations, and collaborative projects that utilize our APIs. Whether you're building a custom dashboard, creating a new tool, or sharing insights, the Developer Community is the place to showcase your work and connect with other developers. For official documentation and curated examples, you're in the right place. For open source products like the Stream API and Protocol definitions, visit [Software Products](https://github.com/Software-Products). ## API Areas
- :material-package-variant: __[NuGet Packages](./atlas/nuget/index.md)__ --- Access all McLaren API packages from the GitHub packages feed, covering data, display, RTA, and streaming. - :material-database: __[SQL Race API](./atlas/sql-race/index.md)__ --- Read, write and record telemetry data using ATLAS's core data layer — sessions, parameters, laps and events. - :material-robot: __[Automation API](./atlas/automation-api/index.md)__ --- Automate ATLAS workflows: session loading, workbook generation, display configuration, and data access. - :material-monitor-dashboard: __[Display API](./atlas/display-api/index.md)__ --- Build custom display plugins that integrate natively with ATLAS, with full access to session and parameter data. - :material-transit-connection: __[Bridge Service](./secu4/bridge_service/index.md)__ --- Decode raw ADS streams and convert telemetry to engineering values via the Stream Protocol. - :material-library: __[Support Library](./secu4/support_library/index.md)__ --- Streaming support library for the MA DataPlatforms ecosystem. - :material-file-search: __[Indexing Services](./secu4/indexing_services/index.md)__ --- Microservice solution for file and configuration indexing across the data platform. - :material-swap-horizontal: __[Stream API](./secu4/stream_api/index.md)__ --- gRPC-based interface for real-time data streaming to and from Kafka brokers as part of the Open Streaming Architecture. - :material-cog: __[Configuration API](./system-monitor/configuration_api/index.md)__ --- Multi-platform gRPC API for ECU programming and monitoring — supports C#, C++, Python, Java, and Go. - :material-application-cog: __[System Monitor ActiveX](./system-monitor/active-x/index.md)__ --- OLE Automation interface for controlling System Monitor and requesting information from external applications. - :material-cloud-sync: __[RTA](./rta/worked-guide/index.md)__ --- Remote Telemetry Architecture — API and toolkit services for integrating ATLAS with existing environments over gRPC.
Explore the topics in this section to accelerate your development. --- # Overview McLaren NuGet libraries are available from our [GitHub packages feed](https://github.com/orgs/mat-docs/packages?repo_name=packages). A free [GitHub](https://github.com/) account is required. [Setup Packages Feed](setup.md){ .md-button .md-button--primary } !!! info The packages feed includes all dependencies that are not available on [nuget.org](https://nuget.org/). This includes some third-party libraries that have been patched for extended maintenance. ## Data and Recorder APIs [_Read and Write McLaren Telemetry data_](../sql-race/index.md) **MESL.SQLRace.API** : Data API, for reading and writing telemetry with McLaren SSN2 files and SQL Server databases; Supports .NET Framework 4.8 **MESL.SQLRace.RTA** : Adds [RTA](#rta) support for SQL Race (above); Supports .NET Framework 4.8 **MAT.ATLAS.SupportFiles** : Adds SSN (read-only) support to SQL Race (above), and provides headless Recorder APIs; Supports .NET Framework 4.8; Requires [Microsoft Visual C++ 2015-19 Redistributable (x64)](https://aka.ms/vs/16/release/vc_redist.x64.exe) ## ATLAS Display API [_Create custom display plugins for ATLAS_](../display-api/index.md) **Atlas.DisplayAPI** : ATLAS plugin API for creating custom displays; Supports .NET Framework 4.8 ## RTA [_Integrate ATLAS with existing environments using Web Services_](../../rta/downloads.md) **MAT.OCS.RTA.API** : Client API for testing and integration; Supports .NET Standard 2.0 and 2.1 **MAT.OCS.RTA.Services.AspNetCore** : By far the quickest way to get a new RTA service off the ground; Supports ASP.NET Core 2.1, 3.1 and 5.0 **MAT.OCS.RTA.Toolkit.API.GrpcClients** : Pre-compiled gRPC clients for the RTA Toolkit Services; Supports .NET Standard 2.1 and .NET 5.0 ## ATLAS Streaming **MAT.OCS.Streaming** : Streaming telemetry client compatible with the ATLAS technology platform **MAT.OCS.Streaming.Kafka** : Provides an Apache Kafka client for MAT.OCS.Streaming **MAT.OCS.Streaming.Codecs.Protobuf** : Provides a faster, more compact serialization for telemetry data and samples; Prefer this codec when working with large topics --- # Setup Guide First, [Create a Personal Access Token (PAT)](https://github.com/settings/tokens/new) including the _read:packages_ scope. You'll need a free [GitHub account](https://github.com/signup). Copy the token for reference. It will look something like: ghp_yLKCoOLozKbaErgl4mCLDtkEKFw6Uq4D3Yyx === "Visual Studio" Select _Tools > NuGet Package Manager > Package Manager Settings > Package Sources_ Click the ➕ icon, and fill in: _Name:_ McLaren Applied GitHub Packages _Source:_ https://nuget.pkg.github.com/mat-docs/index.json ![NuGet Sources dialog](assets/vs-nuget-sources.png) Click _Update_ to save the changes. When prompted for credentials, enter your GitHub username and PAT: ![Credentials dialog](assets/credentials-dialog.png){ style="background-color: white; border: 1px solid #282829; padding-top: 4px" } === "Command Line" From the terminal, replacing `"your_username"` and `"your_pat"`: nuget sources Add -Name "McLaren Applied GitHub Packages" -Username "your_username" -Password "your_pat" -Source "https://nuget.pkg.github.com/mat-docs/index.json" or: dotnet nuget add source --name "McLaren Applied GitHub Packages" --username "your_username" --password "your_pat" "https://nuget.pkg.github.com/mat-docs/index.json" === "Adding to NuGet.Config" In scripted environments, it can be convenient to add the PAT to a _NuGet.Config_ file. !!! caution Do not commit credentials into source control. ```xml ``` To add this `#!xml ` section from an _Azure DevOps_ YAML build: ```yaml steps: - task: NuGetToolInstaller@1 inputs: versionSpec: "5.8.0" - task: DotNetCoreCLI@2 displayName: "dotnet nuget update source (authenticate)" inputs: command: custom custom: nuget arguments: update source McLarenApplied --username "your_username" --password "your_pat" --store-password-in-clear-text --configfile NuGet.Config verbosity: "Normal" ``` --- # Overview The configuration API is a new multi-platform API for ECU programming and monitoring, which makes it easier to handle multiple units, and to integrate across the network with clients written in languages including C#, C++, Python, Java or Go. The Configuration Service provides a modern, cross-platform [gRPC](https://grpc.io/) interface, and mediates access to the units – via System Monitor – across the network from multiple clients. ![Diagram of the configuration API](assets/ConfigAPI_light.png#only-light) ![Diagram of the configuration API](assets/ConfigAPI_dark.png#only-dark) All information available via the COM API will be available via the Configuration Service. All calls to the API are managed on a single threaded basis and any subsequent calls are blocked until previous requests are completed. Measurement values are cached within the API server to provide the fastest possible response time to requests, these calls are not blocked due to other actions. The rate of cache update is configurable. A .NET Standard [client NuGet package](https://github.com/mat-docs/packages/pkgs/nuget/SystemMonitorConfigurationAPI) and [interface definitions](https://github.com/mat-docs/System.Monitor.Configuration.API.Client.Sample/tree/cf34ad478c086dc43d2863b74f5516946049eaca/Protos) are provided to make it easy to create interoperable tools using any mainstream language or platform. This feature is available from System Monitor version 8.85 onwards. ## Migrated Methods Tracker --- # Protocol Documentation ## Table of Contents - [Protos/system_monitor_common.proto](#Protos_system_monitor_common-proto) - [AppParametersFileRequest](#system_monitor_common-AppParametersFileRequest) - [AppParametersRequest](#system_monitor_common-AppParametersRequest) - [AppRequest](#system_monitor_common-AppRequest) - [ConversionRequest](#system_monitor_common-ConversionRequest) - [FileRequest](#system_monitor_common-FileRequest) - [ParameterRequest](#system_monitor_common-ParameterRequest) - [ParametersFileRequest](#system_monitor_common-ParametersFileRequest) - [ParametersRequest](#system_monitor_common-ParametersRequest) - [Return](#system_monitor_common-Return) - [BufferType](#system_monitor_common-BufferType) - [ByteOrder](#system_monitor_common-ByteOrder) - [ConversionType](#system_monitor_common-ConversionType) - [DataType](#system_monitor_common-DataType) - [ErrorCode](#system_monitor_common-ErrorCode) - [ErrorStatus](#system_monitor_common-ErrorStatus) - [EventPriority](#system_monitor_common-EventPriority) - [FileType](#system_monitor_common-FileType) - [LoggingType](#system_monitor_common-LoggingType) - [ParameterType](#system_monitor_common-ParameterType) - [Reason](#system_monitor_common-Reason) - [TriggerOperator](#system_monitor_common-TriggerOperator) - [TriggerType](#system_monitor_common-TriggerType) - [Protos/system_monitor_logging.proto](#Protos_system_monitor_logging-proto) - [AddParameterRequest](#system_monitor_logging-AddParameterRequest) - [AddVirtualParameterRequest](#system_monitor_logging-AddVirtualParameterRequest) - [ChannelProperties](#system_monitor_logging-ChannelProperties) - [ChannelPropertiesReply](#system_monitor_logging-ChannelPropertiesReply) - [ChannelRequest](#system_monitor_logging-ChannelRequest) - [ClearRequest](#system_monitor_logging-ClearRequest) - [ConfigReply](#system_monitor_logging-ConfigReply) - [DownloadProgressReply](#system_monitor_logging-DownloadProgressReply) - [DownloadReply](#system_monitor_logging-DownloadReply) - [DownloadRequest](#system_monitor_logging-DownloadRequest) - [GetSessionDetailReply](#system_monitor_logging-GetSessionDetailReply) - [GetSessionDetailRequest](#system_monitor_logging-GetSessionDetailRequest) - [LoggingChannelValue](#system_monitor_logging-LoggingChannelValue) - [LoggingDurationReply](#system_monitor_logging-LoggingDurationReply) - [LoggingOffsetReply](#system_monitor_logging-LoggingOffsetReply) - [LoggingOffsetRequest](#system_monitor_logging-LoggingOffsetRequest) - [LoggingParameter](#system_monitor_logging-LoggingParameter) - [LoggingParametersReply](#system_monitor_logging-LoggingParametersReply) - [SetSessionDetailRequest](#system_monitor_logging-SetSessionDetailRequest) - [SlotCountReply](#system_monitor_logging-SlotCountReply) - [SlotPercentageReply](#system_monitor_logging-SlotPercentageReply) - [Trigger](#system_monitor_logging-Trigger) - [TriggerCondition](#system_monitor_logging-TriggerCondition) - [TriggerRequest](#system_monitor_logging-TriggerRequest) - [TriggersReply](#system_monitor_logging-TriggersReply) - [WrapReply](#system_monitor_logging-WrapReply) - [WrapRequest](#system_monitor_logging-WrapRequest) - [SystemMonitorLogging](#system_monitor_logging-SystemMonitorLogging) - [Protos/system_monitor_parameter.proto](#Protos_system_monitor_parameter-proto) - [AddressReply](#system_monitor_parameter-AddressReply) - [AppArray1dParameterValuesRequest](#system_monitor_parameter-AppArray1dParameterValuesRequest) - [AppArray2dParameterValuesRequest](#system_monitor_parameter-AppArray2dParameterValuesRequest) - [AppParameterValuesRequest](#system_monitor_parameter-AppParameterValuesRequest) - [AppStringParameterValuesRequest](#system_monitor_parameter-AppStringParameterValuesRequest) - [AppTypeRequest](#system_monitor_parameter-AppTypeRequest) - [Array1dParameterErrorsReply](#system_monitor_parameter-Array1dParameterErrorsReply) - [Array1dParameterSetValue](#system_monitor_parameter-Array1dParameterSetValue) - [Array1dParameterValue](#system_monitor_parameter-Array1dParameterValue) - [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) - [Array1dValues](#system_monitor_parameter-Array1dValues) - [Array2dParameterErrorsReply](#system_monitor_parameter-Array2dParameterErrorsReply) - [Array2dParameterSetValue](#system_monitor_parameter-Array2dParameterSetValue) - [Array2dParameterValue](#system_monitor_parameter-Array2dParameterValue) - [Array2dValueReply](#system_monitor_parameter-Array2dValueReply) - [Array2dValues](#system_monitor_parameter-Array2dValues) - [AxisParametersReply](#system_monitor_parameter-AxisParametersReply) - [BitMaskReply](#system_monitor_parameter-BitMaskReply) - [BitShiftReply](#system_monitor_parameter-BitShiftReply) - [ByteOrderReply](#system_monitor_parameter-ByteOrderReply) - [CANParameterProperties](#system_monitor_parameter-CANParameterProperties) - [CANParameterPropertiesReply](#system_monitor_parameter-CANParameterPropertiesReply) - [Conversion](#system_monitor_parameter-Conversion) - [ConversionListReply](#system_monitor_parameter-ConversionListReply) - [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) - [ConversionTypeReply](#system_monitor_parameter-ConversionTypeReply) - [ExternalParameterRequest](#system_monitor_parameter-ExternalParameterRequest) - [ExternalReply](#system_monitor_parameter-ExternalReply) - [ExternalRequest](#system_monitor_parameter-ExternalRequest) - [FormulaConversionReply](#system_monitor_parameter-FormulaConversionReply) - [FormulaConversionRequest](#system_monitor_parameter-FormulaConversionRequest) - [LoggableReply](#system_monitor_parameter-LoggableReply) - [MapPropertiesReply](#system_monitor_parameter-MapPropertiesReply) - [OffsetReply](#system_monitor_parameter-OffsetReply) - [OffsetRequest](#system_monitor_parameter-OffsetRequest) - [Parameter](#system_monitor_parameter-Parameter) - [ParameterErrorsReply](#system_monitor_parameter-ParameterErrorsReply) - [ParameterGroup](#system_monitor_parameter-ParameterGroup) - [ParameterGroupsReply](#system_monitor_parameter-ParameterGroupsReply) - [ParameterListReply](#system_monitor_parameter-ParameterListReply) - [ParameterProperties](#system_monitor_parameter-ParameterProperties) - [ParameterPropertiesReply](#system_monitor_parameter-ParameterPropertiesReply) - [ParameterSetValue](#system_monitor_parameter-ParameterSetValue) - [ParameterTypeRequest](#system_monitor_parameter-ParameterTypeRequest) - [ParameterValue](#system_monitor_parameter-ParameterValue) - [ParametersReply](#system_monitor_parameter-ParametersReply) - [RationalConversionReply](#system_monitor_parameter-RationalConversionReply) - [RationalConversionRequest](#system_monitor_parameter-RationalConversionRequest) - [RowDetailsReply](#system_monitor_parameter-RowDetailsReply) - [RowValues](#system_monitor_parameter-RowValues) - [StringParameterErrorsReply](#system_monitor_parameter-StringParameterErrorsReply) - [StringParameterSetValue](#system_monitor_parameter-StringParameterSetValue) - [StringParameterValue](#system_monitor_parameter-StringParameterValue) - [StringValueReply](#system_monitor_parameter-StringValueReply) - [TableConversion](#system_monitor_parameter-TableConversion) - [TableConversionReply](#system_monitor_parameter-TableConversionReply) - [TableConversionRequest](#system_monitor_parameter-TableConversionRequest) - [TextConversion](#system_monitor_parameter-TextConversion) - [TextConversionReply](#system_monitor_parameter-TextConversionReply) - [TextConversionRequest](#system_monitor_parameter-TextConversionRequest) - [UndoRequest](#system_monitor_parameter-UndoRequest) - [ValueReply](#system_monitor_parameter-ValueReply) - [WarningLimitsReply](#system_monitor_parameter-WarningLimitsReply) - [WarningLimitsRequest](#system_monitor_parameter-WarningLimitsRequest) - [SystemMonitorParameter](#system_monitor_parameter-SystemMonitorParameter) - [Protos/system_monitor_project.proto](#Protos_system_monitor_project-proto) - [ActiveAppReply](#system_monitor_project-ActiveAppReply) - [AppFileRequest](#system_monitor_project-AppFileRequest) - [AppReply](#system_monitor_project-AppReply) - [Application](#system_monitor_project-Application) - [CANMergeRequest](#system_monitor_project-CANMergeRequest) - [CANRequest](#system_monitor_project-CANRequest) - [CompareAppReply](#system_monitor_project-CompareAppReply) - [CompareAppRequest](#system_monitor_project-CompareAppRequest) - [CompareParameter](#system_monitor_project-CompareParameter) - [DTVModifiedReply](#system_monitor_project-DTVModifiedReply) - [DTVSaveCopyRequest](#system_monitor_project-DTVSaveCopyRequest) - [DTVSaveIncrementRequest](#system_monitor_project-DTVSaveIncrementRequest) - [DTVSaveRequest](#system_monitor_project-DTVSaveRequest) - [DTVSavedOnReply](#system_monitor_project-DTVSavedOnReply) - [DetailsRequest](#system_monitor_project-DetailsRequest) - [EnableRequest](#system_monitor_project-EnableRequest) - [ErrorDefinition](#system_monitor_project-ErrorDefinition) - [ErrorDefinitionsReply](#system_monitor_project-ErrorDefinitionsReply) - [ErrorInstance](#system_monitor_project-ErrorInstance) - [ErrorReply](#system_monitor_project-ErrorReply) - [Event](#system_monitor_project-Event) - [EventReply](#system_monitor_project-EventReply) - [EventRequest](#system_monitor_project-EventRequest) - [EventsReply](#system_monitor_project-EventsReply) - [ExistsReply](#system_monitor_project-ExistsReply) - [ExistsRequest](#system_monitor_project-ExistsRequest) - [FileDetailsReply](#system_monitor_project-FileDetailsReply) - [FileNameRequest](#system_monitor_project-FileNameRequest) - [FileNewRequest](#system_monitor_project-FileNewRequest) - [FileOpenRequest](#system_monitor_project-FileOpenRequest) - [FileReply](#system_monitor_project-FileReply) - [FileSaveRequest](#system_monitor_project-FileSaveRequest) - [GetAppDetailsReply](#system_monitor_project-GetAppDetailsReply) - [GetBuildNumberReply](#system_monitor_project-GetBuildNumberReply) - [GetVersionNumberReply](#system_monitor_project-GetVersionNumberReply) - [MatlabDTVRequest](#system_monitor_project-MatlabDTVRequest) - [MatlabRequest](#system_monitor_project-MatlabRequest) - [MatlabSelectedRequest](#system_monitor_project-MatlabSelectedRequest) - [MultiAppReply](#system_monitor_project-MultiAppReply) - [MultiAppRequest](#system_monitor_project-MultiAppRequest) - [PGVIDReply](#system_monitor_project-PGVIDReply) - [ParameterIdRequest](#system_monitor_project-ParameterIdRequest) - [ProjectCloseRequest](#system_monitor_project-ProjectCloseRequest) - [ProjectCreateRequest](#system_monitor_project-ProjectCreateRequest) - [ProjectExportRequest](#system_monitor_project-ProjectExportRequest) - [ProjectImportRequest](#system_monitor_project-ProjectImportRequest) - [ProjectSaveAsRequest](#system_monitor_project-ProjectSaveAsRequest) - [ProjectSaveRequest](#system_monitor_project-ProjectSaveRequest) - [ReasonCode](#system_monitor_project-ReasonCode) - [ReprogramRequest](#system_monitor_project-ReprogramRequest) - [SensorRequest](#system_monitor_project-SensorRequest) - [SlotActiveRequest](#system_monitor_project-SlotActiveRequest) - [SlotReply](#system_monitor_project-SlotReply) - [SlotRequest](#system_monitor_project-SlotRequest) - [SyncedReply](#system_monitor_project-SyncedReply) - [SystemMonitorProject](#system_monitor_project-SystemMonitorProject) - [Protos/system_monitor_system.proto](#Protos_system_monitor_system-proto) - [BatchModeRequest](#system_monitor_system-BatchModeRequest) - [CreatePGVReply](#system_monitor_system-CreatePGVReply) - [CreatePGVRequest](#system_monitor_system-CreatePGVRequest) - [DeviceProperties](#system_monitor_system-DeviceProperties) - [DevicePropertiesReply](#system_monitor_system-DevicePropertiesReply) - [FolderReply](#system_monitor_system-FolderReply) - [LicenceDetailsReply](#system_monitor_system-LicenceDetailsReply) - [LiveLoggingReply](#system_monitor_system-LiveLoggingReply) - [LiveLoggingRequest](#system_monitor_system-LiveLoggingRequest) - [LiveUpdateRequest](#system_monitor_system-LiveUpdateRequest) - [MultiApplicationBaseInfo](#system_monitor_system-MultiApplicationBaseInfo) - [MultiApplicationBasesReply](#system_monitor_system-MultiApplicationBasesReply) - [MultiApplicationBasesRequest](#system_monitor_system-MultiApplicationBasesRequest) - [OnlineRequest](#system_monitor_system-OnlineRequest) - [SendMessageReply](#system_monitor_system-SendMessageReply) - [SendMessageRequest](#system_monitor_system-SendMessageRequest) - [StatusReply](#system_monitor_system-StatusReply) - [UnitByIndexRequest](#system_monitor_system-UnitByIndexRequest) - [UnitByIndexTypeRequest](#system_monitor_system-UnitByIndexTypeRequest) - [UnitInfo](#system_monitor_system-UnitInfo) - [UnitListReply](#system_monitor_system-UnitListReply) - [UnitNameReply](#system_monitor_system-UnitNameReply) - [LinkStatus](#system_monitor_system-LinkStatus) - [SystemMonitorSystem](#system_monitor_system-SystemMonitorSystem) - [Protos/system_monitor_virtual.proto](#Protos_system_monitor_virtual-proto) - [AddGroupRequest](#system_monitor_virtual-AddGroupRequest) - [VirtualExportRequest](#system_monitor_virtual-VirtualExportRequest) - [VirtualGroupReply](#system_monitor_virtual-VirtualGroupReply) - [VirtualGroupRequest](#system_monitor_virtual-VirtualGroupRequest) - [VirtualGroupsReply](#system_monitor_virtual-VirtualGroupsReply) - [VirtualParameter](#system_monitor_virtual-VirtualParameter) - [VirtualParameterDataTypeRequest](#system_monitor_virtual-VirtualParameterDataTypeRequest) - [VirtualParameterProperties](#system_monitor_virtual-VirtualParameterProperties) - [VirtualParameterPropertiesReply](#system_monitor_virtual-VirtualParameterPropertiesReply) - [VirtualParameterRequest](#system_monitor_virtual-VirtualParameterRequest) - [VirtualReply](#system_monitor_virtual-VirtualReply) - [VirtualsRequest](#system_monitor_virtual-VirtualsRequest) - [SystemMonitorVirtual](#system_monitor_virtual-SystemMonitorVirtual) - [Scalar Value Types](#scalar-value-types)

Top

## Protos/system_monitor_common.proto ### AppParametersFileRequest System Monitor Application Parameters File Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application identifier | | parameter_ids | [string](#string) | repeated | Parameter identifiers. | | file_path | [string](#string) | | File path. | ### AppParametersRequest System Monitor Application Parameters Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. 0 for all applications including Virtual and CAN. | | parameter_ids | [string](#string) | repeated | Parameter identifiers. | ### AppRequest System Monitor App Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | ### ConversionRequest System Monitor Conversion Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | | conversion_id | [string](#string) | | Conversion id. | ### FileRequest System Monitor File Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_path | [string](#string) | | File path. | ### ParameterRequest System Monitor Parameter Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | | parameter_id | [string](#string) | | Parameter id. | ### ParametersFileRequest System Monitor Parameters File Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_ids | [string](#string) | repeated | Parameter identifiers. | | file_path | [string](#string) | | File path. | ### ParametersRequest System Monitor Parameters Request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_ids | [string](#string) | repeated | Identifiers of parameters to request | ### Return System Monitor Return Message | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | return_code | [ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### BufferType System Monitor Buffer Types | Name | Number | Description | | ---- | ------ | ----------- | | unit_buffer | 0 | Data Changes in Unit. | | edit_buffer | 1 | Data Changes in Edit Buffer. | | unit_and_edit_buffer | 2 | Data Changes in Unit & Edit Buffer. | ### ByteOrder System Monitor Byte Orders | Name | Number | Description | | ---- | ------ | ----------- | | msb_first | 0 | Most significant bit first | | msb_last | 1 | Most significant bit last | ### ConversionType System Monitor Conversion Types | Name | Number | Description | | ---- | ------ | ----------- | | rational | 0 | Rational conversion | | table | 1 | Table conversion | | text | 2 | Text conversion | | formula | 3 | Formula conversion | ### DataType System Monitor Data Types | Name | Number | Description | | ---- | ------ | ----------- | | ubyte | 0 | Unsigned byte | | byte | 1 | Signed Byte | | uword | 2 | Unsigned word | | word | 3 | Signed word | | ulong | 4 | Unsigned long | | long | 5 | Signed long | | float | 6 | Float | | unknown | 7 | Unknown | | qword | 8 | Quad word | | sqword | 9 | Signed quad word | | double | 10 | Double | ### ErrorCode System Monitor Error Codes | Name | Number | Description | | ---- | ------ | ----------- | | no_error | 0 | No error, function successful | | no_project | -1 | No project loaded in SM-V7 | | no_licence | -2 | No licence for requested function | | non_specific | -3 | Non-specific error | | data_version_mismatch | -4 | Data version mismatch | | no_data_version | -5 | No data version loaded | | no_program_version | -6 | No program version loaded | | no_ECU | -7 | No ECU connected | | invalid_file | -8 | Invalid file specified | | no_application | -9 | The open project does not cover the specified application | | application_inactive | -10 | Application must be active in order to support this operation | | live_updates_on | -11 | Operation not allowed while live updates are on | | TAGtronic_onlu | -12 | This command only valid for TAGtronic systems. | | SM_busy | -13 | SM cannot accept API calls at the moment (might be in critical operation). | | message_argument_mismatch | -20 | Type mismatch in argument of SendMessage , expected array of WORDs passed by reference | | message_dimension_mismatch | -21 | Array dimension mismatch in argument of SendMessage, expected a 1-dimensional array | | message_lower_bound_non_zero | -22 | Lower bound of argument of SendMessage call was not 0 | | bounds_error | -23 | Unexpected error getting bounds of SendMessage argument | | message_argument_error | -24 | Error in ReplyMessage argument | | message_argument_invalid | -25 | Error in SendMessage argument | | fdl_not_parsed | -26 | Virtual Parameter contains invalid FDL. | | conversion_invalid | -27 | Conversion for Parameter does not exist/is invalid. | | parameter_invalid | -28 | Parameter name does not exist/is invalid. | | parameter_override_not_allowed | -29 | Parameter exists and not allowed to override. | | bad_state | -30 | Request cannot be actioned as incorrect SM state. | | invalid_command | -31 | Command passed to SM is not valid. | | no_data_present | -32 | Missing document or similar | | bad_memory_allocation | -33 | Memory allocation failed | | partially_complete | -34 | Operation only partially complete. Example: ClearRemoteParams left Parameters used in trigger conditions. | | document_full | -35 | Indicates the current document (config) is full. | | parameter_identifier_already_exists | -36 | Parameter Identifier already exists in another application. | | parameter_read_only | -37 | Parameter is read-only - write access denied. | | parameter_non_live_tuneable | -38 | Parameter is non-live tuneable - write access denied. | | group_not_found | -39 | Requested group is not found | | file_requires_saving | -40 | Previous virtual parameters file has been modified and not saved | | frequency_overridden | -41 | No customer base found for the project being opened | | no_customer_base | -42 | Requested parameter not found | | parameter_not_found | -100 | Specified parameter does not exist in current program version | | error_read_only | -101 | Tried to set a read only parameter | | error_limits | -102 | Tried to set a parameter outside its limits | | error_monotony | -103 | Axis monotony restrictions not adhered to | | error_axis_pt | -104 | Tried to get or set parameter or axis value at invalid breakpoint | | error_address | -105 | Parameter had invalid address | | error_non_num | -106 | String containing non numeric characters supplied to set function | | error_size | -107 | Exceeded specified size or array or string parameter | | error_live_tune | -108 | Live tune enabled, failed to update value in ECU | | error_intp | -109 | Tried to get interpolated value but no input quantity or axis points | | error_activelayer | -110 | No active live auto tune "layer" (group) | | error_tolerance | -111 | Operating point is outside breakpoint tolerances (e.g. for live auto tune) | | error_axis_change | -112 | Number of axis breakpoints has changed between program versions | | error_no_live_tune | -113 | Parameter is not modifiable in live tune area (warning only, not error) | | error_validation | -114 | The attempt to validate the value in the unit failed (not the same as the value being invalid) | | error_live_tune_data_invalid | -115 | The live tune data is invalid | | error_serial_not_found | -116 | Serial number not found for sensor channel | | error_unknown | -117 | An unknown error | | error_cancel | -118 | Attempt to set value cancelled (by user or because of invalid input) | | error_locked_param | -119 | Access denied, parameter locked by RDA | | error_value_not_matching_entry | -120 | Value set does not match value in the unit | | detail_unknown | -200 | Session detail does not exist | | dump_row_data_failed | -201 | Failed to dump row data | | live_update_failed | -300 | Failed to enter live tune | | online_failed | -301 | Failed to go online | | download_data_failed | -302 | Failed to download data | | system_not_running | -303 | System monitor is only running in OLE mode, hence can't perform requested action | | parameter_locked | -304 | Attempt to access a locked parameter | | comms_base | -1000 | Communications base error | ### ErrorStatus System Monitor Error Statuses | Name | Number | Description | | ---- | ------ | ----------- | | status_unknown | 0 | Unknown error status. | | status_current | 1 | Current error status. | | status_logged | 2 | Logged error status. | ### EventPriority System Monitor Event Priority | Name | Number | Description | | ---- | ------ | ----------- | | event_high | 0 | High priority event. | | event_medium | 1 | Medium priority event. | | event_low | 2 | Low priority event. | | event_debug | 3 | Debug event. | ### FileType System Monitor File Types | Name | Number | Description | | ---- | ------ | ----------- | | Project | 0 | Project file | | PGV | 1 | Program version file | | DTV | 2 | Data version file | | desktop | 3 | Desktop file | | logging_config | 4 | Logging config file | | virtuals | 5 | Virtual parameter file | | CAN | 6 | Can parameter file | | live_logging | 7 | Live logging configuration file | | pot_board | 8 | Pot board file | ### LoggingType System Monitor Logging Types | Name | Number | Description | | ---- | ------ | ----------- | | frequency | 0 | Frequency value | | cylinder | 1 | Engine segment (cylinder) | | cycle | 2 | Cycle (wValue not used). | | unknown_logging | 3 | Unknown | | edge | 4 | Edge | ### ParameterType System Monitor Parameter Types | Name | Number | Description | | ---- | ------ | ----------- | | undefined | 0 | Undefined parameter type | | scalar | 1 | Scalar parameter | | axis_1 | 2 | Axis 1 parameter | | axis_2 | 4 | Axis 2 parameter | | array | 16 | Array parameter | | string | 32 | String parameter | | ecu | 128 | Ecu parameter | | can | 256 | Can parameter | | tsb | 512 | TAG Sensor Bus parameter | | virtual | 1024 | Virtual parameter | | axis | 196608 | Axis parameter | | input | 268435456 | Input parameter | | measurement | 268437376 | Measurement parameter | ### Reason System Monitor Reasons | Name | Number | Description | | ---- | ------ | ----------- | | none | 0 | No difference. | | absent | 1 | Not present in program version of the candidate. | | different | 2 | Major difference that prevents merge. | | equal | 4 | Same data type, size and value. | | different_value | 8 | Same data type, size, units, diff value. | | different_size | 16 | Same data type, different size (maps only). | | different_conv | 32 | Different display format conversion. | | different_units | 64 | Different units. | | different_type | 128 | Different data type. | | different_comment | 256 | Different comment. | | different_def_value | 512 | Different default value. | | absent_master | 1024 | Not present in program version of the edit buffer. | | locked | 268435456 | Locked. | ### TriggerOperator System Monitor Trigger Operators | Name | Number | Description | | ---- | ------ | ----------- | | equals | 0 | Equals. | | less_than | 1 | Less than. | | greater_than | 2 | Greater than. | | not_equal_to | 3 | Not equal to. | | greater_than_or_equal | 4 | Greater than or equal to. | | less_than_or_equal | 5 | Less than or equal to. | ### TriggerType System Monitor Trigger Types | Name | Number | Description | | ---- | ------ | ----------- | | on_data | 0 | On Data (with parameter) | | driver_push | 1 | Driver push (not supported) | | ignition_on | 2 | Ignition On | | lap_trigger | 3 | Lap trigger | | no_condition | 4 | No condition | | external_trigger | 5 | External trigger |

Top

## Protos/system_monitor_logging.proto ### AddParameterRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id | | parameter_id | [string](#string) | | Unique parameter identifier | | logging_rate | [LoggingChannelValue](#system_monitor_logging-LoggingChannelValue) | repeated | Logging channel values | ### AddVirtualParameterRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Unique parameter identifier | | logging_rate | [LoggingChannelValue](#system_monitor_logging-LoggingChannelValue) | repeated | The logging channel value | ### ChannelProperties | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | The __1__ based channel index to modify | | name | [string](#string) | | Name of the channel | | log_logging | [bool](#bool) | | Retrieves the Log to Unit flag | | log_telemetry | [bool](#bool) | | Retrieves the telemetry flag | | logging_rate | [double](#double) | | The channel logging rate | | telemetry_rate | [double](#double) | | The telemetry logging rate | | trigger_rearm | [bool](#bool) | | Trigger re-arms after stop condition | | slot | [uint32](#uint32) | | Retrieves the slot position of the configuration defining the channel | ### ChannelPropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | channels | [ChannelProperties](#system_monitor_logging-ChannelProperties) | repeated | The logging channel properties | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### ChannelRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | The 0 based channel index to modify | | name | [string](#string) | | Name of the channel | | log_to_unit | [bool](#bool) | | Retrieves the Log to Unit flag | | log_telemetry | [bool](#bool) | | Retrieves the telemetry flag | | trigger_rearm | [bool](#bool) | | Trigger re-arms after stop condition | ### ClearRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | remove_triggers | [bool](#bool) | | Request System Monitor removes all parameters, including those used by triggers | ### ConfigReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | config_name | [string](#string) | | Name of the ECU logging configuration | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### DownloadProgressReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | in_progress | [bool](#bool) | | Indicates whether a download is in progress | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### DownloadReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | optional_value | [string](#string) | | Indicates whether the logging configuration download is complete | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### DownloadRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | optional_app_id | [uint32](#uint32) | | Application id | | optional_parameter_id | [string](#string) | | Unique identifier of logging configuration parameter | | optional_delay_ms | [uint32](#uint32) | | Read delay in ms before retrieving parameter value allowing ECU configuration processing time | ### GetSessionDetailReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Name of session detail | | value | [string](#string) | | Value of session detail | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### GetSessionDetailRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Name of session detail | ### LoggingChannelValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | channel_id | [uint32](#uint32) | | Channel id | | type | [system_monitor_common.LoggingType](#system_monitor_common-LoggingType) | | Logging type | | value | [uint32](#uint32) | | Logging channel value | ### LoggingDurationReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | estimated_time | [google.protobuf.Duration](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Duration.html) | | Estimated time available in recording memory | | estimated_laps | [double](#double) | | Estimated laps available in recording memory. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### LoggingOffsetReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | offset | [uint32](#uint32) | | The current logging offset | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### LoggingOffsetRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | offset | [uint32](#uint32) | | The logging offset | ### LoggingParameter | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id | | parameter_id | [string](#string) | | Parameter id | | parameter_name | [string](#string) | | Parameter name | | parameter_description | [string](#string) | | Parameter description | | data_size | [uint32](#uint32) | | Size of data | | values | [LoggingChannelValue](#system_monitor_logging-LoggingChannelValue) | repeated | Logging channel values | | slot | [uint32](#uint32) | | Slot position of configuration defining the parameter | ### LoggingParametersReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [LoggingParameter](#system_monitor_logging-LoggingParameter) | repeated | The logging parameters | | channel_names | [string](#string) | repeated | The channel names | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### SetSessionDetailRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Name of session detail | | value | [string](#string) | | New value for session detail | ### SlotCountReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | slot_count | [uint32](#uint32) | | Number of available slots | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### SlotPercentageReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id | | parameter_id | [string](#string) | | Parameter id | | slot_percentage | [double](#double) | | The current percentage | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### Trigger | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | The 0 based channel index to retrieve | | start_conditions | [TriggerCondition](#system_monitor_logging-TriggerCondition) | repeated | The trigger start conditions | | stop_conditions | [TriggerCondition](#system_monitor_logging-TriggerCondition) | repeated | The trigger start conditions | | start_post_trigger | [int32](#int32) | | Retrieves the delay for the start trigger | | stop_post_trigger | [int32](#int32) | | Retrieves the delay for the stop trigger | | slot | [uint32](#uint32) | | Retrieves the slot position of the configuration defining the channel | ### TriggerCondition | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | Specifies which condition index (0 <= n < 6), where the first 3 are start conditions, and the later, stop conditions | | type | [system_monitor_common.TriggerType](#system_monitor_common-TriggerType) | | Retrieves the trigger condition type | | parameter_id | [string](#string) | | Retrieves the parameter used in > On Data = condition type. If any | | app_id | [uint32](#uint32) | | Retrieves the application ID, that the pszParamName belongs to | | operator | [system_monitor_common.TriggerOperator](#system_monitor_common-TriggerOperator) | | Retrieves the condition operator of the > On Data = type | | threshold | [double](#double) | | Retrieves the floating point number to compare in condition against specified parameter for type > On Data = | | repeat_count | [uint32](#uint32) | | Retrieves the number of times a condition should occur before it is considered true | ### TriggerRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | The 0 based channel index to set | | start_conditions | [TriggerCondition](#system_monitor_logging-TriggerCondition) | repeated | The trigger start conditions | | stop_conditions | [TriggerCondition](#system_monitor_logging-TriggerCondition) | repeated | The trigger end conditions | | start_post_trigger | [int32](#int32) | | The delay for the start trigger | | stop_post_trigger | [int32](#int32) | | The delay for the stop trigger | ### TriggersReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | triggers | [Trigger](#system_monitor_logging-Trigger) | repeated | The available triggers | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### WrapReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | wrap | [bool](#bool) | | Required wrapping memory strategy | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### WrapRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | wrap | [bool](#bool) | | Required wrapping memory strategy | ### SystemMonitorLogging | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | GetLoggingChannelProperties | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [ChannelPropertiesReply](#system_monitor_logging-ChannelPropertiesReply) | Gets the properties of the available logging channels | | SetLoggingChannelProperties | [ChannelRequest](#system_monitor_logging-ChannelRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Alters the properties of any given Remote Logging channel. (COM: SetRemoteChannelProperties) | | GetLoggingTriggers | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [TriggersReply](#system_monitor_logging-TriggersReply) | Retrieves the properties of all channel trigger settings | | SetLoggingTrigger | [TriggerRequest](#system_monitor_logging-TriggerRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Modify or create a given channel's trigger condition | | GetLoggingWrap | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [WrapReply](#system_monitor_logging-WrapReply) | Gets the remote logging config memory wrapping strategy | | SetLoggingWrap | [WrapRequest](#system_monitor_logging-WrapRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the remote logging config memory wrapping strategy | | GetLoggingOffset | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [LoggingOffsetReply](#system_monitor_logging-LoggingOffsetReply) | Gets the remote logging config offset | | SetLoggingOffset | [LoggingOffsetRequest](#system_monitor_logging-LoggingOffsetRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the remote logging config offset | | GetLoggingSessionDetails | [GetSessionDetailRequest](#system_monitor_logging-GetSessionDetailRequest) | [GetSessionDetailReply](#system_monitor_logging-GetSessionDetailReply) | Returns the current value of a specified session detail from the remote logging configuration | | SetLoggingSessionDetails | [SetSessionDetailRequest](#system_monitor_logging-SetSessionDetailRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets a specified session detail in the remote logging configuration | | GetLoggingDuration | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [LoggingDurationReply](#system_monitor_logging-LoggingDurationReply) | Get the estimated length in time and number of laps available for recording in the remote logging config | | GetLoggingParameterDetails | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [LoggingParametersReply](#system_monitor_logging-LoggingParametersReply) | Gets the remote logging parameter details from the config | | LoggingConfigDownloadInProgress | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [DownloadProgressReply](#system_monitor_logging-DownloadProgressReply) | Gets whether a logging config download is in progress | | LoggingConfigDownload | [DownloadRequest](#system_monitor_logging-DownloadRequest) | [DownloadReply](#system_monitor_logging-DownloadReply) | Downloads the remote logging configuration to the ECU then reads the given measurement value to ensure configuration write is complete | | LoggingConfigUpload | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Uploads the remote logging configuration from the ECU | | RemoveLoggingParameter | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Remove a parameter from the Remote Logging Configuration | | ClearAllLoggingParameters | [ClearRequest](#system_monitor_logging-ClearRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes all the parameters from the remote logging config except those used by triggers If the caller specifies a forceful removal all parameters will be removed and any >On Data= trigger conditions will be reset | | GetLoggingSlotsUsed | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [SlotCountReply](#system_monitor_logging-SlotCountReply) | Gets the number of logging slots | | GetLoggingSlotPercentage | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [SlotPercentageReply](#system_monitor_logging-SlotPercentageReply) | Gets the percentage of a logging slot used by a parameter | | GetECULoggingConfig | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [ConfigReply](#system_monitor_logging-ConfigReply) | Gets the logging configuration from the ECU | | AddLoggingParameter | [AddParameterRequest](#system_monitor_logging-AddParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Adds a parameter to the Remote Logging Configuration | | AddVirtualLoggingParameter | [AddVirtualParameterRequest](#system_monitor_logging-AddVirtualParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Adds a virtual parameter to the Remote Logging Configuration |

Top

## Protos/system_monitor_parameter.proto ### AddressReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | address | [uint32](#uint32) | | Parameter address. | | ident | [uint32](#uint32) | | Ident returned. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### AppArray1dParameterValuesRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameters | [Array1dParameterSetValue](#system_monitor_parameter-Array1dParameterSetValue) | repeated | Parameter values to set. | ### AppArray2dParameterValuesRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameters | [Array2dParameterSetValue](#system_monitor_parameter-Array2dParameterSetValue) | repeated | Parameter values to set. | ### AppParameterValuesRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameters | [ParameterSetValue](#system_monitor_parameter-ParameterSetValue) | repeated | Parameter values to set. | ### AppStringParameterValuesRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameters | [StringParameterSetValue](#system_monitor_parameter-StringParameterSetValue) | repeated | Parameter values to set. | ### AppTypeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application | | data_type | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | | Parameter data type. - One of ParameterType.Undefined, ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | ### Array1dParameterErrorsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [Array1dParameterValue](#system_monitor_parameter-Array1dParameterValue) | repeated | Parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array1dParameterSetValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | values | [double](#double) | repeated | Values for the parameter array. | ### Array1dParameterValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | values | [double](#double) | repeated | Values for the parameter array. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array1dValueReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [Array1dValues](#system_monitor_parameter-Array1dValues) | repeated | 1d Array parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array1dValues | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | values | [double](#double) | repeated | Values for the parameter array. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array2dParameterErrorsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [Array2dParameterValue](#system_monitor_parameter-Array2dParameterValue) | repeated | Parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array2dParameterSetValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | rows | [RowValues](#system_monitor_parameter-RowValues) | repeated | Values for the parameter array. | ### Array2dParameterValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | rows | [RowValues](#system_monitor_parameter-RowValues) | repeated | Values for the parameter array. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array2dValueReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [Array2dValues](#system_monitor_parameter-Array2dValues) | repeated | 2d Array parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Array2dValues | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | rows | [RowValues](#system_monitor_parameter-RowValues) | repeated | Values for the row. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### AxisParametersReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_ids | [string](#string) | repeated | Parameter identifiers. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### BitMaskReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | mask | [int32](#int32) | | Parameter bit mask. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### BitShiftReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | shift | [int32](#int32) | | Parameter bit shift. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ByteOrderReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | byte_order | [system_monitor_common.ByteOrder](#system_monitor_common-ByteOrder) | | Byte order. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### CANParameterProperties | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | Id | [string](#string) | | The unique identifier of the CAN Parameter | | name | [string](#string) | | The display name of the parameter | | description | [string](#string) | | The description of the parameter | | lower_display_limit | [double](#double) | | The minimum display value | | upper_display_limit | [double](#double) | | The maximum display value | | min_logging_rate | [uint32](#uint32) | | The minimum logging rate | | scaling_factor | [uint32](#uint32) | | The parameter scaling factor (see see EScalingUnit). | | min_not_defined | [bool](#bool) | | State of the min/max values. | | conversion_id | [string](#string) | | Unique identifier of the conversion rule used in this parameter. | | rx | [bool](#bool) | | Flag to indicate Rx / Tx status of the parameter | | data_type | [system_monitor_common.DataType](#system_monitor_common-DataType) | | Parameter data type | | can_bus | [string](#string) | | CAN Bus name. | | can_message | [string](#string) | | CAN Message name. | | can_start_bit | [uint32](#uint32) | | CAN parameter start bit. | | can_bit_length | [uint32](#uint32) | | CAN parameter bit length. | | can_gain | [double](#double) | | CAN parameter gain value. | | can_offset | [double](#double) | | CAN parameter offset value. | | can_mux_id | [string](#string) | | CAN parameter multiplexer id. Blank if parameter is not in a multiplexed message. | | can_byte_order | [system_monitor_common.ByteOrder](#system_monitor_common-ByteOrder) | | CAN parameter byte order. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### CANParameterPropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [CANParameterProperties](#system_monitor_parameter-CANParameterProperties) | repeated | The CAN parameter properties. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Conversion | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | Id of the conversion. | | type | [system_monitor_common.ConversionType](#system_monitor_common-ConversionType) | | Type of the conversion. - One of ConversionType.Rational, ConversionType.Table, ConversionType.Text or ConversionType.Formula. | ### ConversionListReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversions | [Conversion](#system_monitor_parameter-Conversion) | repeated | The conversions. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ConversionNoAppRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Id of the conversion. | ### ConversionTypeReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Id of the conversion. | | type | [system_monitor_common.ConversionType](#system_monitor_common-ConversionType) | | | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ExternalParameterRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | ### ExternalReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | gain | [double](#double) | | Gain. | | offset | [double](#double) | | Offset. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ExternalRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | gain | [double](#double) | | Gain. | | offset | [double](#double) | | Offset. | ### FormulaConversionReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | formula | [string](#string) | | Formula value. | | inverse | [string](#string) | | Inverse value. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### FormulaConversionRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | formula | [string](#string) | | Formula value. | | inverse | [string](#string) | | Inverse value. | | overwrite | [bool](#bool) | | Specifies to overwrite existing values. | ### LoggableReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | loggable | [bool](#bool) | | Whether the parameter is loggable. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### MapPropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | x_axis_id | [string](#string) | | Identifier of the x axis parameter | | y_axis_id | [string](#string) | | Identifier of the y axis parameter | | x_points | [uint32](#uint32) | | Number on breakpoints in the x axis. | | y_points | [uint32](#uint32) | | Number on breakpoints in the y axis. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### OffsetReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | offset | [double](#double) | | Parameter offset. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### OffsetRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | offset | [double](#double) | | Parameter offset. | ### Parameter | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | Id of the parameter. | | name | [string](#string) | | Name of the parameter. | ### ParameterErrorsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [ParameterValue](#system_monitor_parameter-ParameterValue) | repeated | Parameter values | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParameterGroup | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | Id of the parameter group. | | group | [string](#string) | | Name of the parameter group. | ### ParameterGroupsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [ParameterGroup](#system_monitor_parameter-ParameterGroup) | repeated | The parameter groups. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParameterListReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [Parameter](#system_monitor_parameter-Parameter) | repeated | The parameters. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParameterProperties | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | Id | [string](#string) | | Id of the parameter | | name | [string](#string) | | Name of the parameter. | | description | [string](#string) | | Description of the parameter. | | type | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | | Parameter data type. - One of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Axis or ParameterType.Measurement. | | units | [string](#string) | | Units of the parameter. | | format | [string](#string) | | Format used to display the parameter's values. | | conversion_id | [string](#string) | | Id of the conversion rule for the parameter. | | groups | [string](#string) | repeated | Names of the groups the parameter belongs to. | | data_type | [system_monitor_common.DataType](#system_monitor_common-DataType) | | Data type of the parameter. | | data_size | [uint32](#uint32) | | Size of the parameter data. | | lower_engineering_limit | [double](#double) | | Lower engineering limit. | | upper_engineering_limit | [double](#double) | | Upper engineering limit. | | max_logging_rate | [uint32](#uint32) | | Maximum logging rate. | | prime | [bool](#bool) | | Whether the parameter is marked as a Prime parameter. | | read_only | [bool](#bool) | | Whether the parameter is readonly. | | tuneable | [bool](#bool) | | Whether the parameter is tuneable. | | multiplexed_ids | [string](#string) | repeated | Array of parameter multiplexer ids. Blank if parameter is not in a multiplexed message | ### ParameterPropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [ParameterProperties](#system_monitor_parameter-ParameterProperties) | repeated | The parameter properties. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParameterSetValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | value | [double](#double) | | Value of the parameter. | ### ParameterTypeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application | | parameter_id | [string](#string) | | Id of the parameter. | | data_type | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | | Parameter data type. - One of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | ### ParameterValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | value | [double](#double) | | Value of the parameter. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParametersReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_ids | [string](#string) | repeated | Ids of the parameters. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### RationalConversionReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | coefficient1 | [double](#double) | | Coefficient value 1. | | coefficient2 | [double](#double) | | Coefficient value 2. | | coefficient3 | [double](#double) | | Coefficient value 3. | | coefficient4 | [double](#double) | | Coefficient value 4. | | coefficient5 | [double](#double) | | Coefficient value 5. | | coefficient6 | [double](#double) | | Coefficient value 6. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### RationalConversionRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | coefficient1 | [double](#double) | | Coefficient value 1. | | coefficient2 | [double](#double) | | Coefficient value 2. | | coefficient3 | [double](#double) | | Coefficient value 3. | | coefficient4 | [double](#double) | | Coefficient value 4. | | coefficient5 | [double](#double) | | Coefficient value 5. | | coefficient6 | [double](#double) | | Coefficient value 6. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | overwrite | [bool](#bool) | | Specifies to overwrite existing values. | ### RowDetailsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | row_id | [uint32](#uint32) | | Row identifier | | ident_offset | [int32](#int32) | | Parameter ident offset | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### RowValues | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [double](#double) | repeated | Values for the row. | ### StringParameterErrorsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [StringParameterValue](#system_monitor_parameter-StringParameterValue) | repeated | Parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### StringParameterSetValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | value | [string](#string) | | Value of the parameter. | ### StringParameterValue | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Id of the parameter. | | value | [string](#string) | | Value of the parameter. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### StringValueReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [StringParameterValue](#system_monitor_parameter-StringParameterValue) | repeated | String parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### TableConversion | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | raw | [double](#double) | | Raw numeric value. | | mapped | [double](#double) | | Mapped numeric value. | ### TableConversionReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | interpolate | [bool](#bool) | | Specifies interpolation on or off. | | values | [TableConversion](#system_monitor_parameter-TableConversion) | repeated | Table conversion values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### TableConversionRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | comment | [string](#string) | | Human readable description. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | interpolate | [bool](#bool) | | Specifies interpolation on or off. | | values | [TableConversion](#system_monitor_parameter-TableConversion) | repeated | | | overwrite | [bool](#bool) | | Specifies to overwrite existing values. | ### TextConversion | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | raw | [double](#double) | | Raw numeric value. | | mapped | [string](#string) | | Mapped text value. | ### TextConversionReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | values | [TextConversion](#system_monitor_parameter-TextConversion) | repeated | Text conversion values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### TextConversionRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_id | [string](#string) | | Unique identifier of conversion. | | format | [string](#string) | | Format string (e.g. %1.3f). | | units | [string](#string) | | Unit value for output. | | default | [string](#string) | | Default value. | | values | [TextConversion](#system_monitor_parameter-TextConversion) | repeated | Text conversion values. | | overwrite | [bool](#bool) | | Specifies to overwrite existing values. | ### UndoRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | buffer_type | [system_monitor_common.BufferType](#system_monitor_common-BufferType) | | Buffer type. | ### ValueReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [ParameterValue](#system_monitor_parameter-ParameterValue) | repeated | Numeric parameter values. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### WarningLimitsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | low | [double](#double) | | Lower limit. | | high | [double](#double) | | Upper limit. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### WarningLimitsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Id of the application. | | parameter_id | [string](#string) | | Id of the parameter. | | low | [double](#double) | | Lower limit. | | high | [double](#double) | | Upper limit. | ### SystemMonitorParameter | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | GetParameters | [AppTypeRequest](#system_monitor_parameter-AppTypeRequest) | [ParameterListReply](#system_monitor_parameter-ParameterListReply) | Gets parameters. AppId not required for CAN and Virtual parameters | | GetConversions | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [ConversionListReply](#system_monitor_parameter-ConversionListReply) | Gets parameter conversion rules. | | GetParameterAndGroups | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [ParameterGroupsReply](#system_monitor_parameter-ParameterGroupsReply) | Gets parameters and groups | | GetParameterProperties | [AppTypeRequest](#system_monitor_parameter-AppTypeRequest) | [ParameterPropertiesReply](#system_monitor_parameter-ParameterPropertiesReply) | Gets parameter properties | | GetCANParameterProperties | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [CANParameterPropertiesReply](#system_monitor_parameter-CANParameterPropertiesReply) | Gets CAN Parameter properties. | | GetMapProperties | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [MapPropertiesReply](#system_monitor_parameter-MapPropertiesReply) | Gets map properties. | | GetRowDetails | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [RowDetailsReply](#system_monitor_parameter-RowDetailsReply) | Gets row details. | | GetParameterBitMask | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [BitMaskReply](#system_monitor_parameter-BitMaskReply) | Gets a parameter bit mask. | | GetParameterBitShift | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [BitShiftReply](#system_monitor_parameter-BitShiftReply) | Gets a parameter bit shift. | | GetParameterAddress | [ParameterTypeRequest](#system_monitor_parameter-ParameterTypeRequest) | [AddressReply](#system_monitor_parameter-AddressReply) | Gets a parameter address. | | GetParameterByteOrder | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [ByteOrderReply](#system_monitor_parameter-ByteOrderReply) | Gets a parameter byte order value. | | ParameterLoggable | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [LoggableReply](#system_monitor_parameter-LoggableReply) | Gets whether a parameter is loggable. | | GetExternalInputGainOffset | [ExternalParameterRequest](#system_monitor_parameter-ExternalParameterRequest) | [ExternalReply](#system_monitor_parameter-ExternalReply) | Gets an input gain/offset for an external parameter. | | SetExternalInputGainOffset | [ExternalRequest](#system_monitor_parameter-ExternalRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets an input gain/offset for an external parameter. | | GetModifiedParameters | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [ParameterListReply](#system_monitor_parameter-ParameterListReply) | Gets the modified parameters in a selected application | | GetParameterWarningLimits | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [WarningLimitsReply](#system_monitor_parameter-WarningLimitsReply) | Gets the warning limits from a parameter. | | SetParameterWarningLimits | [WarningLimitsRequest](#system_monitor_parameter-WarningLimitsRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the warning limits for a parameter. | | DeleteMinMax | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Deletes min/max values. | | ExportInputSignals | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports input signals to a file. | | ImportInputSignals | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports input signals from a file. | | RegenerateInputSignalParameters | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Regenerate Input Signal parameters from RAW values. | | UndoDataChanges | [UndoRequest](#system_monitor_parameter-UndoRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Reset all tune buffers on the unit and/or in the edit buffer. | | RestoreValue | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Restore a parameter value its to original saved value. | | GetAxisParameterFromMap | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [AxisParametersReply](#system_monitor_parameter-AxisParametersReply) | Retrieves a list of axis parameter(s) used by the map parameter in the application. | | GetConversionUse | [.system_monitor_common.ConversionRequest](#system_monitor_common-ConversionRequest) | [ParametersReply](#system_monitor_parameter-ParametersReply) | Returns a list of all parameters using the specified conversion in the application. | | GetConversionType | [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) | [ConversionTypeReply](#system_monitor_parameter-ConversionTypeReply) | Returns the type of the specified conversion rule. | | GetRationalConversion | [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) | [RationalConversionReply](#system_monitor_parameter-RationalConversionReply) | Retrieves a selected Rational Conversion | | GetTableConversion | [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) | [TableConversionReply](#system_monitor_parameter-TableConversionReply) | Retrieve a specified Table Conversion rule. | | GetTextConversion | [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) | [TextConversionReply](#system_monitor_parameter-TextConversionReply) | Retrieves enumeration values and text from conversion parameters. | | GetFormulaConversion | [ConversionNoAppRequest](#system_monitor_parameter-ConversionNoAppRequest) | [FormulaConversionReply](#system_monitor_parameter-FormulaConversionReply) | Retrieves a selected Formula (Free) Conversion | | GetAppRationalConversion | [.system_monitor_common.ConversionRequest](#system_monitor_common-ConversionRequest) | [RationalConversionReply](#system_monitor_parameter-RationalConversionReply) | Retrieves the details from a Rational Conversion Rule. | | GetAppTableConversion | [.system_monitor_common.ConversionRequest](#system_monitor_common-ConversionRequest) | [TableConversionReply](#system_monitor_parameter-TableConversionReply) | Retrieve the details from a Table Conversion rule. | | SetRationalConversion | [RationalConversionRequest](#system_monitor_parameter-RationalConversionRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create or modify a Rational Conversion rule. | | SetTableConversion | [TableConversionRequest](#system_monitor_parameter-TableConversionRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create or modify a Table Conversion rule. | | SetTextConversion | [TextConversionRequest](#system_monitor_parameter-TextConversionRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create or modify a text conversion rule. | | SetFormulaConversion | [FormulaConversionRequest](#system_monitor_parameter-FormulaConversionRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create or modify a Formula (Free) Conversion rule. | | GetValueOffset | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [OffsetReply](#system_monitor_parameter-OffsetReply) | Retrieve the live value offset for a parameter. | | SetValueOffset | [OffsetRequest](#system_monitor_parameter-OffsetRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the live value offset for a parameter. | | ZeroLiveValue | [.system_monitor_common.ParameterRequest](#system_monitor_common-ParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Zero's the live value by applying an offset to a measurement parameter. | | GetValueMeasurement | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [ValueReply](#system_monitor_parameter-ValueReply) | Returns the current value of a specified measurement parameter. | | GetValueScalar | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [ValueReply](#system_monitor_parameter-ValueReply) | Returns the current value of a specified scalar (VALUE) editable parameter. | | GetValue1AxisMap | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the current values of a specified 1-axis map (CURVE) editable parameter at all axis points. | | GetValue2AxisMap | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [Array2dValueReply](#system_monitor_parameter-Array2dValueReply) | Returns the current values of a specified 2-axis map (MAP) editable parameter at all axis points. | | GetValueAxis | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the current values of a specified axis (AXIS_PTS) editable parameter at all axis points. | | GetValueArray | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the current values of a specified array (VAL_BLK) editable (characteristic) parameter at all index positions. | | GetValueString | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [StringValueReply](#system_monitor_parameter-StringValueReply) | Returns the current value of a specified string editable parameter | | GetValueCAN | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [ValueReply](#system_monitor_parameter-ValueReply) | Retrieves the value of the specified can parameter. | | GetValueVirtual | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [ValueReply](#system_monitor_parameter-ValueReply) | Retrieve a selected Virtual Parameter. | | GetDTVValueScalar | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [ValueReply](#system_monitor_parameter-ValueReply) | Returns the value of a specified scalar (VALUE) editable parameter from the supplied DTV. | | GetDTVValue1AxisMap | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the values of a specified 1-axis map (CURVE) editable parameter at all axis points from the supplied DTV. | | GetDTVValue2AxisMap | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [Array2dValueReply](#system_monitor_parameter-Array2dValueReply) | Returns the values of a specified 2-axis map (MAP) editable parameter at all axis points from the supplied DTV. | | GetDTVValueAxis | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the values of a specified axis (AXIS_PTS) editable parameter at all axis points from the supplied DTV. | | GetDTVValueArray | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [Array1dValueReply](#system_monitor_parameter-Array1dValueReply) | Returns the values of a specified array (VAL_BLK) editable (characteristic) parameter at all index positions from the supplied DTV. | | GetDTVValueString | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [StringValueReply](#system_monitor_parameter-StringValueReply) | Returns the value of a specified string editable parameter from the supplied DTV. | | SetValueScalar | [AppParameterValuesRequest](#system_monitor_parameter-AppParameterValuesRequest) | [ParameterErrorsReply](#system_monitor_parameter-ParameterErrorsReply) | Sets the current value of a specified scalar (VALUE) editable parameter. | | SetValue1AxisMap | [AppArray1dParameterValuesRequest](#system_monitor_parameter-AppArray1dParameterValuesRequest) | [Array1dParameterErrorsReply](#system_monitor_parameter-Array1dParameterErrorsReply) | Sets the current value of a specified 1-axis map (CURVE) editable parameter at one or more axis points. | | SetValue2AxisMap | [AppArray2dParameterValuesRequest](#system_monitor_parameter-AppArray2dParameterValuesRequest) | [Array2dParameterErrorsReply](#system_monitor_parameter-Array2dParameterErrorsReply) | Sets the current value of a specified 2-axis map (MAP) editable parameter at one or more axis points. | | SetValueAxis | [AppArray1dParameterValuesRequest](#system_monitor_parameter-AppArray1dParameterValuesRequest) | [Array1dParameterErrorsReply](#system_monitor_parameter-Array1dParameterErrorsReply) | Sets the current value of a specified axis (AXIS_PTS) editable parameter at one or more axis points. | | SetValueArray | [AppArray1dParameterValuesRequest](#system_monitor_parameter-AppArray1dParameterValuesRequest) | [Array1dParameterErrorsReply](#system_monitor_parameter-Array1dParameterErrorsReply) | Sets the current value of a specified array (VAL_BLK) parameter at one or more index positions. | | SetValueString | [AppStringParameterValuesRequest](#system_monitor_parameter-AppStringParameterValuesRequest) | [StringParameterErrorsReply](#system_monitor_parameter-StringParameterErrorsReply) | Sets the current value of a specified string editable parameter. |

Top

## Protos/system_monitor_project.proto ### ActiveAppReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_ids | [uint32](#uint32) | repeated | Application ids | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### AppFileRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | | file_path | [string](#string) | | File path. | ### AppReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | text | [string](#string) | | Return text value. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Application | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | app_name | [string](#string) | | Application name. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### CANMergeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | Bus Index. | | file_path | [string](#string) | | Name and path of .csv. | | merge | [bool](#bool) | | Merge with existing messages. | ### CANRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | Bus Index | | file_path | [string](#string) | | Name and path of CAN .csv file. | ### CompareAppReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [CompareParameter](#system_monitor_project-CompareParameter) | repeated | The compared parameter information. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### CompareAppRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | | dtv1_path | [string](#string) | | Filename and full path of the first DTV to compare against (.dtv). | | dtv2_path | [string](#string) | | Filename and full path of the second DTV to compare against (.dtv). | ### CompareParameter | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Parameter id. | | type | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | | Parameter data type. - One of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | | reason1 | [ReasonCode](#system_monitor_project-ReasonCode) | | Reason codes for difference between the parameter in the application in memory and the first DTV supplied. | | reason2 | [ReasonCode](#system_monitor_project-ReasonCode) | | Reason codes for difference between the parameter in the application in memory and the second DTV supplied. | ### DTVModifiedReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | modified | [bool](#bool) | | Whether the DTV is modified. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### DTVSaveCopyRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | save_path | [string](#string) | | Filename and full path. | | comment | [string](#string) | | Prefix to existing comment. | | notes | [string](#string) | | Prefix to existing notes. | | consortium | [string](#string) | | Consortium name to save data version to. Consortium Id in the DTV will be saved with the supplied value if licence permissions allow, otherwise error code -2 is returned. Set to "Disabled" to remove team security and unlock file. | ### DTVSaveIncrementRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | comment | [string](#string) | | Prefix to existing comment. | | notes | [string](#string) | | Prefix to existing notes. | ### DTVSaveRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | save_path | [string](#string) | | Filename and full path. | | comment | [string](#string) | | Prefix to existing comment. | | notes | [string](#string) | | Prefix to existing notes. | ### DTVSavedOnReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | saved_on | [string](#string) | | Timestamp the DTV was saved at. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### DetailsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | text | [string](#string) | | Data Version Notes or Comments | ### EnableRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | enable | [bool](#bool) | | True to enable, false to disable. | ### ErrorDefinition | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | Error definition id. | | name | [string](#string) | | Error definition name. | | description | [string](#string) | | Error definition description. | | group | [string](#string) | | Error definition group. | | bit_number | [uint32](#uint32) | | Error definition bit number. | | current | [string](#string) | | Error definition current parameter identifier. | | logged | [string](#string) | | Error definition logged parameter identifier. | ### ErrorDefinitionsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | error_definitions | [ErrorDefinition](#system_monitor_project-ErrorDefinition) | repeated | The error definitions. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ErrorInstance | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Error identifier. | | description | [string](#string) | | Error description. | | status | [system_monitor_common.ErrorStatus](#system_monitor_common-ErrorStatus) | | Error status level. | ### ErrorReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | error_instances | [ErrorInstance](#system_monitor_project-ErrorInstance) | repeated | The active errors | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### Event | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [uint32](#uint32) | | Event id. | | name | [string](#string) | | Event name. | | priority | [system_monitor_common.EventPriority](#system_monitor_common-EventPriority) | | Event priority. | ### EventReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | event_id | [uint32](#uint32) | | Event Id. | | description | [string](#string) | | Event description. | | conversion_id1 | [string](#string) | | Id of first event conversion. | | conversion_id2 | [string](#string) | | Id of second event conversion. | | conversion_id3 | [string](#string) | | Id of third event conversion. | | priority | [system_monitor_common.EventPriority](#system_monitor_common-EventPriority) | | Event level. ( 0 - High, 1 - Medium, 2 - Low, 3 - Debug ) | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### EventRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application id. | | event_id | [uint32](#uint32) | | Event id. | ### EventsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | events | [Event](#system_monitor_project-Event) | repeated | The events. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ExistsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | exists | [bool](#bool) | | True if exists, False if not | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ExistsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | parameter_id | [string](#string) | | Parameter Id. | | data_type | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | | Parameter data type. - One of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### FileDetailsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | saved_by | [string](#string) | | Name of the last user to save the file. | | saved_on | [google.protobuf.Timestamp](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Timestamp.html) | | Date time of last file save. | | comment | [string](#string) | | File comments. | | notes | [string](#string) | | File notes. | | build | [uint32](#uint32) | | Build number of System Monitor used when saving file. | | consortium | [string](#string) | | Name of the licence consortium used to save the file. Set for supported file types only ( .dtv, .rlc & .dtp ). | | owner | [string](#string) | | Name of the licence owner used to create the file. Set for supported file types only ( .dtv, .rlc & .dtp ). | | rda | [string](#string) | | RDA unlock list checksum for locked DTV's.Set for supported file types only ( .dtv ). | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### FileNameRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_type | [system_monitor_common.FileType](#system_monitor_common-FileType) | | File name of requested type. - One of FileType.Project, FileType.Desktop, FileType.Virtuals, FileType.LoggingConfig or FileType.Can. | | slot | [uint32](#uint32) | | Slot index (1-8). Only required for FileType.LoggingConfig and FileType.Can | ### FileNewRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_type | [system_monitor_common.FileType](#system_monitor_common-FileType) | | Type of file to create. - One of FileType.Virtuals, FileType.LoggingConfig or FileType.Can. | | file_path | [string](#string) | | The path of the file to create. | | save_existing | [bool](#bool) | | Whether to save the existing file. | | overwrite | [bool](#bool) | | Whether to overwrite an existing file with the same path. | ### FileOpenRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_type | [system_monitor_common.FileType](#system_monitor_common-FileType) | | The type of file to open. - One of FileType.Desktop, FileType.Virtuals, FileType.LoggingConfig or FileType.Can. | | file_path | [string](#string) | | The path of the file to open. | | slot | [uint32](#uint32) | | Slot index. Only required when loading Logging and CAN configuration (1-8). | | activate | [bool](#bool) | | True to set active. Only required when loading Logging and CAN configuration (1-8). | ### FileReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_path | [string](#string) | | File path including folder. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### FileSaveRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_type | [system_monitor_common.FileType](#system_monitor_common-FileType) | | Type of file to save. - One of FileType.Desktop, FileType.Virtuals, FileType.LoggingConfig or FileType.Can. | | file_path | [string](#string) | | The path of the file to save. | | comment | [string](#string) | | File comments. | | notes | [string](#string) | | File notes. | | consortium | [string](#string) | | Name of the licence consortium used to save the file. Set for supported file types only ( .dtv, .rlc & .dtp ). | | save_copy_as | [bool](#bool) | | Whether to save a copy without affecting SM modified status. | ### GetAppDetailsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | apps | [Application](#system_monitor_project-Application) | repeated | Application details. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### GetBuildNumberReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | build_number | [uint32](#uint32) | | System Monitor build number. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### GetVersionNumberReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | major_version | [uint32](#uint32) | | Major revision. | | minor_version | [uint32](#uint32) | | Minor revision. | | build_version | [uint32](#uint32) | | Build number. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### MatlabDTVRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | dtv_path | [string](#string) | | Filename and full path of DTV file. | | export_path | [string](#string) | | Filename and full path of exported file. | | data_only | [bool](#bool) | | True if only data is to be exported, false to export data and definitions. | | data_types | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | repeated | Parameter data types included in export. - One or more of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | ### MatlabRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | export_path | [string](#string) | | Filename and full path. | | data_only | [bool](#bool) | | True if only data is to be exported, false to export data and definitions. | | data_types | [system_monitor_common.ParameterType](#system_monitor_common-ParameterType) | repeated | Parameter data types included in export. - One or more of ParameterType.scalar, ParameterType.axis_1, ParameterType.axis_2, ParameterType.array, ParameterType.string, ParameterType.Can, ParameterType.Virtual, ParameterType.Axis, ParameterType.Input or ParameterType.Measurement. | ### MatlabSelectedRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | export_path | [string](#string) | | Filename and full path. | | data_only | [bool](#bool) | | True if only data is to be exported, false to export data and definitions. | | parameter_ids | [string](#string) | repeated | Identifiers of the parameters to be exported | ### MultiAppReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_ids | [uint32](#uint32) | repeated | Applications containing the specified parameter. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | | ### MultiAppRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_ids | [uint32](#uint32) | repeated | Application identifiers. | ### PGVIDReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | pgv_id | [uint32](#uint32) | | Result program version ID. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### ParameterIdRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_id | [string](#string) | | Parameter identifier. | ### ProjectCloseRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | action | [int32](#int32) | | Action to perform when unsaved documents encountered on closing project. 0 - Cancel Save. 1 - Save All. 2 - Save None. | ### ProjectCreateRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | project_path | [string](#string) | | Project name to be created within the current multi-application base. | | app_paths | [string](#string) | repeated | Array of PGV files to be included. | | desktop_path | [string](#string) | | Name of desktop document to be created or included if file already exists. | | virtuals_path | [string](#string) | | Optional name of virtual parameters document to be created or included if file already exists. | | can_path | [string](#string) | | Optional name of CAN configuration document to be created or included if file already exists. | | logging_config_path | [string](#string) | | Optional name of remote logging configuration document to be created or included if file already exists. | ### ProjectExportRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | save_modified | [bool](#bool) | | Flag to indicate if any unsaved files in the project should be saved before exporting. | ### ProjectImportRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | project_path | [string](#string) | | SMX file containing project to be imported. | | base | [string](#string) | | Optional Multi-Application base name for import. If blank, the current default base is used. | ### ProjectSaveAsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | project_name | [string](#string) | | Name of renamed project file. | | save_all | [bool](#bool) | | If TRUE then all modified files are saved. If FALSE then only the project file is saved. | | comments | [string](#string) | | New project comments. | | notes | [string](#string) | | New project notes. | ### ProjectSaveRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | save_all | [bool](#bool) | | If TRUE then all modified files are saved. If FALSE then only the project file is saved. | ### ReasonCode | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | reasons | [system_monitor_common.Reason](#system_monitor_common-Reason) | repeated | Reason codes. | ### ReprogramRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_ids | [uint32](#uint32) | repeated | An array of all app ids to reprogram. | | force | [bool](#bool) | | Whether to force a reprogram. | ### SensorRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. | | sensor | [string](#string) | | Sensor identifier. | | serial_number | [int32](#int32) | | Serial number. | ### SlotActiveRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | slot | [uint32](#uint32) | | Slot index (1-8). | | active | [bool](#bool) | | TRUE to activate the slot. | ### SlotReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | active | [bool](#bool) | | Slot is active. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | return code. | ### SlotRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | slot | [uint32](#uint32) | | Slot index (1-8). | ### SyncedReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | synced | [bool](#bool) | | TRUE if the edit buffer is in sync with the ECU. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### SystemMonitorProject | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | ProjectOpen | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Opens a Project. | | ProjectClose | [ProjectCloseRequest](#system_monitor_project-ProjectCloseRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Closes a Project. | | ProjectCreate | [ProjectCreateRequest](#system_monitor_project-ProjectCreateRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create a new project. If full path names are used, the path must lie within the current multi-application base. | | ProjectSave | [ProjectSaveRequest](#system_monitor_project-ProjectSaveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Saves the current project. | | ProjectSaveAs | [ProjectSaveAsRequest](#system_monitor_project-ProjectSaveAsRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Saves the current project as a new project. | | ProjectImport | [ProjectImportRequest](#system_monitor_project-ProjectImportRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports the project from a .smx file. | | ProjectExport | [ProjectExportRequest](#system_monitor_project-ProjectExportRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the project to a .smx file. | | Reprogram | [ReprogramRequest](#system_monitor_project-ReprogramRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Reprograms the ECU. | | DownloadDataChanges | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Causes any modifications made to the System Monitor edit buffer to be downloaded to the ECU. | | EditBufferSynced | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [SyncedReply](#system_monitor_project-SyncedReply) | Identifies whether the edit buffer is in sync with the ECU. | | UploadDataVersion | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Uploads specified data version to the edit buffer. | | GetVersionNumber | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [GetVersionNumberReply](#system_monitor_project-GetVersionNumberReply) | Gets the version information contained within the System Monitor executable. | | GetPGVVersion | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Gets the filepath for the .pgv of an application by Application Id. | | GetPGVID | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [PGVIDReply](#system_monitor_project-PGVIDReply) | Gets the program version ID of the currently loaded program version. | | GetDTVVersion | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Returns the filename of the data version in the edit buffer. | | GetEcuDTVVersion | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Returns the filename of the data version on the ECU. | | GetNextDTVVersion | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Returns the filename of the next data version that will be saved if the user selects, File, Save. If the edit buffer is unmodified, this will be the name of the current data version in the edit buffer. | | GetDTVModified | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [DTVModifiedReply](#system_monitor_project-DTVModifiedReply) | Gets the modified state associated with the DTV. | | GetDTVSavedOn | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [DTVSavedOnReply](#system_monitor_project-DTVSavedOnReply) | Gets the contents of the saved on attribute associated with the DTV. | | GetDTVNotes | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Returns the contents of the notes attribute associated with the DTV. | | SetDTVNotes | [DetailsRequest](#system_monitor_project-DetailsRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the Data Version notes for the specified application. | | ClearDTVNotes | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Clears the contents of the notes attribute associated with the DTV. | | GetDTVComment | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [AppReply](#system_monitor_project-AppReply) | Gets the contents of the comments attribute associated with the DTV. | | SetDTVComment | [DetailsRequest](#system_monitor_project-DetailsRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the Data Version comment for the specified application. | | EnableDTVBackup | [EnableRequest](#system_monitor_project-EnableRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Enables/Disables the data version backup during parameter value changes. Adds a message to the log file to indicate when changed. NOTE: Any changes to the data version whilst backups are suspended are not flushed to the backup file when the backups are re- enabled. | | DTVOpen | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Opens the specified data version file. | | DTVSave | [DTVSaveRequest](#system_monitor_project-DTVSaveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Saves the current data version to the specified file. | | DTVSaveCopy | [DTVSaveCopyRequest](#system_monitor_project-DTVSaveCopyRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Saves the current data version to the specified file, without modifying the data version in the edit buffer. | | DTVSaveIncrement | [DTVSaveIncrementRequest](#system_monitor_project-DTVSaveIncrementRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Increments the version number of the data version and then performs a save. | | GetBuildNumber | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [GetBuildNumberReply](#system_monitor_project-GetBuildNumberReply) | Gets the build number of System Monitor (SM-V7). Client code may be branched on the basis of build number, thereby providing a way by which a single client script can be written which works with releases of SM-V7 which have different ActiveX interfaces. | | GetAppDetails | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [GetAppDetailsReply](#system_monitor_project-GetAppDetailsReply) | Gets details of the applications in the current project. | | GetActiveApps | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [ActiveAppReply](#system_monitor_project-ActiveAppReply) | Gets details of all the active applications. | | SetActiveApps | [MultiAppRequest](#system_monitor_project-MultiAppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the active application when provided with an application ID. | | AddApp | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Adds the application defined by the given DTV and it's associated PGV into the project. If the application already exists in the project the error: SMAPI_ERR_INVALID_CMD (-31) will be returned. | | RemoveApp | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes the given application from the project. The requested application cannot be the BIOS. | | CompareApp | [CompareAppRequest](#system_monitor_project-CompareAppRequest) | [CompareAppReply](#system_monitor_project-CompareAppReply) | Perform a compare between an application loaded in memory and up to two DTV's. When two DTV's are used both must have the same PGV version. Returns a list of parameters that contain differences along with the parameter data type and a reason code identifying the difference. | | GetAppPULFile | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [FileReply](#system_monitor_project-FileReply) | Get the name of the unlock list PUL file for the given application id. | | SetAppPULFile | [AppFileRequest](#system_monitor_project-AppFileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Set the unlock list PUL file on the given application id. Licence ownership restrictions apply | | GenerateParamSet | [.system_monitor_common.ParametersFileRequest](#system_monitor_common-ParametersFileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Generate a parameter set file. | | GeneratePULFile | [.system_monitor_common.AppParametersFileRequest](#system_monitor_common-AppParametersFileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Generate a PUL file for an application for use with Reduced Data Access. NOTE: On creation of PUL file the name will be adjusted from the requested name by including the checksum. This is required to reduce file lookup times. | | GeneratePULFileFromParamSet | [AppFileRequest](#system_monitor_project-AppFileRequest) | [FileReply](#system_monitor_project-FileReply) | Generate a PUL file for an application for use with Reduced Data Access from a saved Parameter Set file. NOTE: On creation of PUL file the name will be adjusted from the requested name by including the checksum. This is required to reduce file lookup times. The adjusted file name will be returned in pszFileName. | | ChangeSensorSerialNumber | [SensorRequest](#system_monitor_project-SensorRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Changes the serial number of the specified sensor. | | FileOpen | [FileOpenRequest](#system_monitor_project-FileOpenRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Opens a file within System Monitor. Supports FileType.Desktop, FileType.LoggingConfig, FileType.Virtuals, FileType.Can. | | FileSave | [FileSaveRequest](#system_monitor_project-FileSaveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Saves a file within System Monitor. Supports FileType.Desktop, FileType.LoggingConfig, FileType.Virtuals, FileType.Can. | | FileNew | [FileNewRequest](#system_monitor_project-FileNewRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Creates a file within System Monitor. Supports FileType.LoggingConfig, FileType.Virtuals, FileType.Can. | | GetFileName | [FileNameRequest](#system_monitor_project-FileNameRequest) | [FileReply](#system_monitor_project-FileReply) | Gets the name of a file from System Monitor. Supports FileType.Project, FileType.Desktop, FileType.LoggingConfig, FileType.Virtuals, FileType.Can | | GetFileDetails | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [FileDetailsReply](#system_monitor_project-FileDetailsReply) | Gets the file details for any file type supported by System Monitor. | | CreateFFCFromPGV | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Create an FFC File from the given PGV. Enable FFC advanced option must be set to 'Y' and full path name of PGV is required. FFC file will be created in same location as original PGV. | | ExportToHexFile | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the selected applications PGV and DTV to a hexfile. | | GetActiveCANConfig | [SlotRequest](#system_monitor_project-SlotRequest) | [SlotReply](#system_monitor_project-SlotReply) | Gets the active status of the CAN configuration in the given slot. | | SetActiveCANConfig | [SlotActiveRequest](#system_monitor_project-SlotActiveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Set the active status of the CAN configuration in the given slot. | | GetFIACANConfig | [SlotRequest](#system_monitor_project-SlotRequest) | [SlotReply](#system_monitor_project-SlotReply) | Gets the FIA status of the CAN configuration into the given slot. | | SetFIACANConfig | [SlotActiveRequest](#system_monitor_project-SlotActiveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Set the FIA status of the CAN configuration into the given slot. | | CANBuffersExport | [CANRequest](#system_monitor_project-CANRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the CAN Buffer configuration csv from the given CAN Bus index. | | CANBuffersImport | [CANRequest](#system_monitor_project-CANRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports the CAN Buffer configuration csv into the given CAN Bus index. | | CANMessagesExport | [CANRequest](#system_monitor_project-CANRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the CAN Message configuration csv from the given CAN Bus index. | | CANMessagesImport | [CANMergeRequest](#system_monitor_project-CANMergeRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports the CAN Message configuration csv into the given CAN Bus index. | | CANConfigUnload | [SlotRequest](#system_monitor_project-SlotRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Unloads the CAN configuration from the given slot. (Slot index should be 2-8). | | GetActiveLoggingConfig | [SlotRequest](#system_monitor_project-SlotRequest) | [SlotReply](#system_monitor_project-SlotReply) | Get the active status of the remote logging configuration in the given slot. | | SetActiveLoggingConfig | [SlotActiveRequest](#system_monitor_project-SlotActiveRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Set the active status of the remote logging configuration in the given slot. | | LoggingConfigUnload | [SlotRequest](#system_monitor_project-SlotRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Unloads the remote logging configuration from the given slot. | | MatlabImport | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports the specified MatLab-M file into System Monitor. | | MatlabExport | [MatlabRequest](#system_monitor_project-MatlabRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the currently loaded data version to the specified file in MatLab-M format. | | MatlabExportDTV | [MatlabDTVRequest](#system_monitor_project-MatlabDTVRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the specified data version to the specified file in MatLab-M format. | | MatlabExportSelected | [MatlabSelectedRequest](#system_monitor_project-MatlabSelectedRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports the specified list of editable parameters in Matlab format. | | AddParametersToUnlockList | [.system_monitor_common.AppParametersFileRequest](#system_monitor_common-AppParametersFileRequest) | [FileReply](#system_monitor_project-FileReply) | Update a PUL file for an application for use with Reduced Data Access by adding the requested parameter identifiers. NOTE: The PUL file name will be adjusted by including the new checksum and returned. | | RemoveParametersFromUnlockList | [.system_monitor_common.AppParametersFileRequest](#system_monitor_common-AppParametersFileRequest) | [FileReply](#system_monitor_project-FileReply) | Update a PUL file for an application for use with Reduced Data Access by removing the requested parameter identifiers. NOTE: The PUL file the name will be adjusted by including the new checksum and returned. | | GetAppsHoldingParam | [ParameterIdRequest](#system_monitor_project-ParameterIdRequest) | [MultiAppReply](#system_monitor_project-MultiAppReply) | Gets the applications which contain the specified parameter. | | GetAppsHoldingMeasurementParam | [ParameterIdRequest](#system_monitor_project-ParameterIdRequest) | [MultiAppReply](#system_monitor_project-MultiAppReply) | Gets the applications which contain the specified measurement parameter. | | GetAppsHoldingControlParam | [ParameterIdRequest](#system_monitor_project-ParameterIdRequest) | [MultiAppReply](#system_monitor_project-MultiAppReply) | Returns the applications which contain the specified non measurement parameter. | | ParameterExists | [ExistsRequest](#system_monitor_project-ExistsRequest) | [ExistsReply](#system_monitor_project-ExistsReply) | Identifies if a parameter of the given type exists. | | RegisterEnhancedRowParameters | [.system_monitor_common.AppParametersRequest](#system_monitor_common-AppParametersRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Add parameters from application to the enhanced speed row data list. | | ClearEnhancedRowParameters | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Clear all parameters for given application from enhanced speed slow row data list. | | RegisterCANEnhancedRowParameters | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Add CAN parameters to the enhanced speed row data list. | | RegisterVirtualEnhancedRowParameters | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Add virtual parameters to the enhanced speed row data list. All parameters referenced by each virtual will be added to the list. | | ActivateEnhancedRowParameters | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Activate current enhanced speed row data list. | | DumpEvents | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Dumps the contents of the Event window to a specified file. | | DumpErrors | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Any errors that are currently occurring are dumped to disk. | | DumpRowData | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Dumps the current values of all acquired measurement parameters to disk. | | ClearEvents | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Clears the event window in SM-V7. | | GetEvents | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [EventsReply](#system_monitor_project-EventsReply) | Gets the events defined for a given application. | | GetEventDetails | [EventRequest](#system_monitor_project-EventRequest) | [EventReply](#system_monitor_project-EventReply) | Gets the details for a given event defined in an application. | | GetErrorDefinitions | [.system_monitor_common.AppRequest](#system_monitor_common-AppRequest) | [ErrorDefinitionsReply](#system_monitor_project-ErrorDefinitionsReply) | Gets the error definitions for a given application. | | GetErrors | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [ErrorReply](#system_monitor_project-ErrorReply) | Returns list of active errors. | | DeleteErrors | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Clears the active errors. |

Top

## Protos/system_monitor_system.proto ### BatchModeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | mode | [bool](#bool) | | Enter/leave batch mode, true or false respectively. | ### CreatePGVReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | pgv_file_path | [string](#string) | | Path to created PGV file. | | dtv_file_path | [string](#string) | | Path to created DT file. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### CreatePGVRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | location | [string](#string) | | Path to location for output files. | | asap2_file_path | [string](#string) | | Path to base ASAP2 project .a2l file. | | hex_file_path | [string](#string) | | Path to hex definition .hex file. | | controllers_file_path | [string](#string) | | Path to controllers definition .ini file. | | errors_file_path | [string](#string) | | Optional path to errors definition .ini file. | | events_file_path | [string](#string) | | Optional path to events definition .ini file. | | adjustment_file_path | [string](#string) | | Optional path to pot board definition .ini file. | | sensors_file_path | [string](#string) | | Optional path to sensors definition .ini file. | | injector_file_path | [string](#string) | | Optional path to sensor injector definition .ini file. | | sensor_enable_file_path | [string](#string) | | Optional path to sensor enable / disable definition .ini file. | | live_auto_tune_file_path | [string](#string) | | Optional path to live auto tune definition .ini file. | | comments | [string](#string) | | PGV Comments | | notes | [string](#string) | | PGV Notes | ### DeviceProperties | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | comms_path | [string](#string) | | Device communications path. | | device_name | [string](#string) | | Device name. | | ip_address | [string](#string) | | Device ip address. | | serial_number | [int32](#int32) | | Device serial number. | ### DevicePropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | devices | [DeviceProperties](#system_monitor_system-DeviceProperties) | repeated | Device information. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### FolderReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_path | [string](#string) | | File path | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### LicenceDetailsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | consortium | [string](#string) | | Licence Consortium. | | owner | [string](#string) | | Licence Owner. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### LiveLoggingReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | live_logging_state | [bool](#bool) | | Whether Live Logging is enabled or disabled. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### LiveLoggingRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | state | [bool](#bool) | | True to turn Live Logging on, false to turn Live Logging off. | ### LiveUpdateRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | state | [bool](#bool) | | True to enable live updates, false to disable | | action | [uint32](#uint32) | | Action performed on data mismatch: 0 - No Action. 1 - Upload data from unit into edit buffer. 2 - Download edit buffer into unit. (Live tunes in unit may be lost). | ### MultiApplicationBaseInfo | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Name of the multi-application base. | | path | [string](#string) | | Path of the multi-application base. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### MultiApplicationBasesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | info | [MultiApplicationBaseInfo](#system_monitor_system-MultiApplicationBaseInfo) | repeated | Multi-application base information. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### MultiApplicationBasesRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | base_name | [string](#string) | | Name of the multi-application base. | ### OnlineRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | state | [bool](#bool) | | True to turn the ECU online, false to turn the ECU off-line. | ### SendMessageReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | messages | [int32](#int32) | repeated | Sent messages. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### SendMessageRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | app_id | [uint32](#uint32) | | Application Id. (indicates ECU in multiple-ECU systems). | | timeout | [uint32](#uint32) | | Message time out | | retries | [uint32](#uint32) | | How many times to try before returning an error | | messages | [int32](#int32) | repeated | Messages to send. | ### StatusReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | link_status | [LinkStatus](#system_monitor_system-LinkStatus) | | The link status | | online | [bool](#bool) | | Whether the unit is online. | | live_update | [bool](#bool) | | Whether live updates are enabled. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### UnitByIndexRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | Index of unit based on the car/unit list (as returned by GetUnitList()) | ### UnitByIndexTypeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | index | [uint32](#uint32) | | Index of unit based on the car/unit list (as returned by GetUnitList()) | | primary | [bool](#bool) | | Whether the connection is for the main car or spare car | ### UnitInfo | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Unit name. | | type | [string](#string) | | Unit type | | ip_address | [string](#string) | | Unit IP Address | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### UnitListReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | info | [UnitInfo](#system_monitor_system-UnitInfo) | repeated | Unit information. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### UnitNameReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Unit name. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### LinkStatus | Name | Number | Description | | ---- | ------ | ----------- | | Link_OK | 0 | Link OK | | Link_NOK | 1 | Link Not OK | | Controller_Busy | 2 | Controller busy | | In_Boot | 3 | Device in boot | | Zone_1 | 4 | Device in zone one | | Zone_2 | 5 | Device in zone two | | Zone_3 | 6 | Device in zone three | | Bad_Response | 7 | Bad response | | Invalid_Device | 8 | Invalid device | | Unknown | 65535 | Unknown status | ### SystemMonitorSystem | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | GetStatus | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [StatusReply](#system_monitor_system-StatusReply) | Gets the combined current status of online, live updates, link status and active app. | | SetOnline | [OnlineRequest](#system_monitor_system-OnlineRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Turns the ECU on or off-line. | | SetLiveUpdate | [LiveUpdateRequest](#system_monitor_system-LiveUpdateRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Enables or disables the sending of live-updates to the ECU. | | GetUnitList | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [UnitListReply](#system_monitor_system-UnitListReply) | Returns an array of cars/units defined in the carfile.ini (ie. the connections that appear in the connections dialog) | | GetUnitName | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [UnitNameReply](#system_monitor_system-UnitNameReply) | Returns the name of the current car/connection. | | GetUnitByIndex | [UnitByIndexRequest](#system_monitor_system-UnitByIndexRequest) | [UnitInfo](#system_monitor_system-UnitInfo) | Returns the index of the car/unit entry within the car/unit list (as returned by GetCarList()). Obtains connection name, type and IP address. | | SetUnitByIndex | [UnitByIndexTypeRequest](#system_monitor_system-UnitByIndexTypeRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the car/unit connection based on the car/unit list (as returned by GetCarList()) | | GetMultiApplicationBases | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [MultiApplicationBasesReply](#system_monitor_system-MultiApplicationBasesReply) | Gets details of all defined Multi-Application Bases. | | GetMultiApplicationBase | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [MultiApplicationBaseInfo](#system_monitor_system-MultiApplicationBaseInfo) | Gets details of a specific Multi-Application Base. | | SetMultiApplicationBase | [MultiApplicationBasesRequest](#system_monitor_system-MultiApplicationBasesRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets details of a specific Multi-Application Base. | | GetLicenceDetails | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [LicenceDetailsReply](#system_monitor_system-LicenceDetailsReply) | Get the active consortium name and licence owner. | | GetDeviceProperties | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [DevicePropertiesReply](#system_monitor_system-DevicePropertiesReply) | Get the properties of connected devices from the project. | | GetLiveLogging | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [LiveLoggingReply](#system_monitor_system-LiveLoggingReply) | Returns whether the Live Logging is currently recording or not. | | SetLiveLogging | [LiveLoggingRequest](#system_monitor_system-LiveLoggingRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Starts or stops Live Logging. | | SetBatchMode | [BatchModeRequest](#system_monitor_system-BatchModeRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Call this to enter (or leave) batch mode. This is used when creating virtual parameters; entering batch mode before the creation of multiple Virtual Parameters and leaving batch mode afterwards. Using this method, all created/updated Virtual Parameters will only be available in System Monitor after leaving batch mode. | | SendMessage | [SendMessageRequest](#system_monitor_system-SendMessageRequest) | [SendMessageReply](#system_monitor_system-SendMessageReply) | Sends a message to the ECU. | | GetLogFolder | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [FolderReply](#system_monitor_system-FolderReply) | Get the current log folder location. | | GetPPOFileName | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [FolderReply](#system_monitor_system-FolderReply) | Get the name of the PPO file in the project. | | CreatePGV | [CreatePGVRequest](#system_monitor_system-CreatePGVRequest) | [CreatePGVReply](#system_monitor_system-CreatePGVReply) | Create a new PGV from ASAP2 definitions. |

Top

## Protos/system_monitor_virtual.proto ### AddGroupRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | group_path | [string](#string) | | The path where the groups is to be added, e.g. \\TopLevelGroup\\SubGroup | | name | [string](#string) | | The display name for the virtual parameter group. | | description | [string](#string) | | Description of the virtual parameter group. | | read_only | [bool](#bool) | | If set to true, the virtual parameter group will be marked as a Read Only. | ### VirtualExportRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | file_path | [string](#string) | | File path to export parameters into. | | group | [string](#string) | | Group name to export. NULL for all virtual parameters. | ### VirtualGroupReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | name | [string](#string) | | Name of the virtual parameter group. | | description | [string](#string) | | Description of the virtual parameter group. | | read_only | [bool](#bool) | | Whether the virtual parameter group is Read Only | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### VirtualGroupRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | group | [string](#string) | | Identifier of the group. | ### VirtualGroupsReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | ids | [string](#string) | repeated | Identifiers of the virtual parameters. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### VirtualParameter | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | Identifier of the virtual parameter. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### VirtualParameterDataTypeRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | Id | [string](#string) | | The unique identification of the virtual parameter. | | data_type | [system_monitor_common.DataType](#system_monitor_common-DataType) | | The data type. | ### VirtualParameterProperties | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | Id | [string](#string) | | Parameter identifier. | | name | [string](#string) | | Parameter display names. | | description | [string](#string) | | Parameter description. | | lower_display_limit | [double](#double) | | Lower display limit. | | upper_display_limit | [double](#double) | | Upper display limit. | | min_logging_rate | [uint32](#uint32) | | Minimum logging rate. | | scaling_factor | [uint32](#uint32) | | Parameter scaling factor (see EScalingUnit). | | min_not_defined | [bool](#bool) | | State of min/max values. | | expression | [string](#string) | | Expression used to evaluate the virtual parameter. | | units | [string](#string) | | Parameter units. | | format | [string](#string) | | Parameter display format. | | group | [string](#string) | | Parameter group name. | | conversion_id | [string](#string) | | Unique identifier of the parameter conversion rule. | | data_type | [system_monitor_common.DataType](#system_monitor_common-DataType) | | Parameter data type. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code. | ### VirtualParameterPropertiesReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [VirtualParameterProperties](#system_monitor_virtual-VirtualParameterProperties) | repeated | The properties of the virtual parameters | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### VirtualParameterRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [string](#string) | | The unique identifier of the virtual parameter. | | name | [string](#string) | | The display name for the virtual parameter. | | description | [string](#string) | | Description of the virtual parameter. | | min_display | [double](#double) | | Minimum value for display range. | | max_display | [double](#double) | | Maximum value for display range. | | Min_logging_rate | [int32](#int32) | | Minimum logging rate. | | scaling_factor | [int32](#int32) | | Scaling factor. See EScalingUnit in Enumeration Definitions | | is_min_not_def | [bool](#bool) | | If set to true, the virtual parameter will be marked as a Prime Parameter. | | expression | [string](#string) | | The expression to evaluate in the virtual parameter. | | conversion_id | [string](#string) | | The conversion rule identifier for converting the output value to appropriate unit values. | | overwrite | [bool](#bool) | | Overwrites existing data | | units | [string](#string) | | Sets the overriding units in which to display the virtual parameter’s value. | | format_override | [string](#string) | | Sets the overriding format in which to display the virtual parameter’s value. | | group | [string](#string) | | The path of the group in which to add the virtual parameter, e.g. \\TopLevelGroup\\SubGroup | | data_type | [system_monitor_common.DataType](#system_monitor_common-DataType) | | The data type. | | lower_warning | [double](#double) | | Lower warning limit value. | | upper_warning | [double](#double) | | Upper warning limit value. | ### VirtualReply | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | ids | [VirtualParameter](#system_monitor_virtual-VirtualParameter) | repeated | Identifiers of the virtual parameters. | | return_code | [system_monitor_common.ErrorCode](#system_monitor_common-ErrorCode) | | Return code | ### VirtualsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | ids | [string](#string) | repeated | Identifiers of the virtual parameters. | ### SystemMonitorVirtual | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | SetVirtualParameter | [VirtualParameterRequest](#system_monitor_virtual-VirtualParameterRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Modify or Create a Virtual Parameter. | | GetVirtualParameterProperties | [.system_monitor_common.ParametersRequest](#system_monitor_common-ParametersRequest) | [VirtualParameterPropertiesReply](#system_monitor_virtual-VirtualParameterPropertiesReply) | Gets the properties of the given virtual parameters | | RemoveVirtualParameters | [VirtualsRequest](#system_monitor_virtual-VirtualsRequest) | [VirtualReply](#system_monitor_virtual-VirtualReply) | Removes virtual parameters. | | RemoveAllVirtualParameters | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes all virtual parameters. | | RemoveVirtualConversions | [VirtualsRequest](#system_monitor_virtual-VirtualsRequest) | [VirtualReply](#system_monitor_virtual-VirtualReply) | Removes virtual parameter conversion rules. | | RemoveAllVirtualConversions | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes all virtual parameter conversion rules. | | GetVirtualParameterGroups | [.google.protobuf.Empty](https://protobuf.dev/reference/protobuf/google.protobuf/#empty) | [VirtualGroupsReply](#system_monitor_virtual-VirtualGroupsReply) | Gets the virtual parameter groups. | | GetVirtualParameterGroup | [VirtualGroupRequest](#system_monitor_virtual-VirtualGroupRequest) | [VirtualGroupReply](#system_monitor_virtual-VirtualGroupReply) | Gets a virtual parameter group. | | GetVirtualParametersInGroup | [VirtualGroupRequest](#system_monitor_virtual-VirtualGroupRequest) | [VirtualGroupsReply](#system_monitor_virtual-VirtualGroupsReply) | Gets the virtual parameters within a group. | | VirtualParametersExport | [VirtualExportRequest](#system_monitor_virtual-VirtualExportRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Exports virtual parameters. | | VirtualParametersImport | [.system_monitor_common.FileRequest](#system_monitor_common-FileRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Imports virtual parameters. | | AddVirtualParameterGroup | [AddGroupRequest](#system_monitor_virtual-AddGroupRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Adds a virtual parameter group. | | RemoveVirtualParameterGroup | [VirtualGroupRequest](#system_monitor_virtual-VirtualGroupRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes a virtual parameter group. | | RemoveAllVirtualParametersFromGroup | [VirtualGroupRequest](#system_monitor_virtual-VirtualGroupRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Removes all virtual parameters from a group. | | SetVirtualParameterDataType | [VirtualParameterDataTypeRequest](#system_monitor_virtual-VirtualParameterDataTypeRequest) | [.system_monitor_common.Return](#system_monitor_common-Return) | Sets the data type of a virtual parameter. | ## Scalar Value Types | .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | | ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | | double | | double | double | float | float64 | double | float | Float | | float | | float | float | float | float32 | float | float | Float | | int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | | uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | | uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | | sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | | fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | | fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | | sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | | bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | | string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | | bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | --- # Overview System Monitor includes an ActiveX/OLE Automation interface, which exposes programmable System Monitor objects to other applications. This allows external applications to control or request information from a running copy of System Monitor. External applications may be written in any language that supports ActiveX Automation interfaces (see ActiveX Client Languages). ActiveX applications, or Scripts, may be run from a menu command in System Monitor by Custom Menus. We take all possible steps to ensure that any Automation Client code will be compatible with future versions of System Monitor. However, the Automation Interface may change in future releases, both in terms of the functions provided, and the arguments supplied to these functions. We cannot respond to these changes until they occur. !!! note ActiveX scripts compiled in VS2003 and earlier will need to be re-compiled in VS2008 before they can be used with System Monitor 8.0 and later. Please note that this is a Microsoft requirement/compatibility issue and not one that has been introduced by MESL. ## Accessing ActiveX Functions In order to gain access to System Monitor, a function must be called to create an instance of the System Monitor Automation Server. The exact syntax of this function varies depending on the client programming language, but it is typically called `CreateObject`. The argument supplied to this function should be `'System Monitor API’`. Please refer to the documentation for your language for further details. The call to `CreateObject` will fail unless System Monitor is already running. System Monitor cannot be started by an ActiveX function call. The instance of the System Monitor Automation Server created by `CreateObject` must be released once the client script has finished executing. Failing to release the Automation Server will prevent other clients from accessing the server. The syntax of the call to release the server is language specific but is typically achieved by setting the returned instance to `Nothing`. !!! note Only one Automation Client can have access to System Monitor at any one time. Once a client has called `CreateObject`, any other calls to `CreateObject` will be rejected until the instance has been released. All functions require a project to be loaded in System Monitor. System Monitor Version 7 Automation supports Methods. Microsoft’s Object Description Language (ODL) is used to describe function arguments. Some functions take an argument (`wAppID`) which indicates the application in a multiple-ASAP environment (System, Chassis or Engine). The embedded code Release Notes should be referred to for this information. All functions return a signed integer `ErrorCode`, where `0` indicates success, and negative values indicate error codes. The contents of messages and responses are described in the System Monitor Message Descriptions document. ## Supported Languages for Client Scripts Any language that supports ActiveX automation may be used to write client scripts for System Monitor. === "SM Script" System Monitor includes its own proprietary scripting language and script development environment based on Sax Basic. The SM Script Interactive Development Environment (IDE) is distributed with System Monitor. It is installed, in the same location as System Monitor, as part of a Custom Installation. This version of SM Script is only licensed for use with System Monitor and must not be redistributed. To open the SM Script Editor from the Start menus select: **All Programs > System Monitor V7 > SM Script Editor**. SM Script is similar to Microsoft Visual Basic for Applications, and therefore does not support pointer types. An ActiveX function that receives a pointer argument may be supplied with a non-pointer variable of the correct type. !!! note There are some differences between SM Script and Microsoft Visual Basic for Applications - refer to the SM Script help for further details. This help includes several example programs written in SM Script functions. The System Monitor type library can be accessed from SM Script. This allows the ActiveX automation methods available in System Monitor to be viewed and selected. **Configure SM Script to recognize the System Monitor Type Library:** 1. Select: **Edit > References** - The Reference dialog box opens. 2. Select: **System Monitor Type Library** and click **OK**. **View or select System Monitor methods and data types:** 1. Select: **Debug > Browse**. 2. In the ActiveX Automation Members dialog box select **SystemMonitor** from the Library drop-down list. === "Visual C++" System Monitor includes a type library, which is installed in the same directory as System Monitor. This type library may be imported into Visual C++ projects to automatically generate a C++ wrapper class for the System Monitor ActiveX interface. For example: ```cpp #import "C:\\Program Files\\Mclaren Electronic Systems\\System Monitor V7\\SystemMonitor.tlb" rename("SendMessage", "SmSendMessage") named_guids ``` !!! note The 'rename' statement prevents any conflict with the SendMessage API included in ``. This generates a class which can then be declared and used as follows: ```cpp HRESULT hr = S_OK; // ITSysMonApiPtr is the same as a _com_ptr_t SystemMonitor::ITSysMonApiDualPtr pSmApi; // Create instance if system monitor is running if (SUCCEEDED(hr) && IsSysMonRunning()) { hr = pSmApi.CreateInstance(SystemMonitor::CLSID_TSysMonApi); } ``` === "Visual Basic" Visual Basic does not support pointer types. An ActiveX function that receives a pointer argument may be supplied with a non-pointer variable of the correct type. The System Monitor type library can be accessed from Visual Basic. This allows the ActiveX automation methods available in System Monitor to be viewed and selected. **Configure Visual Basic Script to recognize the System Monitor Type Library:** 1. Select: **Project > References**. 2. In the References dialog box check **System Monitor Type Library** and click **OK**. Visual Basic lists available ActiveX function calls when entering code (the Auto List Methods feature). --- # Methods ## ECU Control ### **ClearEvents** **Description:** Clears the event window in SM-V7. **Arguments:** - None ### **DownloadDataChanges** **Description:** Causes any modifications made to the System Monitor edit buffer to be downloaded to the ECU. **Arguments:** - `unsigned short wAppId` - Application Id ### **DownloadRemoteInProgress** **Description:** Returns whether a remote logging is currently in progress. **Arguments:** - `bool* bResult` - whether a remote logging is currently in progress ### **EditBufferSynced** **Description:** Returns true if the edit buffer is in sync with the ECU. **Arguments:** - `unsigned short wAppId` - Application Id ### **ForceReprogram** **Description:** Performs a Force reprogram on the ECU - error code or `SMAPI_ERR_NONE` if function successful. Accepts an array of app ids to reprogram. **Arguments:** - `VARIANT FAR & awAppIds` ### **GetCarByIndex** **Description:** Returns the index of the car/unit entry within the car/unit list (as returned by `GetCarList()`). Obtains connection name, type and IP address. **Arguments:** - `unsigned short wIndex` - `BSTR FAR* pszName` - `BSTR FAR* pszType` - `BSTR FAR* pszIPAddress` ### **GetCarList** **Description:** Returns an array of cars/units defined in the `carfile.ini` (connections that appear in the connections dialog). **Arguments:** - `VARIANT* pszName` - `VARIANT* pszType` - `VARIANT* pszIPAddress` ### **GetCarName** **Description:** Returns the name of the current car/connection. **Arguments:** - `BSTR* pszResult` - Car name ### **GetLinkStatus** **Description:** Gets the ECU link status. Possible return values: ``` 0 - Link OK 1 - Link Not OK 2 - Controller is Busy 3 - Device is in boot 4 - Device is in Zone 1 5 - Device is in Zone 2 6 - Device is in Zone 3 7 - Bad response 8 - Wrong device connected -1 - Unknown Status ``` **Arguments:** - `unsigned short wAppId` - Application Id ### **GetLiveUpdates** **Description:** Gets the status of Live Update mode from the ECU. **Arguments:** - `bool* bResult` - state of Live Updates ### **GetOnline** **Description:** Returns whether the ECU is online or not. **Arguments:** - `bool* bResult` - whether the ECU is online or offline ### **GetStatus** **Description:** Gets the combined current status of online, live updates, link status and active app. **Arguments:** - `BOOL* pbOnlineResult` - On or offline - `BOOL* pbLiveUpdatesResult` - Live updates on/off - `short wLinkStatusAppId` - Application Id to check link status - `short* pwLinkStatusResult` – status of link to ECU. Valid values: ``` 0 = Link OK 1 = Link NOK 2 = Controller Busy 3 = Device In Boot 4 = Device In Zone One 5 = Device In Zone Two 6 = Device In Zone Three 7 = Bad Response 8 = Invalid Device 9 = Unknown Status ``` - `short* pwActiveAppResult` - Mask of active app ids ### **Reprogram** **Description:** Reprograms the ECU. Returns long - error code or `SMAPI_ERR_NONE` if successful. Application IDs can be combined to reprogram multiple apps. **Arguments:** - `unsigned short wAppId` - Application Id - `BSTR FAR* pszResult` ### **ReprogramEx** **Description:** Accepts an array of all app ids to reprogram. **Arguments:** - `const VARIANT FAR& awAppIds` ### **SendMessage** **Description:** Sends a message to the ECU and returns the reply. **Arguments:** - `unsigned short wAppId` - Application Id - `VARIANT SendMessage` - Message to send - `VARIANT* ReplyMessage` – Reply - `unsigned short wTimeOut` - Message timeout ### **SendMessageEx** **Description:** Sends a message to the ECU and returns the reply. Allows specifying number of tries. **Arguments:** - `unsigned short wAppId` - Application Id - `VARIANT SendMessage` - Message to send - `VARIANT* ReplyMessage` - Reply - `unsigned short wTimeOut` - Message timeout - `unsigned short wTries` - Number of tries before error ### **SendModemMessage** **Description:** Sends a message to the ECU (via modem) and returns the reply. **Arguments:** - `unsigned short wAppId` - Application Id - `VARIANT SendMessage` - Message to send - `VARIANT* ReplyMessage` - Reply - `unsigned short wTimeOut` - Message timeout ### **SendModemMessageEx** **Description:** Sends a message to the ECU (via modem) and returns the reply. Allows specifying number of tries. **Arguments:** - `unsigned short wAppId` - Application Id - `VARIANT SendMessage` - Message to send - `VARIANT* ReplyMessage` - Reply - `unsigned short wTimeOut` - Message timeout - `unsigned short wTries` - Number of tries before error ### **SetCarByIndex** **Description:** Sets the car/unit connection based on the car/unit list (as returned by `GetCarList()`). **Arguments:** - `short wIndex` - `bool bPrimaryCar` - True if main car, False if spare ### **SetLiveUpdates** **Description:** Enables or disables sending of live updates to the ECU. **Arguments:** - `bool bLiveUpdates` - True to enable, False to disable ### **SetLiveUpdatesEx** **Description:** Enables/disables live updates with ability to define default action when data version matches but contains different data. **Arguments:** - `bool bState` - True to enable, False to disable - `WORD wAction` - Action on data mismatch: ``` 0 - No Action 1 - Upload data from unit into edit buffer 2 - Download edit buffer into unit (Live tunes in unit may be lost) ``` ### **SetOnline** **Description:** Turns the ECU on or off-line. **Arguments:** - `bool bOnLine` - True to turn online, False to turn offline ### **UndoDataChanges** **Description:** Reset all tune buffers on the unit and/or in the edit buffer. **Values for reset type:** ``` 1 - Undo Data Changes in Unit 2 - Undo Data Changes in Edit Buffer 3 - Undo Data Changes in Unit & Edit Buffer ``` **Arguments:** - `short wType` - Undo type ## Project Management ### **AddApplication** **Description:** Adds the application defined by the given DTV and it's associated PGV into the project. If the application already exists in the project the error: SMAPI_ERR_INVALID_CMD (-31) will be returned. **Arguments:** - `BSTR szDTVPath - Filename of the DTV to load.` ### **AddParametersToUnlockList** **Description:** Update a PUL file for an application for use with Reduced Data Access by adding the requested parameter identifiers. **Arguments:** - `short wAppId - Application Id.` - `VARIANT aszIdentifiers – Safe Array (string) of parameter identifiers to be added to the PUL file.` - `BSTR* pszFileName - Returns the updated PUL file name.` ### **CANConfigGet** **Description:** Get the name of the CAN configuration from the given slot. **Arguments:** - `short wIndex - slot index (1-8)` - `BSTR* pszFileName - Name and path of the config file.` ### **CANConfigLoad** **Description:** Loads the specified CAN configuration file **Arguments:** - `BSTR szIdentifier - Filename and full path of the CAN Config file (.clc).` ### **CANConfigLoadEx** **Description:** Loads the specified CAN configuration file into the given slot and sets active if required. **Arguments:** - `short wIndex - Slot index (1-8).` - `BSTR szIdentifier - Filename and full path of the CAN Config file (.clc).` - `bool bActive - True to set active.` ### **CANConfigSave** **Description:** Saves the current CAN configuration to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `BSTR szFullPath - File path to save.` - `BSTR szComment - Comment.` - `BSTR szNotes - Notes.` ### **CANConfigUnload** **Description:** Unloads the CAN configuration from the given slot. **Arguments:** - `short wIndex - Slot index (2-8).` ### **ClearDataVersionNotes** **Description:** Clears the contents of the notes attribute associated with the DTV. **Arguments:** - `unsigned short wAppId - Application Id.` ### **CloseProject** **Description:** Closes the Project. **Arguments:** - `None` ### **CloseProjectEx** **Description:** Closes the Project. **Arguments:** - `unsigned short wAction - Action to perform when unsaved documents encountered on closing project.` - `0 - Cancel Save.1 - Save All.2 - Save None.` ### **CompareApp** **Description:** Perform a compare between an application loaded in memory and up to two DTV's. When two DTV's are used both must have the same PGV version. Returns a list of parameters that contain differences along with the parameter data type and a reason code identifying the difference. **Arguments:** - `short wAppId – Application Id.` - `BSTR szDTV1- Filename and full path of the first DTV to compare against (.dtv).` - `BSTR szDTV2 - Filename and full path of the second DTV to compare against (.dtv).` - `VARIANT* str1xArrParamIds - Array of unique parameter identifiers.` - `VARIANT* str1xParamType - Array of parameter types.` - `VARIANT* w1xArrReason1 - Array of reason codes for difference between the parameter in the application in memory and the first DTV supplied.` - `VARIANT* w1xArrReason2 - Array of reason codes for difference between the parameter in the application in memory and the first DTV supplied.` - `Reason Codes (combination of one or more of):` - `0x0001 -EQUATE_ABSENT              - Not present in program version of the candidate.0x0002 -EQUATE_DIFFERENT           - Major difference that prevents merge.0x0004 -EQUATE_EQUAL               - Same data type, size and value.0x0008 -EQUATE_DIFF_VALUE          - Same data type, size, units, diff value.0x0010 -EQUATE_DIFF_SIZE           - Same data type, different size (maps only).0x0020 -EQUATE_DIFF_CONV           - Different display format conversion.0x0040 -EQUATE_DIFF_UNITS          - Different units.0x0080 -EQUATE_DIFF_DATA_TYPE      - Different data type.0x0100 -EQUATE_DIFF_COMMENT        - Different comment.0x0200 -EQUATE_DIFF_DEFAULT_VALUE  - Different default value.0x0400 -EQUATE_ABSENT_MASTER       - Not present in program version of the edit buffer.` ### **CreateNewCANConfig** **Description:** Create a new CAN configuration file. **Arguments:** - `BSTR szIdentifier - Filename and full path of the Config file (.clc).` - `bool bSaveExisting - Save current configuration before load.` - `bool bOverwrite - Overwrite file if exists.` ### **CreateNewRemoteConfig** **Description:** Create a new remote logging configuration file. **Arguments:** - `BSTR szIdentifier - Filename and full path of the Config file (.rlc).` - `bool bSaveExisting - Save current configuration before load.` - `bool bOverwrite - Overwrite file if exists.` ### **CreateNewVirtualParameters** **Description:** Create a new virtual parameters file. **Arguments:** - `BSTR szIdentifier - Filename and full path of the virtual parameter file (.vpm).` - `bool bSaveExisting - Save current configuration before load.` - `bool bOverwrite - Overwrite file if exists.` ### **CreatePGV** **Description:** Create a new PGV from ASAP2 definitions. **Arguments:** - `BSTR szLocation - Path to location for output files.` - `BSTR szASAP2File - Path to base ASAP2 project .a2l file.` - `BSTR szHexFile - Path to hex definition .hex file.` - `BSTR szControllersFile - Path to controllers definition .ini file.` - `BSTR szErrorsFile - Optional path to errors definition .ini file.` - `BSTR szEventsFile - Optional path to events definition .ini file.` - `BSTR szAdjustmentParametersFile - Optional path to pot board definition .ini file.` - `BSTR szSensorsFile - Optional path to sensors definition .ini file.` - `BSTR szSensorInjectorFile - Optional path to sensor injector definition .ini file.` - `BSTR szSensorEnableFile - Optional path to sensor enable / disable definition .ini file.` - `BSTR szLiveAutoTuneFile - Optional path to live auto tunedefinition .ini file.` - `BSTR szComment - PGV Comments.` - `BSTR szNotes - PGV Notes.` - `BSTR* pszPGV - Path to created PGV file.` - `BSTR* pszDTV - Path to created DTV file.` ### **CreateProject** **Description:** Create a new project. **Arguments:** - `BSTR szProject - Project name to be created within the current multi-application base.` - `VARIANT aszApps - Safe Array (string) of PGV files to be included.` - `BSTR szDesktop - Name of desktop document to be created or included if file already exists.` - `BSTR szVirtuals - Optional name of virtual parameters document to be created or included if file already exists.` - `BSTR szCAN - Optional name of CAN configuration document to be created or included if file already exists.` - `BSTR szRemoteConfig - Optional name of remote logging configuration document to be created or included if file already exists.` ### **CreateFFCFromPGV** **Description:** Create an FFC File from the given PGV. **Arguments:** - `BSTR szPGV - PGV filename to be processed into FFC.` ### **DataVersionModified** **Description:** Returns the modified state associated with the DTV. **Arguments:** - `unsigned short wAppId - Application Id.` - `bool* pbResult` ### **DataVersionOpen** **Description:** Opens the specified data version. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path of data version.` ### **DataVersionSave** **Description:** Saves the current data version to the specified file. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `BSTR szComments - Prefix to existing comment.` - `BSTR szNotes - Prefix to existing notes.` ### **DataVersionSaveCopyAs** **Description:** Saves the current data version to the specified file, without modifying the data version in the edit buffer. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `BSTR szComments - Prefix to existing comment.` - `BSTR szNotes - Prefix to existing notes.` ### **DataVersionSaveCopyAsEx** **Description:** Saves the current data version to the specified file, without modifying the data version in the edit buffer. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `BSTR szComments - Prefix to existing comment.` - `BSTR szNotes - Prefix to existing notes.` - `BSTR szConsortium - Consortium name to save data version to.` ### **DataVersionSaveIncrement** **Description:** Increments the version number of the data version and then performs a save. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szComments - Prefix to existing comment.` - `BSTR szNotes - Prefix to existing notes.` ### **DesktopLoad** **Description:** Loads the specified System Monitor desktop file **Arguments:** - `BSTR szIdentifier - Filename and full path of the Desktop file (.dtp).` ### **DesktopSave** **Description:** Saves the current desktop to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `LPCTSTR szFullPath - File path to save.` - `LPCTSTR szComment - Comment.` - `LPCTSTR szNotes - Notes.` ### **DesktopSaveEx** **Description:** Saves the current desktop to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `BSTR szFullPath - File path to save.` - `BSTR szComment -Comment.` - `BSTR szNotes - Notes.` - `BSTR szConsortium - Consortium name to save data version to. OR "Disabled" to unlock.` ### **EnableDataVersionBackup** **Description:** Enables/Disables the data version backup during parameter value changes. Adds a message to the log file to indicate when changed. **Arguments:** - `boolean - True to enable,false to disable.` ### **ExportInputSignals** **Description:** Exports the Input Signal configuration in XML format. **Arguments:** - `BSTR szFullPath - Filename and full path.` ### **ExportToHexFile** **Description:** Exports the selected applications PGV and DTV to a hexfile. **Arguments:** - `unsigned short wAppId - Application Id.` ### **ExportProject** **Description:** Exports the project to a .smx file. **Arguments:** - `bool bSaveModified - Flag to indicate if any unsaved files in the project should be saved before exporting.` ### **GenerateParamSet** **Description:** Generate a parameter set file. **Arguments:** - `VARIANT aszIdentifiers – Safe Array (string) of parameter identifiers to be included.` - `BSTR pszFileName - Full path name of the parameter set (.pms) file.` ### **GeneratePULFile** **Description:** Generate a PUL file for an application for use with Reduced Data Access. **Arguments:** - `short wAppId - Application Id.` - `VARIANT aszIdentifiers – Safe Array (string) of parameter identifiers to be included.` - `BSTR pszFileName - Full path name of the PUL (.pul) file.` ### **GeneratePULFileFromParamSet** **Description:** Generate a PUL file for an application for use with Reduced Data Access from a saved Parameter Set file. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszPMSFile - Full path name of the Parameter Set (.pms) file.` - `BSTR* pszFileName - Full path name of the PUL (.pul) file. On sucessful creating of PUL file this variable returns the new full PUL file name.` ### **GetActiveApp** **Description:** Returns bitwise OR of the application IDs of all the active applications. **Arguments:** - `unsigned short* pwResult - result bitwise OR of application Ids.` ### **GetActiveAppEx** **Description:** Returns a Variant WORD array containing the id's all active apps. **Arguments:** - `VARIANT FAR * pawAppIds` ### **GetActiveCANConfig** **Description:** Get the active status of the CAN configuration in the given slot. **Arguments:** - `short wSlotId - Slot index (1-8).` - `bool* pbActive - Active state.` ### **GetActiveRemoteLogging** **Description:** Get the active status of the remote logging configuration in the given slot. **Arguments:** - `short wSlotId - Slot index (1-8).` - `bool* pbActive - Active state.` ### **GetAppDetails** **Description:** Retrieves the ID and name for the applications in the current project. **Arguments:** - `VARIANT FAR * pawAppIds - Array of Application Ids.` - `VARIANT FAR * pastrAppNames - Array of Names.` ### **GetAppPULFileName** **Description:** Get the name of the unlock list PUL file for the given application id. **Arguments:** - `short wAppId - Application Id.` - `BSTR* pszFileName - Name of the PUL file, if set.` ### **GetBuildNumber** **Description:** Returns the build number of SM-V7. Client code may be branched on the basis of build number, thereby providing a way by which a single client script can be written which works with releases of SM-V7 which have different ActiveX interfaces. **Arguments:** - `unsigned short* pwResult - Build number.` ### **GetDataVersion** **Description:** Returns the filename of the data version in the edit buffer. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR* pszResult` ### **GetDataVersionComment** **Description:** Returns the contents of the comments attribute associated with the DTV. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR* pszComment` ### **GetDataVersionNotes** **Description:** Returns the contents of the notes attribute associated with the DTV. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR* pszResult` ### **GetDataVersionSavedOn** **Description:** Returns the contents of the saved on attribute associated with the DTV. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR* pszResult` ### **GetDesktopFileName** **Description:** Get the name of the desktop file from the project. **Arguments:** - `BSTR* pszFileName - Name and path of the file.` ### **GetDeviceProperties** **Description:** Get the properties of connected devices from the project. **Arguments:** - `VARIANT* strw1xArrCommsPath - String array of device comms paths.` - `VARIANT* str1xArrDeviceName - String array of device names.` - `VARIANT* dw1xArrSerialNumber - Integer array of device serial numbers.` - `VARIANT* str1xArrAddress - String array of device addresses.` ### **GetEcuDataVersion** **Description:** Returns the filename of the data version on the ECU. **Arguments:** - `short wIndex - Slot index (1-8).` - `bool bActivate - Active state.` ### **GetErrors** **Description:** Returns list of active errors. **Arguments:** - `VARIANT* str1xError - Safe Array of error identifiers.` - `VARIANT* str1xDescription - Safe Array of error descriptions.` - `VARIANT* w1xStatus - Safe Array of error status levels. ( 0 - Unknown, 1 - Current, 2 - Logged )` ### **GetEvents** **Description:** Returns variant string arrays of defined events for a given application.. **Arguments:** - `unsigned short wAppId - Application Id.` - `unsigned short* pwCount - Number of events defined in the application.` - `VARIANT* pwEventIds - Safe Array of event identifiers.` - `VARIANT* pszEvents - Safe Array of event descriptions.` - `VARIANT* pwLevels - Safe Array of event levels. ( 0 - High, 1 - Medium, 2 - Low, 3 - Debug )` ### **GetEventDetails** **Description:** Returns the details for a given event definition in an application.. **Arguments:** - `unsigned short wEventId - Event Id.` - `unsigned short wAppId - Application Id.` - `BSTR* pszDescription - Event description.` - `BSTR* pszConv1 - Id of first event conversion.` - `BSTR* pszConv2 - Id of second event conversion.` - `BSTR* pszConv3 - Id of third event conversion.` - `unsigned short*  pwLevels - Event level. ( 0 - High, 1 - Medium, 2 - Low, 3 - Debug )` ### **GetFIACANConfig** **Description:** Get the active status of the FIA CAN configuration in the given slot. **Arguments:** - `short wSlotId - Slot index (1-8).` - `bool* pbActive - Active state.` ### **GetFileDetails** **Description:** Get the file details for any file type supported by System Monitorr. **Arguments:** - `BSTR  szFileName - Full pathname of file to interrogate for file details.` - `BSTR* pszSavedBy - Name of the last user to save the file.` - `DATE* pdtSavedOn - Date time of last file save.` - `BSTR* pszNotes - File notes.` - `BSTR* pszComments - File comments.` - `WORD* pwBuild - Build number of System Monitor used when saving file.` - `BSTR* pszConsortium - Name of the licence consortium used to save the file. Set for supported file types only ( .dtv, .rlc & .dtp ).` - `BSTR* pszOwner - Name of the licence owner used to create the file. Set for supported file types only ( .dtv, .rlc & .dtp ).` - `BSTR* pszRDA - RDA unlock list checksum for locked DTV's.Set for supported file types only ( .dtv ).` ### **GetLicenceDetails** **Description:** Get the active consortium name and licence owner. **Arguments:** - `BSTR* pszConsortium - Name of the active consortium set in the licence.` - `BSTR* pszOwner - Name of the owner of the licence.` ### **GetLogFolder** **Description:** Get the current log folder location. **Arguments:** - `BSTR* pszFolder- Path of the log folder.` ### **GetMultiApplicationBase** **Description:** Returns strings containing the name and path of the active Multi-Application Base. **Arguments:** - `BSTR* pszName` - `BSTR* pszPath` ### **GetMultiApplicationBases** **Description:** Returns variant string arrays containing the name and paths of all defined Multi-Application Bases. **Arguments:** - `VARIANT FAR* paNames` - `VARIANT FAR* paPaths` ### **GetNextDataVersion** **Description:** Returns the filename of the next data version that will be saved if the user selects, File, Save. If the edit buffer is unmodified, this will be the name of the current data version in the edit buffer. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR* pszResult` ### **GetPPOFileName** **Description:** Get the name of the PPO file in the project. **Arguments:** - `BSTR* pszFileName - Name and path of the PPO file.` ### **GetProgramVersionID** **Description:** Returns the program version ID of the currently loaded program version. **Arguments:** - `short wAppId - Application Id.` - `short* pwResult - Result program version ID.` ### **GetProgramVersionIDEx** **Description:** Returns the program version ID of the currently loaded program version. **Arguments:** - `unsigned short wAppId - Application Id.` - `unsigned short* pwResult - Result program version ID.` ### **GetProjectFileName** **Description:** Get the name of the project file. **Arguments:** - `BSTR* pszFileName - Name and path of the project.` ### **GetRemoteLoggingDuration** **Description:** Get the estimated length in time and number of laps available for recording in the remote logging config. **Arguments:** - `DATETIME* pdtEstimatedLoggingTime - Estimated time available in recording memory.` - `long* pdEstimatedLoggingLaps - Estimated laps available in recording memory..` ### **GetRemoteLoggingRates** **Description:** Get the Logging and Telemetry rates of all channels in the remote logging config. **Arguments:** - `VARIANT* pafLogging - Array of doubles holding channel logging rates.` - `VARIANT* pafTelemetry - Array of doubles holding channel telemetry rates.` ### **GetRemoteSlotCount** **Description:** Get the number of logging slots used in the remote logging config. **Arguments:** - `short* wSlotCount - Total slots used.` ### **GetVersionNumber** **Description:** Returns the version information contained within the System Monitor executable. **Arguments:** - `short* pwMajorVersion – Major revision.` - `short* pwMinorVersion – Minor revision.` - `short* pwBuildNumber – Build number.` ### **GetVirtualParametersFileName** **Description:** Get the name of the virtual parameters file from the project. **Arguments:** - `BSTR* pszFileName - Name and path of the file.` ### **ImportInputSignals** **Description:** Imports the Input Signal configuration from a XML file. **Arguments:** - `BSTR szFullPath - Filename and full path.` ### **ImportProject** **Description:** Imports the project from a .smx file. **Arguments:** - `BSTR szFullPath - SMX file containing project to be imported.` - `BSTR szBase - Optional Multi-Application base name for import. If blank, the current default base is used.` ### **MatLabExportSelected** **Description:** Exports the specified list of editable parameters in Matlab format. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `bool bDataOnly - True if only data is to be exported, false to export data and definitions.` - `VARIANT* SelectedParams – array of parameter identifiers for the parameters to be exported` ### **MatlabImport** **Description:** Inports the specified MatLab-M file into System Monitor. **Arguments:** - `BSTR szFullPath - Full path to the MatLab-M file.` ### **MatLabMExport** **Description:** Exports the currently loaded data version to the specified file in MatLab-M format. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `bool bDataOnly - True if only data is to be exported, false to export data and definitions.` ### **MatLabMExportEx** **Description:** Exports the currently loaded data version to the specified file in MatLab-M format. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Filename and full path.` - `bool bDataOnly - True if only data is to be exported, false to export data and definitions.` - `unsigned int dwType - Data types included in export.` - `OB_TYPE_SCALAR        = 0x00000001` - `OB_TYPE_1_AXIS_MAP = 0x00000002` - `OB_TYPE_2_AXIS_MAP = 0x00000004` - `OB_TYPE_ARRAY          = 0x00000010` - `OB_TYPE_STRING          = 0x00000020` - `OB_TYPE_ECU                = 0x00000080` - `OB_TYPE_ANY_AXIS     = 0x00030000` ### **MatLabMExportForDTV** **Description:** Exports the specified data version to the specified file in MatLab-M format. **Arguments:** - `BSTR szDTV - Filename and full path of DTV file.` - `BSTR szIdentifier - Filename and full path of exported file.` - `bool bDataOnly - True if only data is to be exported, false to export data and definitions.` - `unsigned int dwType - Data types included in export.` - `OB_TYPE_SCALAR        = 0x00000001` - `OB_TYPE_1_AXIS_MAP = 0x00000002` - `OB_TYPE_2_AXIS_MAP = 0x00000004` - `OB_TYPE_ARRAY          = 0x00000010` - `OB_TYPE_STRING          = 0x00000020` - `OB_TYPE_ECU                = 0x00000080` - `OB_TYPE_ANY_AXIS     = 0x00030000` ### **OpenProject** **Description:** Opens a Project. **Arguments:** - `String szFullPath - The file location to open the project from.` ### **RemoteGet** **Description:** Get the name of the remote logging configuration from the given slot. **Arguments:** - `short wIndex - Slot index (1-8).` - `BSTR* pszFileName - Name and path of the config file.` ### **RemoteGetAll** **Description:** Get the name of the remote logging configuration from the given slot. **Arguments:** - `str1xArrFileNames - Safe Array of names and paths of all loaded config files.` ### **RemoteLoggingLoad** **Description:** Loads the specified remote logging configuration. **Arguments:** - `BSTR szFullPath - File path to load.` ### **RemoteLoggingLoadEx** **Description:** Loads the specified remote logging configuration file into the given slot and sets active if required. **Arguments:** - `short wIndex - Slot index (1-8).` - `BSTR szIdentifier - Filename and full path of the Config file (.rlc).` - `bool bActive - True to set active.` ### **RemoteLoggingSave** **Description:** Saves the current remote logging configuration to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `BSTR szFullPath - File path to save.` - `BSTR szComment -Comment.` - `BSTR szNotes - Notes.` ### **RemoteLoggingSaveEx** **Description:** Saves the current remote logging configuration to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `BSTR szFullPath - File path to save.` - `BSTR szComment -Comment.` - `BSTR szNotes - Notes.` - `BSTR szConsortium - Consortium name to save data version to. OR "Disabled" to unlock.` ### **RemoteLoggingUnload** **Description:** Unloads the remote logging configuration from the given slot. **Arguments:** - `short wIndex - Slot index (2-8).` ### **RemoveApplication** **Description:** Removes the given application from the project. The requested application cannot be the BIOS. **Arguments:** - `unsigned short wAppId - Application Id.` ### **RemoveParametersFromUnlockList** **Description:** Update a PUL file for an application for use with Reduced Data Access by removing the requested parameter identifiers. **Arguments:** - `short wAppId - Application Id.` - `VARIANT aszIdentifiers – Safe Array (string) of parameter identifiers to be removed from the PUL file.` - `BSTR* pszFileName - Returns the updated PUL file name.` ### **SaveProject** **Description:** Saves the current project. **Arguments:** - `bool bSaveAll - If TRUE then all modified files are saved. If FALSE then only the project file is saved.` ### **SaveProjectAs** **Description:** Saves the current project. **Arguments:** - `LPCTSTR pszNewName - Name of renamed project file.` - `bool bSaveAll - If TRUE then all modified files are saved. If FALSE then only the project file is saved.` - `LPCTSTR pszComment - New project comments.` - `LPCTSTR pszNotes - New project notes.` ### **SetActiveApp** **Description:** Sets the active application when provided with an application ID. **Arguments:** - `unsigned short wAppId - Application Id.` ### **SetActiveAppEx** **Description:** This accepts an array of all app ids to make active. Any apps not in the array are made inactive. **Arguments:** - `const VARIANT FAR& awAppIds` ### **SetActiveCANConfig** **Description:** Set the active status of the CAN configuration in the given slot. **Arguments:** - `short wIndex - Slot index (1-8).` - `bool bActivate - Active state.` ### **SetActiveRemoteLogging** **Description:** Set the active status of the remote logging configuration in the given slot. **Arguments:** - `short wIndex - Slot index (1-8).` - `bool bActivate - Active state.` ### **SetAppPULFile** **Description:** Set the unlock list PUL file on the given application id. Licence ownership restrictions apply **Arguments:** - `short wAppId - Application Id.` - `BSTR pszFileName - Name of the PUL file.` - `Error messages:SMAPI_ERR_DATAVER_NONE (-5) if application id not valid.SMAPI_ERR_NOLICENCE (-2) if licence not valid to lock the application.SMAPI_ERR_FILE (-8) if PUL file invalid.` ### **SetDataVersionComment** **Description:** Sets the Data Version comment for the specified application. **Arguments:** - `unsigned short wAppId - Application Id.` - `LPCTSTR pszComment - Data Version Comment.` ### **SetDataVersionNotes** **Description:** Sets the Data Version notes for the specified application. **Arguments:** - `unsigned short wAppId - Application Id.` - `LPCTSTR pszNotes - Data Version Notes.` ### **SetFIACANConfig** **Description:** Set the FIA status of the CAN configuration into the given slot. **Arguments:** - `short wIndex - Slot index (2-8).` ### **SetMultiApplicationBase** **Description:** Sets the current Multi-Application Base. **Arguments:** - `LPCTSTR pszBase - Base name.` ### **UploadDataVersion** **Description:** Uploads specified data version to the edit buffer. **Arguments:** - `unsigned short wAppId - Application Id.` ### **VirtualParametersExport** **Description:** Export virtual parameters in XML format. If supplied, the group name specifies the sub-tree to include in export. All virtual parameter dependants of exported parameters will also be included. **Arguments:** - `LPCTSTR szName - Fiile path to export parameters into.` - `LPCTSTR szGroup - Group name to export. NULL for all virtual parameters.` ### **VirtualParametersImport** **Description:** Import and merge the specified virtual parameter file. **Arguments:** - `LPCTSTR szName - File path to import.` ### **VirtualParametersLoad** **Description:** Loads the specified virtual parameter file (long) **Arguments:** - `LPCTSTR szFullPath - File path to load.` ### **VirtualParametersLoadNoSave** **Description:** Loads the specified virtual parameter file without saving current file (long) **Arguments:** - `LPCTSTR szFullPath - File path to load.` ### **VirtualParametersSave** **Description:** Saves the current virtual parameters as long to the specified file. **Arguments:** - `bool  bSaveCopyAs - Save copy without affecting SM modified status.` - `LPCTSTR szFullPath - File path to save.` - `LPCTSTR szComment - Comment.` - `LPCTSTR szNotes - Notes.` ## Data Acquisition ### **DownloadExternalInputs** **Description:** Downloads the external inputs configuration to the ECU. **Arguments:** - None ### **DumpErrors** **Description:** Any errors that are currently occurring are dumped to disk. **Arguments:** - `BSTR szIdentifier - Filename and full path.` ### **DumpEvents** **Description:** Dumps the contents of the Event window to a specified file. **Arguments:** - `BSTR szIdentifier - Filename and full path.` ### **DumpRowData** **Description:** Dumps the current values of all acquired measurement parameters to disk. **Arguments:** - `BSTR szIdentifier - Filename and full path.` ### **GetErrorDefinitions** **Description:** Gets the error definitions for a given application. **Arguments:** - `short wAppId - Application Id to get the defintions for.` - `pCount - Total number of error defintions in application.` - `paszNames - Array of error defintion names.` - `paszIdentifiers - Array of error defintion identifiers.` - `paszDescriptions - Array of error defintion descriptions.` - `paszGroup - Array of error defintion groups.` - `pawBitNumber - Array of error defintion bit numbers.` - `paszCurrent - Array of error defintion current parameter identifiers.` - `paszLogged - Array of error defintion logged paramer identifiers.` ### **GetLiveLogging** **Description:** Returns whether the Live Logging is currently recording or not. **Arguments:** - `bool* bResult - whether Live Logging is enabled or disabled.` ### **GetProgramVersionPath** **Description:** Gets the filepath for the .pgv of an application by Application Id. **Arguments:** - `short wAppId - Application Id to get the PGV for.` - `BSTR FAR* pszResult - PGV filepath.` ### **SelectCar** **Description:** Select car to which any following API functions will apply. **Arguments:** - `short wCar - Car number 1 (main) or 2 (spare).` ### **SetLiveLogging** **Description:** Starts or stops Live Logging. **Arguments:** - `bool bOnLine - True to turn Live Logging on, false to turn Live Logging off.` ### **BurstLoggingSave** **Description:** Sends the current burst logging configuration to the specified file. **Arguments:** - `bool bSaveCopyAs - Save copy without affecting SM modified status.` - `BSTR szFullPath - File path to save.` - `BSTR szComment - Comment.` - `BSTR szNotes - Notes.` ### **DownloadBurst** **Description:** Downloads the burst logging configuration to the ECU. **Arguments:** - None ### **GetBurstSessionDetail** **Description:** Returns the current value of a specified session detail from the burst logging configuration. Note that this and other session detail functions will operate on the logging configuration loaded in SM-V7, which does not necessarily match the ECU configuration. **Arguments:** - `BSTR szName - Name of session detail.` - `BSTR* pszValue - Returned value for session detail.` ### **SetBurstSessionDetail** **Description:** Sets a specified session detail in the burst logging configuration. **Arguments:** - `BSTR szName - Name of session detail.` - `BSTR szValue - New value for session detail.` ### **UploadBurst** **Description:** Uploads the burst logging configuration from the ECU. **Arguments:** - None ### **AddRemoteParam** **Description:** Adds a parameter to the Remote Logging Configuration. **Arguments:** - `short wAppId – Application Id.` - `LPCTSTR pszParamId – Unique parameter identifier` - `short wChannel – 0 indexed channel to set value for. I.e. Channel 1 = 0.` - `short wValue – The frequency or engine segment value.` - `short wValueType (see ETypeFreqValue):` ### **AddRemoteParamEx** **Description:** Adds a parameter to the Remote Logging Configuration. **Arguments:** - `short wAppId – Application Id.` - `LPCTSTR pszParamId – Unique parameter identifier` - `short wChannel – 0 indexed channel to set value for. I.e. Channel 1 = 0.` - `long wValue – The frequency or engine segment value.` - `short wValueType (see ETypeFreqValue):` ### **AddRemoteVirtualParam** **Description:** Adds a parameter to the Remote Logging Configuration. **Arguments:** - `LPCTSTR pszParamId – Unique parameter identifier` - `short wChannel – 0 indexed channel to set value for. I.e. Channel 1 = 0.` - `short wValue – The frequency or engine segment value.` - `short wValueType (see ETypeFreqValue):` ### **AddRemoteVirtualParamEx** **Description:** Adds a parameter to the Remote Logging Configuration. **Arguments:** - `LPCTSTR pszParamId – Unique parameter identifier` - `short wChannel – 0 indexed channel to set value for. I.e. Channel 1 = 0.` - `long dwValue – The frequency or engine segment value.` - `short wValueType (see ETypeFreqValue):` ### **ClearRemoteParams** **Description:** Removes all the parameters from the remote logging config except those used by triggers. If the caller specifies a forceful removal all parameters will be removed and any >On Data= trigger conditions will be reset. **Arguments:** - `boolean bForceRemovalFromTriggers - Request System Monitor removes all parameters, including those used by triggers.` ### **DownloadRemote** **Description:** Downloads the remote logging configuration to the ECU. **Arguments:** - None ### **DownloadRemoteEx** **Description:** Downloads the remote logging configuration to the ECU the reads the given measurement value to ensure configuration write is complete. **Arguments:** - `short wAppId – Application Id.` - `LPCTSTR pszParamId – Unique identifier of logging configuration parameter.` - `short wReadDelay – Read delay in ms before retrieving parameter value allowing ECU configuration processing time.` - `BSTR* pszValue - Parameters current value.` ### **GetRemoteChannelCount** **Description:** Retrieves the number of channels in current Remote Logging configuration. **Arguments:** - `pnChannelCount - Output parameter to receive the current channel count.` ### **GetRemoteChannelNames** **Description:** Retrieves an array of strings containing the channel names. **Arguments:** - `str1xArrChannelNames - SAFEARRAY contained array of strings.` ### **GetRemoteChannelNamesAll** **Description:** Retrieves an array of strings containing the channel names. **Arguments:** - `str1xArrChannelNames - SAFEARRAY contained array of strings.` - `w1xSlots - Safe Array of matching slot position of configuration defining the channel.` ### **GetRemoteChannelProperties** **Description:** Retrieves the properties of any given Remote Logging channel. **Arguments:** - `short wChannel - The 0 based channel index to modify.` - `BSTR* pstrChannelName - Name of the channel.` - `boolean* pbLogToUnit - Retrieves the Log to Unit flag.` - `boolean* pbTelemetry - Retrieves the telemetry flag.` - `boolean* pbTriggerRearmsAfterStopped - Trigger re-arms after stop condition.` ### **GetRemoteChannelPropertiesAll** **Description:** Retrieves the properties of any given Remote Logging channel. **Arguments:** - `short wChannel - The 0 based channel index to modify.` - `BSTR* pstrChannelName - Name of the channel.` - `boolean* pbLogToUnit - Retrieves the Log to Unit flag.` - `boolean* pbTelemetry - Retrieves the telemetry flag.` - `boolean* pbTriggerRearmsAfterStopped - Trigger re-arms after stop condition.` - `short* - Retrieves the slot position of the configuration defining the channel.` ### **GetRemoteOffset** **Description:** Gets the remote logging config offset. **Arguments:** - `pwOffset- Output parameter to receive the current logging offset.` - `Error messages:SMAPI_ERR_BAD_STATE (-30) if logging config document not present.` - `SMAPI_ERR_INVALID_CMD (-31) if Logging offset adjustment not allowed.` ### **GetRemoteParams** **Description:** Gets the remote logging parameter details from the config. **Arguments:** - `w1xArrAppIds - Safe Array of matching application identifiers.` - `str1xArrParamNames - Safe Array of matching parameter identifiers.` - `str1xArrParamDescriptions - Safe Array of parameter descriptions` - `str1xArrChannelNames - Safe Names of each of the channels.` - `w1xArrParamSizes - Safe Array of matching sizes.` - `w2xArrParamValues - Safe Array of matching values.` - `w2xArrParamLogTypes - Safe Array of matching log types.` - `long* nParamCount - Number of parameters retrieved.` - `long* nChannelCount - Number of channels in config.` ### **GetRemoteParamsAll** **Description:** Gets the remote logging parameter details from the config. **Arguments:** - `w1xArrAppIds - Safe Array of matching application identifiers.` - `str1xArrParamNames - Safe Array of matching parameter identifiers.` - `str1xArrParamDescriptions - Safe Array of parameter descriptions` - `str1xArrChannelNames - Safe Names of each of the channels.` - `w1xArrParamSizes - Safe Array of matching sizes.` - `w2xArrParamValues - Safe Array of matching values.` - `w2xArrParamLogTypes - Safe Array of matching log types.` - `long* nParamCount - Number of parameters retrieved.` - `long* nChannelCount - Number of channels in config.` - `w1xSlots - Safe Array of matching slot position of configuration defining the parameter.` ### **GetRemoteParamsEx** **Description:** Gets the remote logging parameter details from the config. **Arguments:** - `w1xArrAppIds - Safe Array of matching application identifiers.` - `str1xArrParamNames - Safe Array of matching parameter identifiers.` - `str1xArrParamDescriptions - Safe Array of parameter descriptions` - `str1xArrChannelNames - Safe Names of each of the channels.` - `w1xArrParamSizes - Safe Array of matching sizes.` - `w4xArrParamValues - Safe Array of matching values.` - `w2xArrParamLogTypes - Safe Array of matching log types.` - `long* nParamCount - Number of parameters retrieved.` - `long* nChannelCount - Number of channels in config.` ### **GetRemoteParamsExAll** **Description:** Gets the remote logging parameter details from the config. **Arguments:** - `w1xArrAppIds - Safe Array of matching application identifiers.` - `str1xArrParamNames - Safe Array of matching parameter Identifiers.` - `str1xArrParamDescriptions - Safe Array of parameter descriptions` - `str1xArrChannelNames - Safe Names of each of the channels.` - `w1xArrParamSizes - Safe Array of matching sizes.` - `w4xArrParamValues - Safe Array of matching values.` - `w2xArrParamLogTypes - Safe Array of matching log types.` - `long* nParamCount - Number of parameters retrieved.` - `long* nChannelCount - Number of channels in config.` - `w1xSlots - Safe Array of matching slot position of configuration defining the parameter.` ### **GetRemoteParamDetails** **Description:** Gets the remote logging parameter details from the config. **Arguments:** - `awAppIds - Array of matching application identifiers. (Unsigned Short)` - `aszParamIdss - Array of matching parameter names.` - `aszParamNames - Array of matching parameter names.` - `aszParamDescriptions - Array of parameter descriptions` - `aszChannelNames - Names of each of the channels.` - `awParamSizes - Array of matching sizes. (Unsigned Short)` - `a2dwParamValues - Array of matching values. (Array of Unsigned Long)` - `a2wParamLogTypes - Array of matching log types. (Array of Unsigned Short)` - `long* pnParamCount - Number of parameters retrieved.` - `long* pnChannelCount - Number of channels in config.` - `awSlots - Array of matching slot position of configuration defining the parameter. (Unsigned Short).` ### **GetRemoteSessionDetail** **Description:** Returns the current value of a specified session detail from the remote logging configuration. **Arguments:** - `BSTR szName - Name of session detail.` - `BSTR* pszValue - Returned value for session detail.` ### **GetRemoteTrigger** **Description:** Retrieves the properties of a given channel's trigger settings. **Arguments:** - `short wChannel - The 0 based channel index to retrieve.` - `long* pnStartPostTriggerMs - Retrieves the delay for the start trigger.` - `long* pnStopPostTriggerMs - Retrieves the delay for the stop trigger.` ### **GetRemoteTriggerAll** **Description:** Retrieves the properties of a given channel's trigger settings. **Arguments:** - `short wChannel - The 0 based channel index to retrieve.` - `long* pnStartPostTriggerMs - Retrieves the delay for the start trigger.` - `long* pnStopPostTriggerMs - Retrieves the delay for the stop trigger.` - `short* - Retrieves the slot position of the configuration defining the channel.` ### **GetRemoteTriggerCondition** **Description:** Retrieve a given channel's trigger condition. **Arguments:** - `short wChannel - The 0 based channel index to retrieve.` - `long nTriggerIndex - Specifies which condition  index (0 <= n < 6), where the first 3 are start conditions, and the later, stop conditions.` - `long* pnTriggerType - Retrieves the trigger condition type (see ECONDITONTYPE):` - `BSTR* pszParamName - Retrieves the parameter used in > On Data = condition type. If any.` - `short pwParamAppId - Retrieves the application ID, that the pszParamName belongs to.` - `long nTriggerCondition - Retrieves the condition operator of the > On Data = type. (See ECOMPARISON) Where:` - `double* pfTriggerThreshold - Retrieves the floating point number to compare in condition against specified parameter for type > On Data =.` - `long* pnTriggerCount - Retrieves the number of times a condition should occur before it is considered true.` ### **GetRemoteTriggerConditionAll** **Description:** Retrieve a given channel's trigger condition. **Arguments:** - `short wChannel - The 0 based channel index to retrieve.` - `long nTriggerIndex - Specifies which condition  index (0 <= n < 6), where the first 3 are start conditions, and the later, stop conditions.` - `long* pnTriggerType - Retrieves the trigger condition type (see ECONDITONTYPE):` - `BSTR* pszParamName - Retrieves the parameter used in > On Data = condition type. If any.` - `short pwParamAppId - Retrieves the application ID, that the pszParamName belongs to.` - `long nTriggerCondition - Retrieves the condition operator of the > On Data = type. (See ECOMPARISON) Where:` - `double* pfTriggerThreshold - Retrieves the floating point number to compare in condition against specified parameter for type > On Data =.` - `long* pnTriggerCount - Retrieves the number of times a condition should occur before it is considered true.` - `short* - Retrieves the slot position of the configuration defining the channel.` ### **GetRemoteWrap** **Description:** Gets the remote logging config memory wrapping strategy. **Arguments:** - `pbWrap- Output parameter to receive the current logging memory wrapping strategy.` - `Error messages:SMAPI_ERR_BAD_STATE (-30) if logging config document not present.` ### **GetSlotPercentage** **Description:** Gets the percentage of a logging slot used by a parameter (As Long) **Arguments:** - `INT wAppId - Application Id.` - `STRING szIdentifier - Filename and full path.` - `DOUBLE pdPercentage - Output parameter to receive the current percentage` ### **RemoveRemoteParam** **Description:** Remove a parameter from the Remote Logging Configuration. **Arguments:** - `short wAppId – Application Id.` - `LPCTSTR pszParamId – Unique parameter identifier` ### **SetRemoteChannelProperties** **Description:** Alters the properties of any given Remote Logging channel. **Arguments:** - `short wChannel - The 0 based channel index to modify.` - `BSTR strChannelName - New name of the channel.` - `boolean bLogToUnit - Alters the Log to Unit flag.` - `boolean bTelemetry - Alters the telemetry flag.` - `boolean bTriggerRearmsAfterStopped - Trigger re-arms after stop condition.` ### **SetRemoteOffset** **Description:** Sets the remote logging config offset. **Arguments:** - `wOffset- Required logging offset.` - `Error messages:SMAPI_ERR_BAD_STATE (-30) if logging config document not present.` - `SMAPI_ERR_INVALID_CMD (-31) if Logging offset adjustment not allowed.` ### **SetRemoteSessionDetail** **Description:** Sets a specified session detail in the remote logging configuration. **Arguments:** - `BSTR szName - Name of session detail.` - `BSTR szValue - New value for session detail.` ### **SetRemoteTrigger** **Description:** Alters the properties of a given channel's trigger settings.. **Arguments:** - `short wChannel - The 0 based channel index to modify.` - `long nStartPostTriggerMs - Set the delay for the start trigger.` - `long nStopPostTriggerMs - Set the delay for the stop trigger.` ### **SetRemoteTriggerCondition** **Description:** Modify or create a given channel's trigger condition. **Arguments:** - `short wChannel - The 0 based channel index to set.` - `long nTriggerIndex - Specifies which condition  index (0 <= n < 6), where the first 3 are start conditions, and the later, stop conditions.` - `long nTriggerType - Sets the trigger condition type (see ECONDITONTYPE):` - `BSTR pszParamName - Sets the parameter used in >On Data= condition type.` - `short wParamAppId - Sets the Application Id, that the pszParamName belongs to.` - `long nTriggerCondition - Sets the condition operator of the >On Data= type (see ECOMPARISON), where:` - `double fTriggerThreshold - Sets the floating point number to compare in condition against specified parameter for type > On Data =.` - `long nTriggerCount - Sets the number of times a condition should occur before it is considered true.` ### **SetRemoteWrap** **Description:** Sets the remote logging config memory wrapping strategy. **Arguments:** - `bWrap- Required wrapping memory stategy.` - `Error messages:SMAPI_ERR_BAD_STATE (-30) if logging config document not present.` ### **UploadRemote** **Description:** Uploads the remote logging configuration from the ECU. **Arguments:** - None ## Parameters ### **CANBuffersExport** **Description:** Exports the CAN Buffer configuration csv from the given CAN Bus index. **Arguments:** - `short wIndex - Bus Index.` - `BSTR pszFilename - Name and path of .csv to create.` ### **CANBuffersImport** **Description:** Imports the CAN Buffer configuration csv into the given CAN Bus index. **Arguments:** - `short wIndex - Bus Index.` - `BSTR pszFilename - Name and path of .csv to import.` ### **CANMessagesExport** **Description:** Exports the CAN Message configuration csv from the given CAN Bus index. **Arguments:** - `short wIndex - Bus Index.` - `BSTR pszFilename - Name and path of .csv to create.` ### **CANMessagesImport** **Description:** Imports the CAN Message configuration csv into the given CAN Bus index. **Arguments:** - `short wIndex - Bus Index.` - `BSTR pszFilename - Name and path of .csv.` - `bool bMerge - Merge with existing messages.` ### **GetCANParameter** **Description:** Retrieve a selected CAN Parameter. **Arguments:** - `BSTR rszParamId - The unique identifier of the CAN Parameter to receive.` - `BSTR *rszName - Receives the display name of the parameter.` - `BSTR *rszDescription - Receives the description of the parameter.` - `double * pdwMaxDisplay - Receives the maximum value.` - `double * pdwMinDisplay - Receives the minimum value.` - `int * pnMinLoggingRate - Receives the minimum logging rate.` - `int * pnScalingFactor - Receives the scalling factor (see EScalingUnit).` - `boolean * pbIsMinNotDef - Receives state of min/max values.` - `BSTR * rszConvId - Receives the unique identifier of the conversion rule used in this parameter.` - `boolean * pbIsRx - Receives a flag to indicate Rx / Tx status of the parameter.` ### **GetCANParameterBatch** **Description:** Retrieve CAN parameter details for the given list of parameters. **Arguments:** - `VARIANT aszIdentifiers – Array (string) of CAN parameter identifiers.` - `VARIANT* azNames -  Array of parameter display names.` - `VARIANT* astrDescriptions - Array of parameter descriptions.` - `VARIANT* adMaxDisplays - Array of parameter maximum values ( doubles ).` - `VARIANT* adMinDisplays - Array of parameter minimum values ( doubles ).` - `VARIANT* adwMinLoggingRates - Array of parameter minimum logging rates ( unsigned integers ).` - `VARIANT* awScalingFactor - Array of parameter scaling factors ( unsigned shorts, see EScalingUnit ).` - `VARIANT* abIsMinNotDefs - Array of parameter state of min/max values ( boolean ).` - `VARIANT* astrConvId - Array of parameter unique identifiers of the conversion rules.` - `VARIANT* pbIsRxs - Array of parameter flags to indicate Rx / Tx status ( boolean ).` - `VARIANT* aszDataTypes - Array of parameter data types.` - `VARIANT* pszCANBus - Array of parameter CAN Bus names.` - `VARIANT* pszCANMsg - Array of parameter CAN Message names.` - `VARIANT* pwStartBit - Array of CAN parameter start bits ( unsigned integers ).` - `VARIANT* pwBitLength - Array of CAN parameter bit lengths ( unsogned integers ).` - `VARIANT* pdGain - Array of CAN parameter gain values (doubles ).` - `VARIANT* pdOffset - Array of CAN parameter offset values (doubles ).` - `VARIANT* pwByteOrder - Array of CAN parameter byte orders (unsigned shorts ).` - `VARIANT* pszMuxId -  Array of CAN parameter multiplexer ids. Blank if parameter is not in a multiplexed message..` - `VARIANT* palErrorCodes - Array of error codes ( integers ).` ### **GetCANParameterEx** **Description:** Retrieve a selected CAN Parameter. **Arguments:** - `BSTR szParamId - The unique identifier of the CAN Parameter to receive.` - `BSTR* pszName - Receives the display name of the parameter.` - `BSTR* pszDescription - Receives the description of the parameter.` - `double* pdwMaxDisplay - Receives the maximum value.` - `double* pdwMinDisplay - Receives the minimum value.` - `unsigned int* pdwMinLoggingRate - Receives the minimum logging rate.` - `unsigned short* pwScalingFactor - Receives the scalling factor (see EScalingUnit).` - `boolean* pbIsMinNotDef - Receives state of min/max values.` - `BSTR* pszConvId - Receives the unique identifier of the conversion rule used in this parameter.` - `boolean* pbIsRx - Receives a flag to indicate Rx / Tx status of the parameter.` - `BSTR* pszDataType - Receives the data type of the parameter.` - `BSTR* pszCANBus - Receives the CAN Bus name containing the parameter.` - `BSTR* pszCANMsg - Receives the CAN Message name containing the parameter.` - `unsigned short* pwStartBit - Receives the start bit of the parameter within the message.` - `unsigned short* pwBitLength - Receives the bit length of the CAN parameter.` - `double* pdGain - Receives the gain of the CAN parameter.` - `double* pdOffset - Receives the offset of the CAN parameter.` - `unsigned short* pwByteOrder - Receives the byte order  of the CAN parameter.` - `BSTR* pszMuxId -  Receives the multiplexer id of the CAN parameter. Blank if parameter is not in a multiplexed message..` ### **GetCANParameterList** **Description:** Return a list of identifiers for all CAN parameters. **Arguments:** - `VARIANT FAR* paParams - Variant array of parameter identifiers.` ### **GetCANParameterListEx** **Description:** Return a list of CAN Parameter identifiers, names and count in the current project. **Arguments:** - `paParameterIds` - `paParameterNames` - `pnParameterCount As Long` ### **GetValueCAN** **Description:** Retrieves the value of the specified can parameter. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `double FAR* pdResult - Value of the CAN parameter.` ### **RegenerateInputSignalParameters** **Description:** Regenerate Input Signal parameters from RAW values. **Arguments:** - `None.` - `.` ### **ChangeSensorSerialNumber** **Description:** Changes the serial number of the specified sensor. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR pszIdentifier - Sensor identifier.` - `LONG dwSerialNumber - Serial number.` ### **GetAppsHoldingControlParameter** **Description:** Returns the application(s) which contain the specified non measurement parameter. **Arguments:** - `BSTR szParamIdent - Parameter identifier.` - `VARIANT FAR * pawAppIds - Applications containing the specified parameter.` ### **GetAppsHolding** **Description:** Returns the application(s) which contain the specified measurement parameter. **Arguments:** - `BSTR szParamIdent - Parameter identifier.` - `VARIANT FAR * pawAppIds - Applications containing the specified parameter.` ### **GetAppsHoldingParam** **Description:** Returns the application(s) which contain the specified parameter. **Arguments:** - `BSTR szParamIdent - Parameter identifier.` - `short* pwAppsMask - Applications containing the specified parameter.` ### **GetAppsHoldingParamEx** **Description:** Returns the application(s) which contain the specified parameter. **Arguments:** - `BSTR szParamIdent - Parameter identifier.` - `VARIANT FAR * pawAppIds - Applications containing the specified parameter.` ### **GetAxisAddress** **Description:** Retrieves the address of the parameter. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Parameter identifier.` - `long* plAddress - Address returned` ### **GetAxisParameterFromMap** **Description:** Retrieves a list of axis parameter(s) used by the map parameter in the application. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Map parameter identifier.` - `VARIANT* str1xParameters - Safe Array of axis parameter(s) used by the map parameter.` ### **GetDTVValue1AxisMap** **Description:** Returns the values of a specified 1-axis map (CURVE) editable parameter at all axis points from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* Value - 1-d array of doubles holding current values.` ### **GetDTVValue1AxisMapBatch** **Description:** Returns a list of values from a specified list of  1-axis maps (CURVE) editable parameters at all axis points from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetDTVValue2AxisMap** **Description:** Returns the values of a specified 2-axis map (MAP) editable parameter at all axis points from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* Value - 2-d array of doubles holding current values.` ### **GetDTVValue2AxisMapBatch** **Description:** Returns a list of values from a specified list of  2-axis maps (MAP) editable parameters at all axis points from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* Value - 2-d array of doubles holding current values.` - `VARIANT* pafValues - 3-d array of doubles holding lists of parameters arrays.                                 Note: The first value in each row and column contains the number of values in the list.` ### **GetDTVValueArray** **Description:** Returns the values of a specified array (VAL_BLK) editable (characteristic) parameter at all index positions from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* Value - 1-d array of doubles holding current values.` ### **GetDTVValueArrayBatch** **Description:** Returns a list of values from a specified list of arrays (VAL_BLK) editable parameters at all index positions from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetDTVValueAxis** **Description:** Returns the values of a specified axis (AXIS_PTS) editable parameter at all axis points from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* Value - 2-d array of doubles holding current values.` ### **GetDTVValueAxisBatch** **Description:** Returns a list of values from a specified list of axis (AXIS_PTS editable parameters at all axis points from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetDTVValueScalar** **Description:** Returns the value of a specified scalar (VALUE) editable parameter from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `double* pdResult - Parameters current value.` ### **GetDTVValueScalarBatch** **Description:** Returns a list of values from a specified list of scalar (VALUE) editable parameters from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* pafValues - List of parameters values.` ### **GetDTVValueString** **Description:** Returns the value of a specified string editable parameter from the supplied DTV. **Arguments:** - `BSTR szIdentifier - Parameter identifier.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `BSTR FAR* pszValue - Returned value.` ### **GetDTVValueStringBatch** **Description:** Returns a list of values from a specified list of string editable parameters from the supplied DTV. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `BSTR szDTV- Filename and full path of the DTV to retrieve value from.` - `VARIANT* paszValues - List of parameters strings.` ### **GetECUMeasurementAddress** **Description:** Retrieves the address of the parameter. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Parameter identifier.` - `long* plAddress - Address returned` - `BYTE* plIdent - Ident returned` ### **GetExternalInputGainOffset** **Description:** Gets the Gain & Offset of an External Input parameter. **Arguments:** - `BSTR pszParamId - The unique identifier of the External Input parameter.` - `double *pdGain - Receives the value of the parameter gain.` - `double *pdOffset - Receives the value of the parameter offset.` ### **GetMapProperties** **Description:** Return common attributes of a given map parameter. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - parameter identifier.` - `LPBSTR pszXAxisId - Identifier of the x axis parameter.` - `LPBSTR pszYAxisId - Identifier of the y axis parameter.` - `short* psXAxisPts - Number on breakpoints in the x axis.` - `short* psYAxisPts - Number on breakpoints in the y axis.` ### **GetModifiedParametersList** **Description:** Return a list of string IDs and names for the modified parameters in a selected application. **Arguments:** - `short wAppId - Application Id.` - `VARIANT* str1xArrParameterIds - Variant wrapped SAFEARRAY of strings containing the unique parameter identifiers.` - `VARIANT* str1xArrParameterNames - Variant wrapped SAFEARRAY of strings containing the matching parameter names.` - `long* pnParameterCount - Returns the number of parameters in the returned SAFEARRAYS.` ### **GetParameterBitMask** **Description:** Retrieves a bit mask of a given parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `long* plBitMask - Receives the parameter bit mask.` ### **GetParameterBitShift** **Description:** Retrieves a bit shift of a given parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `int* plBitShift - Receives the parameter bit shift.` ### **GetParameterByteOrder** **Description:** Retrieves a bit shift of a given parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `int* plByteOrder - Receives the parameter byte order.` - `0: MSB First 1: MSB Last 2: Undefined` ### **GetParameterList** **Description:** Return a list of string IDs and names for the parameters in a selected application. **Arguments:** - `short wAppId - Application Id.` - `BSTR* pszAppName - String representation of the selected application.` - `VARIANT* str1xArrParameterIds - Variant wrapped SAFEARRAY of strings containing the unique parameter identifiers.` - `VARIANT* str1xArrParameterNames - Variant wrapped SAFEARRAY of strings containing the matching parameter names.` - `long* pnParameterCount - Returns the number of parameters in the returned SAFEARRAYS.` ### **GetParameterListByType** **Description:** Return a list of string IDs and names of the parameters in a selected application for a specific parameter type. **Arguments:** - `short wAppId - Application Id.` - `BSTR* pszParamType - Type of parameters to return.  Valid inputs are MEAS, SCALAR, AXIS, 1AXIS, 2AXIS, ARRAY, STRING.` - `BSTR* pszAppName - String representation of the selected application.` - `VARIANT* str1xArrParameterIds - Variant wrapped SAFEARRAY of strings containing the unique parameter identifiers.` - `VARIANT* str1xArrParameterNames - Variant wrapped SAFEARRAY of strings containing the matching parameter names.` - `long* pnParameterCount - Returns the number of parameters in the returned SAFEARRAYS.` ### **GetParameterProperties** **Description:** Return common attributes of a given parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `BSTR* pszUnits - Receives the parameters units.` - `double* pdLowerEngLimit - Receives the lower engineering limit of the parameter.` - `double* pdUpperEngLimit - Receives the upper engineering limit of the parameter.` - `BSTR* pszFormat - Receives the display format of the parameter.` - `BSTR* pszConversionId - Receives the conversion id of the parameter (if available).` - `BSTR* pszDescription - Receives the readable description/comment of the parameter` ### **GetParameterPropertiesByApp** **Description:** Return common attributes of all parameters in the specified application. **Arguments:** - `short wAppId - Application Id.` - `VARIANT* aszIdentifier - Array of unique parameter identifiers.` - `VARIANT* aszName - Array of parameter names.` - `VARIANT* aszDescription - Array of parameter descriptions.` - `VARIANT* aszParamType - Array of parameter types.` - `VARIANT* aszUnits - Array of units.` - `VARIANT* aszFormats - Array of parameter formats.` - `VARIANT* aszConversionId - Array of conversion Ids.` - `VARIANT* aszGroups - Array of parameter group arrays.` - `VARIANT* aszDataType - Array of parameter data types.` - `VARIANT* avwDataSize - Array of parameter data sizes.` - `VARIANT* avdLowerEngLimit - Array of lower engineering limits.` - `VARIANT* avdUpperEngLimit - Array of upper engineering limits.` - `VARIANT* avdwMaxLoggingRate - Array of maximum logging rates.` - `VARIANT* avbPrime - Array of prime flags.` - `VARIANT* avbReadOnly - Array of read only flags.` - `VARIANT* avbLiveTuneable - Array of live tuneable flags.` ### **GetParameterPropertiesByAppandType** **Description:** Return common attributes of all parameters of a specified type in the specified application . **Arguments:** - `short wAppId - Application Id.` - `BSTR pszParamType - Type of parameters to return.  Valid inputs are MEAS, SCALAR, AXIS, 1AXIS, 2AXIS, ARRAY, STRING.` - `VARIANT* aszIdentifier - Array of unique parameter identifiers.` - `VARIANT* aszName - Array of parameter names.` - `VARIANT* aszDescription - Array of parameter descriptions.` - `VARIANT* aszUnits - Array of units` - `VARIANT* aszFormats - Array of parameter formats` - `VARIANT* aszConversionId - Array of conversion IDs` - `VARIANT* aszGroups - Array of parameter group arrays` - `VARIANT* aszDataType - Array of parameter data types` - `VARIANT* avwDataSize - Array of parameter data sizes` - `VARIANT* avdLowerEngLimit - Array of lower engineering limits` - `VARIANT* avdUpperEngLimit - Array of upper engineering limits` - `VARIANT* avdwMaxLoggingRate - Array of maximum logging rates` - `VARIANT* avbPrime - Array of prime flags` - `VARIANT* avbReadOnly - Array of read only flags` - `VARIANT* avbLiveTuneable - Array of live tuneable flags` ### **GetParameterPropertiesByType** **Description:** Return common attributes of a given parameter of a specified type. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `BSTR pszParamType - Type of parameters to return.  Valid inputs are MEAS, SCALAR, AXIS, 1AXIS, 2AXIS, ARRAY, STRING.` - `BSTR* pszUnits - Receives the parameters units.` - `double* pdLowerEngLimit - Receives the lower engineering limit of the parameter.` - `double* pdUpperEngLimit Receives the upper engineering limit of the parameter.` - `BSTR* pszFormat - Receives the display format of the parameter.` - `BSTR* pszConversionId - Receives the conversion id of the parameter (if available).` - `BSTR* pszDescription - Receives the readable description/comment of the parameter` ### **GetParameterPropertiesEx** **Description:** Get method for retrieving extended parameter properties. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Parameter identifier.` - `LPBSTR pszName` - `LPBSTR pszParamType - SCALAR, MEAS etc.` - `VARIANT* pszGroups - Array of group name strings.` - `LPBSTR pszDataType` - `short* psDataSize` - `long* plMaxLoggingRate` - `VARIANT_BOOL FAR* pbPrime` - `VARIANT_BOOL FAR* pbReadOnly` - `VARIANT_BOOL FAR* pbLiveTuneable` ### **GetParameterPropertiesExByType** **Description:** Get method for retrieving extended parameter properties based on a specific parameter type. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Parameter identifier.` - `LPBSTR pszParamType - Type of parameters to return.  Valid inputs are:   MEAS,   SCALAR,   AXIS,   1AXIS,   2AXIS,   ARRAY,   STRING.` - `LPBSTR pszName` - `LPBSTR pszParamType - SCALAR, MEAS etc.` - `VARIANT* pszGroups - Array of group name strings.` - `LPBSTR pszDataType` - `short* psDataSize` - `long* plMaxLoggingRate` - `VARIANT_BOOL FAR* pbPrime` - `VARIANT_BOOL FAR* pbReadOnly` - `VARIANT_BOOL FAR* pbLiveTuneable` ### **GetParametersAndGroups** **Description:** Returns a list of all parameters and the group descriptions in the application. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT* str1xIdentifiers - Safe Array of parameter identifiers.` - `VARIANT* str1xGroups - Safe Array of group descriptions for each parameter.` ### **GetParameterWarningLimits** **Description:** Gets the warning limits from a parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `double* pdLow - Lower display warning limit.` - `double* pdHigh - Upper display warning limit.` ### **GetRowDetails** **Description:** Retrieves the row detailsof a given parmeter. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `short* pnId - Receives the parameter Id.` - `long* plIdentOffset - Receives the parameter ident offset.` ### **GetScalarAddress** **Description:** Retrieves the address of the parameter. **Arguments:** - `short wAppId - Application Id.` - `LPCTSTR pszIdentifier - Parameter identifier.` - `long* plAddress - Address returned` ### **GetStringValue1AxisMap** **Description:** Returns the current values of a specified 1-axis map (CURVE) Editable Parameter at all axis points as formatted strings **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of BSTRSs holding current values.` ### **GetStringValue2AxisMap** **Description:** Returns the current values of a specified 2-axis map (MAP) Editable Parameter at all axis points as formatted strings. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 2-d array of BSTRSs holding current values.` ### **GetStringValueArray** **Description:** Returns the current values of a specified array (VAL_BLK) Editable (characteristic) Parameter at all index positions as formatted text values **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of BSTRs holding current values.` ### **GetStringValueAxis** **Description:** Returns the current values of a specified axis (AXIS_PTS) Editable Parameter at all axis points as formatted text values. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of BSTRs holding current values.` ### **GetStringValueMeasurement** **Description:** Returns the current value of a specified measurement parameter as a formatted string. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `BSTR* pszResult - Parameters current value.` ### **GetValue1AxisMap** **Description:** Returns the current values of a specified 1-axis map (CURVE) editable parameter at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of doubles holding current values.` ### **GetValue1AxisMapBatch** **Description:** Returns the current values of a specified  list of 1-axis map (CURVE) editable parameters at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetValue1AxisMapBatchEx** **Description:** Returns the current values of a specified  list of 1-axis map (CURVE) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValue2AxisMap** **Description:** Returns the current values of a specified 2-axis map (MAP) editable parameter at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 2-d array of doubles holding current values.` ### **GetValue2AxisMapBatch** **Description:** Returns the current values of a specified  list of 2-axis map (MAP) editable parameters at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 3-d array of doubles holding lists of parameters arrays.                                 Note: The first value in each row and column contains the number of values in the list.` ### **GetValue2AxisMapBatchEx** **Description:** Returns the current values of a specified  list of 2-axis map (MAP) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 3-d array of doubles holding lists of parameters arrays.                                 Note: The first value in each row and column contains the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueArray** **Description:** Returns the current values of a specified array (VAL_BLK) editable (characteristic) parameter at all index positions. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of doubles holding current values.` ### **GetValueArrayBatch** **Description:** Returns the current values of a specified  list of array (VAL_BLK) editable (characteristic) parameters at all index positions. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetValueArrayBatchEx** **Description:** Returns the current values of a specified  list of array (VAL_BLK) editable (characteristic) parameters at all index positions with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueAxis** **Description:** Returns the current values of a specified axis (AXIS_PTS) editable parameter at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT* Value - 1-d array of doubles holding current values.` ### **GetValueAxisBatch** **Description:** Returns the current values of a specified  list of axis (AXIS_PTS) editable parameters at all axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` ### **GetValueAxisBatchEx** **Description:** Returns the current values of a specified  list of axis (AXIS_PTS) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list contains the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueCANBatch** **Description:** Returns the current value of a specified  list of CAN parameters with an error code for each parameter. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueMeasurement** **Description:** Returns the current value of a specified measurement parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `long* bAddToDaq - (UNUSED PARAM).` - `double* pdResult - Parameters current value.` ### **GetValueMeasurementBatch** **Description:** Returns the current value of a specified  list of measurement parameters. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` ### **GetValueMeasurementBatchEx** **Description:** Returns the current value of a specified  list of measurement parameters with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueOffset** **Description:** Retrieve the live value offset for a parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR szParamIdent - Parameter identifier.` - `double* pdOffset - Live offset value` ### **GetValueScalar** **Description:** Returns the current value of a specified scalar (VALUE) editable parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `double* pdResult - Parameters current value.` ### **GetValueScalarBatch** **Description:** Returns a list of values from a specified list of scalar (VALUE) editable parameters. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` ### **GetValueScalarBatchEx** **Description:** Returns a list of values from a specified list of scalar (VALUE) editable parameters with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueString** **Description:** Returns the current value of a specified string editable parameter **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `BSTR FAR* pszValue - Returned value.` ### **GetValueStringBatch** **Description:** Returns a list of values from a specified list of string editable parameters. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* paszValues - List of parameters strings.` ### **GetValueStringBatchEx** **Description:** Returns a list of values from a specified list of string editable parameters with an error code for each parameter.. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* paszValues - List of parameters strings.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetValueVirtualBatch** **Description:** Returns the current value of a specified  list of virtual parameters with an error code for each parameter. **Arguments:** - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT* pafValues - List of parameters values.` - `VARIANT* palErrorCodes - List of error codes.` ### **IsParameterLoggable** **Description:** Returns the loggable property of a measurement parameter **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `BOOL FAR* pbResult - Returned value.` ### **ParameterExists** **Description:** Identifies if a given parameter exists. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `bool* pbResult` ### **ParameterExistsEx** **Description:** Identifies if a parameter of the given type exists. **Arguments:** - `short wAppId - Application Id.` - `BSTR pszIdentifier - Unique parameter identifier.` - `BSTR pszParamType - One of 'MEAS', 'SCALAR', 1AXIS', '2AXIS', 'ARRAY', 'STRING', 'VIRTUAL' or 'CAN'` - `bool* pbResult` ### **RestoreValue** **Description:** Restore the parameter value to original saved value. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` ### **SetBatchMode** **Description:** Call this to enter (or leave) batch mode. This is used when creating virtual parameters; entering batch mode before the creation of multiple Virtual Parameters and leaving batch mode afterwards. Using this method, all created/updated Virtual Parameters will only be available in System Monitor after leaving batch mode. **Arguments:** - `boolean bEnter - Enter/leave batch mode, true or false respectively.` ### **SetExternalInputGainOffset** **Description:** Sets the Gain & Offset of an External Input parameter. **Arguments:** - `BSTR pszParamId - The unique identifier of the External Input parameter.` - `double dGain - The value to set the Gain.` - `double dOffset - The value to set the Offset.` ### **SetParameterWarningLimits** **Description:** Sets the warning limits on a parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `double dLow - Lower display warning limit.` - `double dHigh - Upper display warning limit.` ### **SetStringValue1AxisMap** **Description:** Sets the current value of a specified 1-axis map (CURVE) Editable Parameter at one or more axis points from an array of string values **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of BSTRs holding new values.` ### **SetStringValue2AxisMap** **Description:** Sets the current value of a specified 2-axis map (MAP) Editable Parameter at one or more axis points from an array of formatted strings. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 2-d array of BSTRs holding new values.` ### **SetStringValueArray** **Description:** Sets the current value of a specified array (VAL_BLK) Parameter at one or more index positions from an array of strings. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of BSTRs holding new values.` ### **SetStringValueAxis** **Description:** Sets the current value of a specified axis (AXIS_PTS) Editable Parameter at one or more axis points from an array of strings. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of BSTRs holding new values.` ### **SetStringValueScalar** **Description:** Sets the current value of a specified scalar (VALUE) Editable Parameter from a string. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `BSTR szValue - New value.` ### **SetValueScalarBatch** **Description:** Set values for a specified list of scalar (VALUE) editable parameters with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT afValues - List of parameters values.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValue1AxisMapBatch** **Description:** Set values for a specified list of 1-axis map (CURVE) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT afValues - 2-d array of doubles holding lists of parameters values.                               Note: The first value in each list should contain the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValue2AxisMapBatch** **Description:** Set values for a specified list of 2-axis map (MAP) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT afValues - 3-d array of doubles holding lists of parameters arrays.                               Note: The first value in each row and column should contain the number of values in row/column.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValueArrayBatch** **Description:** Set values for a specified list of  array (VAL_BLK) editable (characteristic) parameters at all index positions with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT afValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list should contain the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValueAxisBatch** **Description:** Set values for a specified list of axis (AXIS_PTS) editable parameters at all axis points with an error code for each parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT afValues - 2-d array of doubles holding lists of parameters values.                                  Note: The first value in each list should contain the number of values in the list.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValueStringBatch** **Description:** Set values for a specified list of string editable parameters with an error code for each parameter.. **Arguments:** - `unsigned short wAppId - Application Id.` - `VARIANT aszIdentifiers - List of parameter identifiers.` - `VARIANT aszValues - List of parameters strings.` - `VARIANT* palErrorCodes - List of error codes.` ### **SetValue1AxisMap** **Description:** Sets the current value of a specified 1-axis map (CURVE) editable parameter at one or more axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of doubles holding new values.` ### **SetValue2AxisMap** **Description:** Sets the current value of a specified 2-axis map (MAP) editable parameter at one or more axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 2-d array of doubles holding new values.` ### **SetValueArray** **Description:** Sets the current value of a specified array (VAL_BLK) parameter at one or more index positions. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of doubles holding new values.` ### **SetValueAxis** **Description:** Sets the current value of a specified axis (AXIS_PTS) editable parameter at one or more axis points. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `VARIANT Value - 1-d array of doubles holding new values.` ### **SetValueOffset** **Description:** Sets the live value offset for a parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR szParamIdent - Parameter identifier.` - `double pdOffset - Live offset value.` ### **SetValueScalar** **Description:** Sets the current value of a specified scalar (VALUE) editable parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `double dValue - New value.` ### **SetValueString** **Description:** Sets the current value of a specified string editable parameter. **Arguments:** - `unsigned short wAppId - Application Id.` - `BSTR szIdentifier - Parameter identifier.` - `LPCTSTR pszValue - New value.` ### **ZeroLiveValue** **Description:** Zero's the live value by applying an offset to a measurement parameter. **Arguments:** - `short wAppId - Application Id.` - `BSTR szParamIdent - Parameter identifier.` ### **AddVirtualParameterGroup** **Description:** Adds a Virtual Parameter Group **Arguments:** - `LPCTSTR szPath - The path where the groups is to be added, e.g. \\TopLevelGroup\\SubGroup.` - `LPCTSTR szName - The display name for the virtual parameter .` - `LPCTSTR szDescription - Description of the virtual parameter.` - `BOOL bParamsReadOnly - If set to true, the virtual parameter group will be marked as a Read Only.` ### **GetValueVirtual** **Description:** Retrieve a selected Virtual Parameter. **Arguments:** - `BSTR pszIdentifier - Virtual parameter identifier` - `double* pdResult - Value of the virtual parameter` ### **GetVirtualParamCount** **Description:** Retrieve the number of Virtual Parameters in the current project. **Arguments:** - `long* pnParamCount - The number of virtual parameters in the active project.` ### **GetVirtualParamCountGroup** **Description:** Retrieves the number of virtual parameters in a group. **Arguments:** - `LPCTSTR szPath -  The path of group that contains the parameters, e.g. \\TopLevelGroup\\SubGroup.` - `long* pnParamCount - The number of virtual parameters in the group.` ### **GetVirtualParameter** **Description:** Retrieve a selected Virtual Parameter. **Arguments:** - `BSTR rszParamId - The unique identifier of the Virtual Parameter to receive.` - `BSTR *rszName - Receives the display name of the parameter.` - `BSTR *rszDescription - Receives the description of the parameter.` - `double * pdwMaxDisplay - Receives the maximum value.` - `double * pdwMinDisplay - Receives the minimum value.` - `int * pnMinLoggingRate - Receives the minimum logging rate.` - `int * pnScalingFactor - Receives the scalling factor (see EScalingUnit).` - `boolean * pbIsMinNotDef - Receives state of min/max values.` - `BSTR * rszExpression - Receives the expression used to evaluate the virtual parameter.` - `BSTR * rszConvId - Receives the unique identifier of the conversion rule used in this parameter.` ### **GetVirtualParameterBatch** **Description:** Retrieve virtual parameter details for the given list of parameters. **Arguments:** - `VARIANT aszIdentifiers – Safe Array (string) of virtual parameter identifiers.` - `VARIANT* astrNames - Safe Array of parameter display names.` - `VARIANT* astrDescription - Safe Array of parameter descriptions.` - `VARIANT* afMaxDisplay - Safe Array of parameter maximum values ( doubles ).` - `VARIANT* afMinDisplay - Safe Array of parameter minimum values ( doubles ).` - `VARIANT* anMinLoggingRate - Safe Array of parameter minimum logging rates ( unsigned integers ).` - `VARIANT* anScalingFactor - Safe Array of parameter scaling factors (see EScalingUnit).` - `VARIANT* abIsMinNotDef - Safe Array of parameter state of min/max values ( boolean ).` - `VARIANT* astrExpression - Safe Array of parameter expressions used to evaluate the virtual parameters.` - `VARIANT* astrConvId - Safe Array of parameter unique identifiers of the conversion rules.` - `VARIANT* astrUnits - Safe Array of parameter units.` - `VARIANT* astrFormat - Safe Array of parameter display formats.` - `VARIANT* astrGroup - Safe Array of parameter group names.` ### **GetVirtualParameterBatchEx** **Description:** Retrieve virtual parameter details for the given list of parameters. **Arguments:** - `VARIANT aszIdentifiers - Array (string) of virtual parameter identifiers.` - `VARIANT* aszNames - Array of parameter display names.` - `VARIANT* aszDescriptions - Array of parameter descriptions.` - `VARIANT* adMaxDisplays - Array of parameter maximum values ( doubles ).` - `VARIANT* adMinDisplays - Array of parameter minimum values ( doubles ).` - `VARIANT* adwMinLoggingRates - Array of parameter minimum logging rates ( unsigned integers ).` - `VARIANT* awScalingFactors - Array of parameter scaling factors ( unsigned shorts, see EScalingUnit ).` - `VARIANT* abIsMinNotDefs - Array of parameter state of min/max values ( boolean ).` - `VARIANT* aszExpressions - Array of parameter expressions used to evaluate the virtual parameters.` - `VARIANT* aszConvIds - Array of parameter unique identifiers of the conversion rules.` - `VARIANT* aszUnits - Array of parameter units.` - `VARIANT* aszFormat - Array of parameter display formats.` - `VARIANT* aszGroup - Array of parameter group names.` - `VARIANT* szDataTypes - Array of parameter data types.` - `VARIANT* palErrorCodes - List of error codes.` ### **GetVirtualParameterEx** **Description:** Retrieve a selected Virtual Parameter. **Arguments:** - `BSTR rszParamId - The unique identifier of the Virtual Parameter to receive.` - `BSTR *rszName - Receives the display name of the parameter.` - `BSTR *rszDescription - Receives the description of the parameter.` - `double * pdwMaxDisplay - Receives the maximum value.` - `double * pdwMinDisplay - Receives the minimum value.` - `int * pnMinLoggingRate - Receives the minimum logging rate.` - `int * pnScalingFactor - Receives the scalling factor (see EScalingUnit).` - `boolean * pbIsMinNotDef - Receives state of min/max values.` - `BSTR * rszExpression - Receives the expression used to evaluate the virtual parameter.` - `BSTR * rszConvId - Receives the unique identifier of the conversion rule used in this parameter.` - `BSTR * rszUnits - Receives the units of this parameter.` - `BSTR * rszFormat - Receives the display format used for this parameter.` - `BSTR * rszGroup - Receives the group name containing this parameter.` ### **GetVirtualParameterEx2** **Description:** Retrieve a selected Virtual Parameter. **Arguments:** - `BSTR szParamId - The unique identifier of the Virtual Parameter to receive.` - `BSTR* pszName - Receives the display name of the parameter.` - `BSTR* pszDescription - Receives the description of the parameter.` - `double* pdMaxDisplay - Receives the maximum value.` - `double* pdMinDisplay - Receives the minimum value.` - `unsigned int* pdwMinLoggingRate - Receives the minimum logging rate.` - `unsigned short* pwScalingFactor - Receives the scalling factor (see EScalingUnit).` - `boolean* pbIsMinNotDef - Receives state of min/max values.` - `BSTR* prszExpression - Receives the expression used to evaluate the virtual parameter.` - `BSTR* prszConvId - Receives the unique identifier of the conversion rule used in this parameter.` - `BSTR* prszUnits - Receives the units of this parameter.` - `BSTR* prszFormat - Receives the display format used for this parameter.` - `BSTR* prszGroup - Receives the group name containing this parameter.` - `BSTR* pszDataType - Receives the data type of this parameter.` ### **GetVirtualParameterGroup** **Description:** Get Virtual Parameter Group details **Arguments:** - `LPCTSTR szGroup - The identifier of the group to lookup.` - `BSTR* pszName - Receives the display name for the virtual parameter .` - `BSTR* pszDescription - Receives the description of the virtual parameter.` - `BOOL* pbParamsReadOnly - Receives the read only statusof the group.` ### **GetVirtualParameterGroupList** **Description:** Return a list of Virtual Parameter groups in the current project. **Arguments:** - `VARIANT FAR* paGroupPaths - Receives a list of paths of the groups, e.g. \\TopLevelGroup\\SubGroup.` - `Long* pnGroupCount  - Receives the total number of groups.` ### **GetVirtualParameterList** **Description:** Retrieve a list of the virtual parameters in the current project. **Arguments:** - `VARIANT FAR* paParams - VARIANT contained safe-array of string names.` ### **GetVirtualParameterListEx** **Description:** Return a list of Virtual Parameter identifiers, names and count in the current project. **Arguments:** - `VARIANT FAR* paParameterIds` - `VARIANT FAR* paParameterNames` - `Long* pnParameterCount` ### **GetVirtualParameterListGroup** **Description:** Retrieves a list of the virtual parameters in the group. **Arguments:** - `LPCTSTR szPath -  The path of group that contains the parameters, e.g. \\TopLevelGroup\\SubGroup.` - `paParams - VARIANT contains safe-array of parameter Identifiers.` ### **RemoveAllVirtualParameters** **Description:** Removes/deletes all Virtual Parameters in the active project. **Arguments:** ### **RemoveVirtualParameter** **Description:** Removes a selected Virtual Parameter **Arguments:** - `BSTR rszParamId - The unique identifier of the Virtual Parameter to remove.` ### **SetVirtualParameter** **Description:** Modify or Create a Virtual Parameter. **Arguments:** - `BSTR rszParamId - The unique identification of the virtual parameter.` - `BSTR rszName - The display name for the virtual parameter.` - `BSTR rszDescription - Description of the virtual parameter.` - `double dwMaxDisplay  - Maximum value.` - `double dwMinDisplay  - Minimum value.` - `int nMinLoggingRate - Minimum logging rate.` - `int nScalingFactor - See EScalingUnit in Enumeration Definitions.` - `boolean bIsMinNotDef - If set to true, the Virtual Parameter will be marked as a Prime Parameter.` - `BSTR rszExpression - The expression to evaluate in the Virtual Parameter.` - `BSTR rszConvId - The conversion rule identifier for converting the output value to appropriate unit values.` - `boolean bOverwrite - Overwrite existing data.` ### **SetVirtualParameterDataType** **Description:** Set the data type of the Virtual Parameter. **Arguments:** - `BSTR rszParamId - The unique identification of the virtual parameter.` - `enumDataType eDataType - The data type.` ### **SetVirtualParameterEx** **Description:** Set groups, units and formatting  for Virtual Parameters **Arguments:** - `LPCTSTR pszParamId - The unique identification of the virtual parameter.` - `LPCTSTR rszName - The display name for the virtual parameter.` - `LPCTSTR pszDescription - Description of the virtual parameter.` - `double dwMaxDisplay  - Maximum value.` - `double dwMinDisplay  - Minimum value.` - `int nMinLoggingRate - Minimum logging rate.` - `int nScalingFactor - See EScalingUnit in Enumeration Definitions.` - `boolean bIsMinNotDef - If set to true, the Virtual Parameter will be marked as a Prime Parameter.` - `LPCTSTR rszExpression - The expression to evaluate in the Virtual Parameter.` - `LPCTSTR pszConvId - The conversion rule identifier for converting the output value to appropriate unit values.` - `boolean bOverwrite - Overwrite existing data.` - `LPCTSTR szUnits - Sets the overriding units in which to display the virtual parameter’s value` - `LPCTSTR szFormatOverride - Sets the overriding format in which to display the virtual parameter’s value` ### **SetVirtualParameterExtended** **Description:** Adds a virtual parameter with all options. **Arguments:** - `LPCTSTR pszParamId - The unique identification of the virtual parameter.` - `LPCTSTR rszName - The display name for the virtual parameter.` - `LPCTSTR pszDescription - Description of the virtual parameter.` - `double dwMaxDisplay - Maximum value.` - `double dwMinDisplay - Minimum value.` - `int nMinLoggingRate - Minimum logging rate.` - `int nScalingFactor - See EScalingUnit in Enumeration Definitions` - `boolean bIsMinNotDef - If set to true, the virtual parameter will be marked as a Prime Parameter.` - `LPCTSTR rszExpression - The expression to evaluate in the virtual parameter.` - `LPCTSTR pszConvId - The conversion rule identifier for converting the output value to appropriate unit values.` - `boolean bOverwrite - Overwrites existing data.` - `LPCTSTR szUnits - Sets the overriding units in which to display the virtual parameter’s value` - `LPCTSTR szFormatOverride - Sets the overriding format in which to display the virtual parameter’s value` - `LPCTSTR szGroupPath - The path of the group in which to add the virtual parameter, e.g. \\TopLevelGroup\\SubGroup.` - `enumDataType eDataType - The data type.` - `double dwUpperWarningLimit - Upper warning limit value.` - `double dwLowerWarningLimit - Lower warning limit value.` ### **RegisterEnhancedRowParameters** **Description:** Add parameters from application to the enhanced speed row data list. **Arguments:** - `unsigned short wAppId - Application Id.` - `const VARIANT FAR& aszIdentifiers - List of parameters to add to enhanced row data list.` ### **RegisterCANEnhancedRowParameters** **Description:** Add CAN parameters to the enhanced speed row data list. **Arguments:** - `const VARIANT FAR& aszIdentifiers - List of CAN paramteres to add to enhanced row data list.` ### **RegisterVirtualEnhancedRowParameters** **Description:** Add virtual parameters to the enhanced speed row data list. **Arguments:** - `All parameters referenced by each virtual will be added to the list.` - `const VARIANT FAR& aszIdentifiers - List of virtual parameters to add to enhanced row data list..` ### **ClearEnhancedRowParameters** **Description:** Clar all parameters for given application from enhanced speed slow row data list. **Arguments:** - `unsigned short wAppId - Application Id.` - `NULL (0x0000) to clear all Parameters.` ### **ActivateEnhancedRowParameters** **Description:** Activate current enhanced speed row data list. **Arguments:** - `None.` ## Conversion Rules ### **GetConversions** **Description:** Returns a list of all conversions and the types defined in the application. **Arguments:** - `unsigned short wAppId - Application Id. NULL - (0x0000) for Virtual and CAN conversions.` - `VARIANT* str1xIdentifiers - Safe Array of conversion identifiers. ` - `VARIANT* str1xType - Safe Array of conversion types. ` ### **GetConversionType** **Description:** Returns the type of the specified conversion rule. **Arguments:** - `BSTR pszConversionID - Conversion identifier. ` - `BSTR* pszConversionType - Type of conversion. ` ### **GetConversionUse** **Description:** Returns a list of all parameters using the specified conversion in the application. **Arguments:** - `unsigned short wAppId - Application Id. NULL - (0x0000) for all applications including Virtual and CAN. ` - `BSTR pszConversionID - Conversion identifier. ` - `VARIANT* str1xIdentifiers - Safe Array of parameter identifiers. ` ### **GetFormulaConv** **Description:** Retrieves a selected Formula (Free) Conversion **Arguments:** - `BSTR pszConvId – Unique identifier of conversion. ` - `BSTR * pszComment – Human readable description. ` - `BSTR * pszFormat – Format string (e.g. “%1.3f”). ` - `BSTR * pszUnits – Unit value for output. ` - `BSTR * pszFormula – Formula value. ` - `BSTR * pszInverse – Inverse value. ` ### **GetRationalConv** **Description:** Retrieves a selected Rational Conversion **Arguments:** - `BSTR pszConvId – Unique identifier of conversion. ` - `double * pdPar1 – Coefficient value 1. ` - `double * pdPar2 – Coefficient value 2. ` - `double * pdPar3 – Coefficient value 3. ` - `double * pdPar4 – Coefficient value 4. ` - `double * pdPar5 – Coefficient value 5. ` - `double * pdPar6 – Coefficient value 6. ` - `BSTR * pszComment – Human readable description. ` - `BSTR * pszFormat – Format string (e.g. “%1.3f”). ` - `BSTR * pszUnits – Unit value for output. ` - `BSTR * pszDefault – Default value. ` --- # Error Codes ## Error Codes The following error codes are returned by API functions. Use the macros `SMAPI_SUCCEEDED(Status)` to check if a return code indicates success (>= 0), or `SMAPI_FAILED(Status)` to check for failure (< 0). ### General Errors | Code | Constant | Description | |------|----------|-------------| | 0 | `SMAPI_ERR_NONE` | No error | | -1 | `SMAPI_ERR_NOPROJECT` | No project loaded | | -2 | `SMAPI_ERR_NOLICENCE` | No licence for requested function | | -3 | `SMAPI_ERR_NONE_SPECIFIC` | Non-specific error | | -4 | `SMAPI_ERR_DATAVER_MISMATCH` | Data version mismatch | | -5 | `SMAPI_ERR_DATAVER_NONE` | No data version | | -6 | `SMAPI_ERR_PROGVER_NONE` | No program version | | -7 | `SMAPI_ERR_NOECU` | No ECU | | -8 | `SMAPI_ERR_FILE` | Invalid file specified | | -9 | `SMAPI_ERR_NOAPP` | The open project does not cover the given application | | -10 | `SMAPI_ERR_APP_INACTIVE` | Application must be active in order to support this operation | | -11 | `SMAPI_ERR_LIVEUPDATESON` | Operation not allowed while live updates are on | | -12 | `SMAPI_ERR_TAGONLY` | This command only valid for TAG systems | | -13 | `SMAPI_ERR_SM_BUSY` | SM cannot accept API calls at the moment (might be in a critical operation) | ### Message Argument Errors !!! note The range -20 to -36 is reserved for errors in the variant arguments supplied to the SendMessage call. | Code | Constant | Description | |------|----------|-------------| | -20 | `SMAPI_ERR_MSGARGMISMATCH` | Type mismatch in argument of SendMessage. Pass a word array by reference | | -21 | `SMAPI_ERR_MSGARGDIM` | Array dimension mismatch in argument of SendMessage. Pass a 1-d array | | -22 | `SMAPI_ERR_MSGLOWERBOUND` | Lower bound of SendMessage argument was not 0 | | -23 | `SMAPI_ERR_BOUNDERROR` | Unexpected error getting array bounds of SendMessage argument | | -24 | `SMAPI_ERR_MSGREPLYARG` | Error in ReplyMessage argument | | -25 | `SMAPI_ERR_MSGSENDARG` | Error in SendMessage argument | ### Parameter and State Errors | Code | Constant | Description | |------|----------|-------------| | -26 | `SMAPI_ERR_FDL_NOT_PARSED` | Virtual parameter contains invalid FDL | | -27 | `SMAPI_ERR_CONV_NOT_VALID` | Conversion for parameter does not exist/is invalid | | -28 | `SMAPI_ERR_PARAM_NOT_VALID` | Parameter name does not exist/is invalid | | -29 | `SMAPI_ERR_PARAM_EXISTS` | Parameter exists and not allowed to override | | -30 | `SMAPI_ERR_BAD_STATE` | Request cannot be actioned as incorrect SM state | | -31 | `SMAPI_ERR_INVALID_CMD` | Command passed to SM is not valid | | -32 | `SMAPI_ERR_NO_DATA_PRESENT` | Missing document or similar | | -33 | `SMAPI_ERR_BAD_MEM_ALLOC` | Memory allocation failed | | -34 | `SMAPI_ERR_PARTIALCOMPLETE` | Could not fully complete operation | | -35 | `SMAPI_ERR_DOCUMENTFULL` | Indicates the current document (config) is full | | -36 | `SMAPI_ERR_IDENT_EXISTS` | Parameter Identifier already exists in another application | | -37 | `SMAPI_ERR_READ_ONLY` | Parameter read only - write access denied | | -38 | `SMAPI_ERR_NON_LIVE_TUNEABLE` | Parameter non-livetuneable - write access denied | | -39 | `SMAPI_ERR_GROUP_NOTFOUND` | Requested group not found | | -40 | `SMAPI_ERR_FILE_NEEDS_SAVING` | Previous file needs saving before new one is opened | | -41 | `SMAPI_ERR_FREQ_OVERRIDDEN` | The frequency set was overridden to the minimum frequency of the prime parameter | | -42 | `SMAPI_ERR_NO_CUSTOMER_BASE` | No customer base found for the project being opened | | -100 | `SMAPI_ERR_PARAMNOTFOUND` | Requested parameter not found | ### Editable Parameter Errors !!! note The range -100 to -199 is reserved for errors returned by the parameter get/set value functions. | Code | Description | |------|-------------| | -100 | Specified parameter does not exist in current program version | | -101 | Tried to set a read-only parameter | | -102 | Tried to set a parameter outside its limits | | -103 | Axis monotony restrictions not adhered to | | -104 | Tried to get or set parameter or axis value at invalid breakpoint | | -105 | Parameter had invalid address | | -106 | String containing non-numeric characters supplied to set function | | -107 | Exceeded specified size of array or string parameter | | -108 | Live tune enabled, failed to update value in ECU | | -109 | Tried to get interpolated value but no input quantity or axis points | | -110 | No active live autotune "layer" (group) | | -111 | Operating point is outside breakpoint tolerances (e.g. for live auto tune) | | -112 | Number of axis breakpoints has changed between program versions | | -113 | Parameter is not modifiable in live tune area (warning only, not error) | | -114 | The attempt to validate the value in the unit failed (not the same as the value being invalid) | | -116 | Serial number not found for sensor channel | | -117 | An unknown error | | -118 | Attempt to set value cancelled (by user or because of invalid input) | | -119 | Access denied, parameter locked by RDA | | -120 | Value set does not match value in the unit | ### Data Acquisition Errors | Code | Constant | Description | |------|----------|-------------| | -200 | `SMAPI_ERR_DETAIL_UNKNOWN` | Session detail does not exist | | -201 | `SMAPI_ERR_DUMPROW` | Failed to dump row data | ### ECU Control Errors | Code | Constant | Description | |------|----------|-------------| | -300 | `SMAPI_ERR_LIVEUPDATE` | Failed to enter live tune | | -301 | `SMAPI_ERR_ONLINE` | Failed to go online | | -302 | `SMAPI_ERR_DOWNLOAD_DATA` | Failed to download data | ### OLE Errors | Code | Constant | Description | |------|----------|-------------| | -303 | `SMAPI_ERR_SYSMONNOTRUNNING` | System monitor is only running in OLE mode, hence can't perform requested action | | -304 | `SMAPI_ERR_PARAMETER_LOCKED` | Parameter locked | ### Communications Errors !!! note The range -10000 to -1000 is reserved for System Monitor comms, internal CA-card, ethernet or ECU errors which may be returned by a SendMessage call. | Code | Description | |------|-------------| | -5096 | There is a communication problem that is unknown | | -5097 | General timeout during communications | | -5098 | Link to ECU not OK | | -5099 | CA-Card code failed to load | | -5100 | Failed to open the CA-Card device driver | | -5101 | Failed to close the CA-Card device driver | | -5102 | DPC area is blank - CA-Card code is not running | | -5103 | DPR CaPc buffer is full - incoming messages have been lost | | -5104 | Driver Kernel buffer is full - incoming messages have been lost | | -5105 | The driver has timed out waiting for some data | | -5106 | There is no card in the machine | | -5107 | Receive buffer to DeviceIoControl is too small | | -5108 | The CA-Card code has been loaded but has crashed | | -5109 | IOCTL_CA125_CASEND not enough data returned | | -5110 | IOCTL_CA125_CAREAD not enough data returned | | -5111 | Attempt to send more data than the CA-Card said it can accept | | -5112 | The CA-Card has given more data than it said it would | | -5113 | Error response to MESSAGE_ID_SETUP_SESSION message | | -5114 | Error response to MESSAGE_ID_OPEN_SESSION message | | -5115 | Error response to MESSAGE_ID_CLOSE_SESSION message | | -5116 | Error response to MESSAGE_ID_SEND_SESSION message | | -5117 | Error response to MESSAGE_ID_RECEIVE_SESSION message | | -5121 | CA-Session in use | | -5123 | Failed to resolve the address for the session | | -5124 | User cancelled the operation | --- # Using SendMessage() to send a Read RAM message to the ECU ``` 'Declare globally available SM API. Dim SmApi As Object Sub Main 'Get access to System Monitor API Set SmApi = CreateObject("System Monitor API") 'Send Message must be fixed length array, size must match desired message length Dim aSend(2) As Integer 'Contents of message to send aSend(0) = &H0010 'COM_SM_READ_MEMORY and destination aSend(1) = 0 'Start address low aSend(2) = 0 'Start address high 'Reply message must be declared with ReDim – SM-V7 will dynamically allocate 'to match response size. Size used here is ignored, so can be 0 Dim aReply(0) As Integer Dim vaiantReply As Variant variantReply = aReply 'Timeout Dim Timeout As Integer Timeout = 400 'Retries Dim wTries As Integer wTries = 3 'Send the message Dim Err As Long Err = SmApi.SendMessageEx(App, aSend, variantReply, Timeout, wTries) 'Print error code Debug.Print "Error code = "; Debug.Print Hex(Err) 'Print contents of response. You can use UBound() to find size of response Debug.Print "Response = "; For i = 0 To UBound(aReply) Debug.Print Hex(aReply(i)); Debug.Print ", "; Next i 'Newline Debug.Print 'Pause to let user see response Wait 10 'You must free the SM Api before running another script Set SmApi = Nothing End Sub ``` --- # Getting and Setting the Value of a Scalar Editable Parameter ``` 'Declare globally available SM API. Dim SmApi As Object Sub Main 'Get access to System Monitor API Set SmApi = CreateObject("System Monitor API") 'Param to retrieve Dim szIdent As String szIdent = "BiosAC_Hiside1ScaledUpper" 'Current value Dim dValue As Double 'Error return code Dim lErr As Long 'Get current value of param lErr = SmApi.GetValueScalar(0, szIdent, dValue) Debug.Print szIdent Debug.Print " = " Debug.Print dValue 'Add on 10% and set value dValue = dValue + (dValue / 10) Debug.Print "Adding 10% to " Debug.Print szIdent lErr = SmApi.SetValueScalar(0, szIdent, dValue) 'Get current value lErr = SmApi.GetValueScalar(0, szIdent, dValue) Debug.Print szIdent Debug.Print " = " Debug.Print dValue 'Pause to let user see result Wait 10 'You must free the SM Api before running another script Set SmApi = Nothing End Sub ``` ### Sample Output ``` BiosAC_Hiside1ScaledUpper = 2.64 Adding 10% to BiosAC_Hiside1ScaledUpper BiosAC_Hiside1ScaledUpper = 2.904 ``` --- # Getting Values From a 1-Axis Map, Axis, Array and 2-Axis Map ``` Dim SmApi As Object Sub Main Set SmApi = CreateObject("System Monitor API") Dim szIdent As String Dim lErr As Long 'Array to hold values must be declared with ReDim SM V7 will dynamically 'alloc to match number of axis points. Size used here is ignored, so can be 0 Dim adValue As Variant 'Get current values of 1 axis map szIdent = "EC_1147" lErr = SmApi.GetValue1AxisMap(0, szIdent, adValue) PrintIdent(szIdent, lErr) PrintArray(adValue) 'Get current values of axis szIdent = "EC_1145" lErr = SmApi.GetValueAxis(0, szIdent, adValue) PrintIdent(szIdent, lErr) PrintArray(adValue) 'Get current values of array szIdent = "EG_26006" lErr = SmApi.GetValueArray(0, szIdent, adValue) PrintIdent(szIdent, lErr) PrintArray(adValue) ' 2 d Array to hold values must be declared with ReDim SM V7 will dynamically 'alloc To match number of axis points. Size used here Is ignored, so can be 0 Dim adValue2d As Variant 'Get current values of 2 axis map szIdent = "EI_24704" lErr = SmApi.GetValue2AxisMap(0, szIdent, adValue2d) PrintIdent(szIdent, lErr) For i = 0 To UBound(adValue2d, 2) 'First dimension For j = 0 To UBound(adValue2d, 1) 'Second dimension Debug.Print adValue2d(j,i); Debug.Print " "; Next j Debug.Print Next i Wait 100 Set SmApi = Nothing End Sub Function PrintArray(adValue) 'Print out current values. You can use UBound to find number of axis points For i = 0 To UBound(adValue) Debug.Print adValue(i); Debug.Print " "; Next i Debug.Print 'New line End Function Function PrintIdent(szIdent, lErr) Debug.Print"ID = "; 'New line Debug.Print szIdent; 'Parameter Identifier Debug.Print ", Error = "; Debug.Print lErr End Function ``` ### Sample Output ``` ID = EC\_1147, Error = 0 5.00435660800002 4.00458547200003 3.30144972800002 2.00504319999999 1.005272064 5.50092800000357E-03 -1.99404134399998 -3.99358361600002 ID = EC\_1145, Error = 0 -5 20 44 60 80 100 110 120 ID = EG\_26006, Error = 0 0 16000 17000 18000 19380 ID = EI\_24704, Error = 0 ``` | | | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0 | 0 | 0 | 0 | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | | 0 | 0 | 0 | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | | 0 | 0 | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | | 0 | 0.5 | 1.1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | 9 | | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | 9 | 9.5 | | 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | 9 | 9.5 | 10 | --- # Manipulating the Values in a 1-Axis Map ``` 'Declare globally available SM API. Dim SmApi As Object Sub Main 'Get access to System Monitor API Set SmApi = CreateObject("System Monitor API") 'Param to retrieve Dim szIdent As String 'Error return code Dim lErr As Long 'Variables which are to receive map values must be declared as Variants. 'This is because SM needs to dynamically allocate and dimension the array. Dim adValue As Variant 'Get current values of 1-axis map szIdent = "EC_1147" lErr = SmApi.GetValue1AxisMap(0, szIdent, adValue) PrintIdent(szIdent, lErr) PrintArray(adValue) 'Declare array to be passed into SM-V7 to set map value. 'These must be declared as arrays in order that SM-V7 can extract array dimensions, values, etc Dim adValueSet() As Double 'Copy returned values to values we're going to set, and add 10% to all breakpoints 'excluding the first, second and last adValueSet = adValue For i = 2 To (UBound(adValueSet) - 1) adValueSet(i) = adValue(i) + adValue(i) / 10 Next i 'Set map values lErr = SmApi.SetValue1AxisMap(0, szIdent, adValueSet) 'Get current values of 1-axis map and print lErr = SmApi.GetValue1AxisMap(0, szIdent, adValue) PrintIdent(szIdent, lErr) PrintArray(adValue) 'Pause to let user see result Wait 100 'You must free the SM Api before running another script Set SmApi = Nothing End Sub Function PrintArray(adValue) 'Print out current values. You can use UBound to find number of axis points For i = 0 To UBound(adValue) Debug.Print adValue(i); Debug.Print " "; Next i Debug.Print 'New line End Function Function PrintIdent(szIdent, lErr) Debug.Print "ID = "; 'New line Debug.Print szIdent; 'Parameter Identifier Debug.Print ", Error = "; Debug.Print lErr End Function ``` ### Sample Output ``` ID = EC_1147, Error = 0 5.00435660800002 4.00458547200003 3.63104460800002 2.20280012799998 1.10415052799999 5.50092800000357E 03 2.19179827199997 3.99358361600002 ID = EC_1147, Error = 0 5.00435660800002 4.00458547200003 3.99359897599999 2.422530048 1.21401548800003 5.50092800000357E 03 2.41152819199999 3.99358361600002 ``` --- # Demonstrating functions GetValue2AxisMap and SetValue2AxisMap ``` Dim SmApi As Object Sub Main ' Get access to System Monitor API Set SmApi = CreateObject("System Monitor API") ' Param to retrieve Dim szIdent As String ' Error return code Dim lErr As Long ' Variables which are to receive map values must be declared as Variants. ' This is because SM needs to dynamically allocate and dimension the array. Dim adValue As Variant ' Get current values of 2-axis map szIdent = "EC_1147" lErr = SmApi.GetValue2AxisMap(0, szIdent, adValue) PrintIdent(szIdent, lErr) Print2dArray(adValue) ' Declare array to be passed into SM-V7 to set map value. ' These must be declared as arrays in order that SM-V7 can extract array dimensions, values, etc Dim adValueSet() As Double ' Copy returned values to values we're going to set, and add 10% to all breakpoints excluding the first, second and last adValueSet = adValue For i = 2 To (UBound(adValueSet, 2) - 1) For n = 2 To (UBound(adValueSet, 1) - 1) adValueSet(n, i) = adValue(n, i) + adValue(n, i) / 10 Next n Next i ' Set map values lErr = SmApi.SetValue2AxisMap(0, szIdent, adValueSet) ' Get current values of 1-axis map and print lErr = SmApi.GetValue2AxisMap(0, szIdent, adValue) PrintIdent(szIdent, lErr) Print2dArray(adValue) ' Pause to let user see result Wait 100 ' You must free the SM Api before running another script Set SmApi = Nothing End Sub Function Print2dArray(adValue) ' Print out current values. You can use UBound to find number of axis points For i = 0 To UBound(adValue, 2) For n = 0 To UBound(adValue, 1) Debug.Print adValue(n, i); Debug.Print " "; Next n Debug.Print Next i Debug.Print ' New line End Function Function PrintIdent(szIdent, lErr) Debug.Print "ID = "; ' New line Debug.Print szIdent; ' Parameter Identifier Debug.Print ", Error = "; Debug.Print lErr End Function ``` --- # Getting Started This guide walks you through getting ATLAS Open Streaming data flowing from source to screen. Whether you are writing telemetry from your own systems into Kafka, or reading live sessions inside the ATLAS client, start here. ## What is the Open Streaming Architecture? The Open Streaming Architecture is a broker-based system for streaming engineering telemetry data in real time. Data is published and consumed via [Apache Kafka](https://kafka.apache.org/) using a defined [Protobuf protocol](../secu4/docs.md), so any language or platform that can talk to Kafka can participate. ``` Your Data Source ──► Stream API (gRPC) ──► Kafka Broker ──► ATLAS Client │ │ ▼ ▼ Other consumers Stream Recorder (dashboards, (live display & logging, etc.) recording) ``` This architecture is data-source agnostic. It works with ECU telemetry, sim racing data, kart loggers, test rigs, cycling power meters — anything that produces time-series data. For a real-world worked example, see [How I Got iRacing Telemetry Streaming into ATLAS Viewer](../../blog/posts/blog-post-iracing-bridge-service.md). ## Architecture at a Glance | Component | What it does | When you need it | |-----------|-------------|-----------------| | **[Stream API](stream_api/index.md)** | gRPC server that manages sessions, connections, data formats, and packet routing to/from Kafka. Runs as a Docker container or embedded in your app. | Always. Every producer and consumer talks through the Stream API. | | **[Key Generator](stream_api/index.md)** | Generates unique identifiers required by `ConfigurationPacket`. Runs alongside the Stream API. | Always, when sending configuration packets. | | **[Support Library](support_library/reference_docs/index.md)** | .NET convenience library that wraps the Stream API into a higher-level pipeline (session management, buffering, interpolation). | When building a .NET application and you want managed read/write pipelines without handling gRPC directly. | | **[Bridge Service](bridge_service/index.md)** | Standalone service that bridges a raw ADS data stream into the Open Streaming protocol. | When your data source is an ADS (SECU unit) and you need it converted automatically. | | **[Indexing Services](indexing_services/index.md)** | Session indexing and search across historical streaming data. | When you need to query and retrieve past sessions. | ### Stream API vs Support Library This is the most common source of confusion. Here is the distinction: - The **Stream API** is the underlying gRPC service. It is language-agnostic: any language with gRPC support (Python, C++, Java, Go, etc.) can use it directly via the [proto definitions](https://github.com/Software-Products/MA.DataPlatforms.Protocol). - The **Support Library** is available as a **.NET NuGet package** and a **Python package** (via FFI). It wraps the Stream API into higher-level abstractions (automatic buffering, interpolation, session lifecycle). **Choose the Stream API directly** if you need fine-grained control, are using a language other than C#/Python, or want to match the approach used in the example repositories. **Choose the Support Library** if you want the managed pipeline experience with built-in buffering and interpolation. Available in both C# ([NuGet](https://github.com/mat-docs/packages)) and Python ([PyPI package](https://github.com/Software-Products/MA.DataPlatforms.Streaming.Support.Library/releases)). !!! note "Which approach do the examples use?" The [example-stream-api-kafka-setup](https://github.com/atlas-dev-hub/example-stream-api-kafka-setup) and [iRacing bridge](https://github.com/atlas-dev-hub/example-bridge-service-iracing) use **raw gRPC calls to the Stream API**. The [Support Library sample code](https://github.com/mat-docs/MA.DataPlatforms.Streaming.Support.Library.SampleUsage) uses the **Support Library wrapper**. These are two different layers over the same protocol — if translating between them, be aware the API surfaces differ. ## Prerequisites - [Docker Desktop](https://www.docker.com/products/docker-desktop/) (Engine 20+) - A Kafka broker (the quick-start below provides one via Docker) - For .NET development: .NET 8.0+ SDK and the [Motion Applied NuGet feed](https://github.com/mat-docs/packages) - For Python/other languages: protobuf compiler and gRPC tools for your language ## Quick Start: Data Flowing in 30 Minutes The fastest way to get a working environment is the [example repository](https://github.com/atlas-dev-hub/example-stream-api-kafka-setup), which spins up Kafka, the Stream API server, and a Key Generator — all via Docker Compose. ### 1. Clone the Example Repository ```bash git clone https://github.com/atlas-dev-hub/example-stream-api-kafka-setup.git cd example-stream-api-kafka-setup ``` ### 2. Start the Docker Stack ```bash docker compose up -d ``` This starts four containers: | Container | Port | Purpose | |-----------|------|---------| | `atlas-kafka` | 9092 | Kafka message broker (KRaft mode, no Zookeeper) | | `atlas-kafka-ui` | 8080 | Web UI for inspecting Kafka topics and messages | | `atlas-stream-api` | 13579, 10010 | Stream API gRPC server + Prometheus metrics | | `atlas-key-generator` | 15379 | Unique key generation for configuration packets | ### 3. Verify the Stack ```bash docker compose ps ``` All four containers should show as healthy/running. Open [http://localhost:8080](http://localhost:8080) to access the Kafka UI — watching messages flow through topics is the fastest way to debug integration issues. ### 4. Explore with the Notebook Open `atlas_kafka_streaming.ipynb` in VS Code (with the [Jupyter extension](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter)) to walk through producing and consuming streaming data interactively. ### 5. Connect ATLAS Client To view streaming data in the ATLAS client: 1. Open ATLAS and go to **Tools > Options > Recorders > Stream Recorder** 2. Configure the Stream Server URL (e.g. `localhost:13579`) 3. Open the **Session Browser** and add a **Stream Recorder** 4. Set the **Data Source** to match the `dataSource` value used when creating sessions 5. Choose **Live** or **Live with Catchup** mode 6. Click **Start** — ATLAS will display data as it arrives See [Stream Recorder](../../key-functionality/analyse/viewer/handling-data/recorders/stream-recorder.md) for full configuration details. ## The Protocol: Five Steps, Exact Order The session lifecycle has a specific sequence, and the order matters more than you might expect. Getting this wrong is the most common integration issue — most debugging time is spent discovering these invariants. ``` 1. Create the session 2. Announce the session (NewSessionPacket to BOTH streams) 3. Send the ConfigurationPacket (to BOTH streams) 4. Register data formats (get format IDs for your parameters) 5. Stream data (PeriodicDataPacket / RowDataPacket / EventPacket) ``` ### Step 1: Create the Session === "Stream API (gRPC)" ```python resp = session_stub.CreateSession( api_pb2.CreateSessionRequest( data_source="Default", identifier="Spa - McLaren 720S - 2026-03-27 14:30", type="Session", version=1, ) ) session_key = resp.session_key ``` === "Stream API (C#)" ```csharp var sessionManager = streamingApiClient.GetSessionManagementClient(); var response = await sessionManager.CreateSessionAsync(new CreateSessionRequest { DataSource = "Default", Type = "Session", Version = 1, Identifier = "TestRun_001", Details = { { "Driver", "DriverName" }, { "Track", "TrackName" } } }); var sessionKey = response.SessionKey; ``` === "Support Library (.NET)" ```csharp var sessionResult = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "Default", identifier: "TestRun_001", type: "Session", details: [ new SessionDetailDto("Driver", "DriverName"), new SessionDetailDto("Track", "TrackName") ])); var sessionKey = sessionResult.Data.SessionKey; ``` The `identifier` shows up in the ATLAS session browser. Use something descriptive — track, car, timestamp. ### Step 2: Announce the Session Send a `NewSessionPacket` to **both** your data stream and the main stream (`""`). This is how ATLAS discovers a session exists. === "Python" ```python new_sess = open_data_pb2.NewSessionPacket(data_source="Default") write_packet(new_sess, stream="Stream1") # data stream write_packet(new_sess, stream="") # main stream ``` === "C#" ```csharp var newSession = new NewSessionPacket { DataSource = "Default" }; var packet = new Packet { SessionKey = sessionKey, Content = newSession.ToByteString(), Id = packetId++, Type = "NewSession", IsEssential = false }; // Send to BOTH streams writerService.WriteData("Default", "Stream1", sessionKey, packet); writerService.WriteData("Default", "", sessionKey, packet); ``` ### Step 3: Send the ConfigurationPacket !!! warning "This is the step most integrators miss" The `ConfigurationPacket` declares every parameter you will stream. ATLAS will not interpret any data without it. Your session will appear with zero parameters if this is missing. The ConfigurationPacket defines parameter metadata (name, units, min/max, frequency, groups), event definitions, and the group hierarchy. === "Python" ```python # Generate a unique config_id via the Key Generator service config_id = keygen_stub.GenerateUniqueKey( key_generator_pb2.GenerateUniqueKeyRequest( type=key_generator_pb2.KEY_TYPE_STRING ) ).string_key config_pkt = open_data_pb2.ConfigurationPacket( config_id=config_id, parameter_definitions=[ open_data_pb2.ParameterDefinition( identifier="Engine.Rpm", name="Engine RPM", application_name="VehicleTelemetry", description="Engine revolutions per minute", units="RPM", data_type=open_data_pb2.DATA_TYPE_FLOAT64, format_string="%5.0f", min_value=0.0, max_value=12000.0, warning_min_value=0.0, warning_max_value=12000.0, groups=["Engine"], frequencies=[10.0], ), # ... more parameters ], group_definitions=[ open_data_pb2.GroupDefinition( identifier="VehicleTelemetry", application_name="VehicleTelemetry", name="Vehicle Telemetry", description="Vehicle telemetry data", ) ], ) # Send to BOTH streams write_packet(config_pkt, stream="", essential=True) write_packet(config_pkt, stream="Stream1", essential=True) ``` === "C# (Stream API)" ```csharp var telemetryConfig = new ConfigurationPacket { ConfigId = "vehicle-telemetry-v1", // use Key Generator in production ParameterDefinitions = { new ParameterDefinition { Identifier = "Engine.Rpm", Name = "Engine RPM", ApplicationName = "VehicleTelemetry", Description = "Engine revolutions per minute", Units = "RPM", DataType = DataType.Float64, FormatString = "%5.0f", MinValue = 0.0, MaxValue = 12000.0, WarningMinValue = 0.0, WarningMaxValue = 12000.0, Groups = { "Engine" }, Frequencies = { 10.0 }, }, new ParameterDefinition { Identifier = "Engine.Temperature", Name = "Engine Temperature", ApplicationName = "VehicleTelemetry", Description = "Engine coolant temperature", Units = "°C", DataType = DataType.Float64, FormatString = "%5.1f", MinValue = -40.0, MaxValue = 150.0, WarningMinValue = -20.0, WarningMaxValue = 130.0, Groups = { "Engine" }, Frequencies = { 1.0 }, }, }, GroupDefinitions = { new GroupDefinition { Identifier = "VehicleTelemetry", ApplicationName = "VehicleTelemetry", Name = "Vehicle Telemetry", Description = "Vehicle telemetry data", } } }; // Send to BOTH streams await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "Default", SessionKey = sessionKey, Stream = "", // main stream Message = new Packet { Content = telemetryConfig.ToByteString(), Type = "Configuration", IsEssential = true, SessionKey = sessionKey } } }); // Repeat for "Stream1" ``` **Critical rules for ConfigurationPacket:** | Rule | Why | |------|-----| | **Exactly ONE `GroupDefinition`** per ConfigurationPacket | Multiple groups cause zero parameters and `FlushSlotsAsRequired - slots flushed 0` in logs. The group is a session-level container; parameter organisation happens through naming and `application_name`. | | **`config_id` should be unique** | The [Support Library sample](https://github.com/mat-docs/MA.DataPlatforms.Streaming.Support.Library.SampleUsage) uses a hardcoded string (`"ConfigPacket"`) which works for single-session scenarios. For production use with multiple sessions, use the Key Generator service (`localhost:15379`) to generate unique IDs. | | **Send to BOTH streams** | SQLRace processes each Kafka topic independently. Without config on the data stream, markers and data on that stream are silently discarded. Note: the Support Library sample sends config to `""` only — this works for basic data streaming but will cause markers on the data stream to be dropped. Send to both `""` and your data stream (e.g. `"Stream1"`). | ### Step 4: Register Data Formats Each parameter group needs a data format identifier. This links your parameter definitions to the data packets. === "Python" ```python resp = dfmt_stub.GetParameterDataFormatId( api_pb2.GetParameterDataFormatIdRequest( data_source="Default", parameters=["Engine.Rpm", "Engine.Temperature"], ) ) format_id = resp.data_format_identifier ``` === "C# (Stream API)" ```csharp var formatResponse = await dataFormatManager.GetParameterDataFormatIdAsync( new GetParameterDataFormatIdRequest { DataSource = "Default", Parameters = { "Engine.Rpm", "Engine.Temperature" } }); var formatId = formatResponse.DataFormatIdentifier; ``` ### Step 5: Stream Data Send data as `PeriodicDataPacket` (fixed-frequency), `RowDataPacket` (timestamped rows), or `EventPacket`. === "Python" ```python data_pkt = open_data_pb2.PeriodicDataPacket( data_format=open_data_pb2.SampleDataFormat( data_format_identifier=format_id ), start_time=timestamp_ns, interval=16666666, # 60 Hz = 16.67ms in nanoseconds columns=[col], ) write_packet(data_pkt, stream="Stream1") ``` === "C# (Stream API)" ```csharp var engineData = new PeriodicDataPacket { DataFormat = new SampleDataFormat { DataFormatIdentifier = formatId }, StartTime = (ulong)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000000, Interval = 100000000, // 100ms in nanoseconds (10 Hz) Columns = { new SampleColumn { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = 8500.0, Status = DataStatus.Valid } } } } } }; await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "Default", Stream = "Stream1", SessionKey = sessionKey, Message = new Packet { Content = engineData.ToByteString(), Type = "PeriodicData", IsEssential = false, SessionKey = sessionKey } } }); ``` !!! note "All timestamps are nanoseconds" Every timestamp field in the protocol (`StartTime`, `Interval`, `Timestamp`) uses **nanoseconds since UNIX epoch** (1 Jan 1970, UTC). Common intervals: 60 Hz = `16666666` ns, 100 Hz = `10000000` ns, 1 kHz = `1000000` ns. Convert from milliseconds: `ms * 1000000`. ## Packet Routing Reference Different packets go to different streams. Get this wrong and ATLAS either does not see the session or does not see the data: | Packet | Stream | Essential | Notes | |--------|--------|-----------|-------| | `NewSessionPacket` | Both (`Stream1` + `""`) | No | How ATLAS discovers the session | | `ConfigurationPacket` | Both (`Stream1` + `""`) | Yes | Must precede data on every stream | | `PeriodicDataPacket` | Data stream (`Stream1`) | No | Telemetry samples | | `RowDataPacket` | Data stream (`Stream1`) | No | Timestamped rows | | `EventPacket` | Data stream (`Stream1`) | No | Events | | `MarkerPacket` | Both (`Stream1` + `""`) | Yes | Lap markers, pit stops | | `EndOfSessionPacket` | Both (`Stream1` + `""`) | No | Session end signal | ## Making Traces Smooth If your traces appear jittery or discontinuous in ATLAS, the issue is almost always timestamp handling. The key insight: ATLAS needs each `PeriodicDataPacket` to contain contiguous samples with an exact interval. **Do:** - Collect multiple samples into a batch, then send as a single `PeriodicDataPacket` - Advance `start_time` arithmetically: `start_time += interval * sample_count` - Use incrementing packet IDs on every `Packet` wrapper **Don't:** - Send one sample per packet (ATLAS has to infer continuity across packets) - Use wall-clock timestamps per batch (OS scheduling jitter causes gaps) ```python # Collect one second of samples at 60 Hz, paced in real-time for _ in range(batch_size): sample = read_telemetry() batch.append(sample) time.sleep(interval_seconds) # Send as one packet, advance timestamp arithmetically send_periodic_data(batch, start_time_ns, interval_ns) start_time_ns += interval_ns * batch_size ``` For the full worked example including production considerations (per-sample timestamps, non-blocking gRPC sends, clock drift prevention), see the [iRacing bridge blog post](../../blog/posts/blog-post-iracing-bridge-service.md). ## Troubleshooting ### Session appears but parameters are missing 1. **Check `ConfigurationPacket` was sent** — this is the most common cause 2. **Check for exactly ONE `GroupDefinition`** — multiple groups silently produce zero parameters 3. **Check `config_id` is unique** — use the Key Generator service, not a hardcoded string 4. **Check config was sent to BOTH streams** — SQLRace needs config on every stream where data flows Look for `FlushSlotsAsRequired - slots flushed 0` in the Stream API logs — this confirms a config issue. ### Data is being written but ATLAS shows nothing 1. **Stream API reachable?** Can the ATLAS machine reach `:13579`? 2. **Topics exist?** Check Kafka UI at [http://localhost:8080](http://localhost:8080) 3. **Data Source matches?** The name in your writer must match the Stream Recorder config 4. **NewSessionPacket sent to both streams?** ATLAS discovers sessions via the main stream (`""`) ### Lap markers not appearing Markers require the `ConfigurationPacket` on the same stream. If config only went to `""` but markers go to `Stream1`, SQLRace silently drops them. Also ensure marker timestamps fall within the range of your `PeriodicDataPacket` timestamps — markers outside any data packet's time range are discarded. ### Error tracing The Stream API server logs to the `logs/` directory when running via Docker: ```bash docker compose logs -f stream-api ``` Enable debug logging in `AppConfig.json`: ```json { "Serilog": { "MinimumLevel": { "Default": "Debug", "Override": { "MA.Streaming": "Debug" } } } } ``` ### Stream Recorder config breaks ATLAS If a Stream Recorder configuration references a server that no longer exists, ATLAS may fail to start. To recover, delete the Stream Recorder configuration file from the ATLAS config directory and restart. ## Common Pitfalls | Pitfall | Solution | |---------|----------| | Missing `ConfigurationPacket` | Always send after session creation, before any data, to **both** streams | | Multiple `GroupDefinition` entries | Exactly one group per `ConfigurationPacket`. Organise parameters via naming, not multiple groups | | Hardcoded `config_id` | Use the Key Generator service (`localhost:15379`) to generate unique IDs | | Wrong timestamp units | All timestamps are **nanoseconds**, not microseconds or milliseconds | | Config sent to one stream only | Send to both data stream (`Stream1`) and main stream (`""`) | | One sample per packet | Batch samples into a single `PeriodicDataPacket` for smooth traces | | Wall-clock batch timestamps | Advance `start_time` arithmetically, not from `time.time_ns()` | | Wrong order of operations | Session > NewSession > Config > DataFormats > Data > EndSession | ## Example Repositories | Repository | Language | What it demonstrates | |------------|----------|---------------------| | [example-stream-api-kafka-setup](https://github.com/atlas-dev-hub/example-stream-api-kafka-setup) | Python / Jupyter | Docker stack + interactive notebook for exploring the Stream API | | [example-bridge-service-iracing](https://github.com/atlas-dev-hub/example-bridge-service-iracing) | Python | Complete bridge service: iRacing telemetry > Stream API > Kafka > ATLAS | | [MA.Streaming.Api.UsageSample](https://github.com/mat-docs/MA.Streaming.Api.UsageSample) | C# + Python | Sample code for both languages using the Stream API directly | | [Support Library Sample Usage](https://github.com/mat-docs/MA.DataPlatforms.Streaming.Support.Library.SampleUsage) | C# + Python | Sample code using the Support Library wrapper (includes buffering/interpolation examples) | | [Support Library Source](https://dev.azure.com/MAT-OCS/MA.DataPlatforms/_git/MA.DataPlatforms.Streaming.Support.Library) | C# + Python FFI | Support Library source code with E2E samples and unit tests (requires Azure DevOps access) | ## Reference Documentation - **[Stream API](stream_api/index.md)** — Full gRPC service documentation with Docker Compose examples - **[Support Library](support_library/reference_docs/index.md)** — .NET library module reference - **[Protocol Documentation](docs.md)** — Complete Protobuf schema (all packet types, fields, and message structures) - **[Bridge Service](bridge_service/index.md)** — ADS-to-Stream bridge configuration - **[Telemetry Example](stream_api/reference_docs/examples/telemetry-example.md)** — Complete C# producer/consumer with ConfigurationPacket - **[Stream Recorder](../../key-functionality/analyse/viewer/handling-data/recorders/stream-recorder.md)** — ATLAS client-side recording setup --- # Overview The Stream API is a component that manages real-time data streaming to and from Kafka brokers as part of the Open Streaming Architecture. It provides a gRPC-based interface that enables any programming language to interact with live data streams efficiently. For more information, see [Open Streaming Architecture Documentation](https://atlas.motionapplied.com/developer-resources/secu4/docs/) ## Table of Contents - [Overview](#overview) - [Architecture](#architecture) - [Getting Started](#getting-started) - [Services](#services) - [Configuration](#configuration) - [Examples](#examples) - [API Reference](#api-reference) - [Testing](#testing) - [Deployment](#deployment) - [Troubleshooting](#troubleshooting) ## Overview The Stream API enables real-time data streaming with the following key features: - **High-performance streaming**: Optimized for low-latency, high-throughput data transmission - **Multi-language support**: gRPC-based interface accessible from any programming language - **Session management**: Create, manage, and monitor data sessions - **Data format management**: Define and manage parameter and event schemas - **Connection pooling**: Efficient connection management for multiple clients - **Flexible deployment**: Self-hosted or Docker-based server deployment ### Key Benefits - **Real-time processing**: Stream data with minimal latency - **Flexibility**: Support for both partition-based and topic-based routing strategies ## Architecture ### Core Components The Stream API consists of five main services: 1. **Session Management Service**: Handles session lifecycle and state management 2. **Connection Management Service**: Manages client connections to data streams 3. **Packet Writer Service**: Publishes data and info packets (single or batch) 4. **Packet Reader Service**: Reads all packets, essentials, or filtered data packets 5. **Data Format Management Service**: Manages parameter and event format definitions ### Deployment Models #### Local Self-Hosted ```csharp // Embed the Stream API server directly in your application var streamConfiguration = new StreamingApiConfiguration( StreamCreationStrategy.TopicBased, "localhost:9092", partitionMappings, integrateDataFormatManagement: true, integrateSessionManagement: true, batchingResponses: false); var streamingApiClient = StreamingApiClientFactory.Create(streamConfiguration, cancellationTokenProvider, brokerChecker, loggingProvider); streamingApiClient.Initialise(); ``` #### Remote Docker Server ```bash # Run the Stream API server in Docker docker run -p 13579:13579 -p 10010:10010 \ -v ./Configs:/app/Configs \ atlasplatformdocker/streaming-proto-server-host:latest ``` ## Getting Started ### Client Setup #### C# NuGet Package For C# applications, use the NuGet packages from the [Motion Applied NuGet Repository](https://github.com/mat-docs/packages). ```xml ``` #### Other Languages For other programming languages, use the proto files from the [MA.DataPlatforms.Protocol](https://github.com/Software-Products/MA.DataPlatforms.Protocol) repository to generate client code. ### Quick Start Example ```csharp // Initialize the client var configuration = new StreamingApiConfiguration( StreamCreationStrategy.TopicBased, "localhost:9092", []); var streamingApiClient = StreamingApiClientFactory.Create(configuration, tokenProvider, brokerChecker, loggingProvider); streamingApiClient.Initialise(); // Get service clients var connectionManager = streamingApiClient.GetConnectionManagerClient(); var packetWriter = streamingApiClient.GetPacketWriterClient(); var packetReader = streamingApiClient.GetPacketReaderClient(); // Create connection var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "MyDataSource", SessionKey = "MySession", Streams = { "Stream1" } } }); // Write data await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "MyDataSource", Stream = "Stream1", Message = new Packet { Content = ByteString.CopyFrom(data), SessionKey = "MySession", Type = "MyDataType" } } }); // Read data var readStream = packetReader.ReadPackets(new ReadPacketsRequest { Connection = connection.Connection }); await foreach (var response in readStream.ResponseStream.ReadAllAsync()) { foreach (var packet in response.Response) { // Process received packet ProcessPacket(packet); } } ``` ### Server Setup with Docker #### Docker Compose ```yaml name: kafka-compose services: kafka: image: apache/kafka:latest hostname: kafka container_name: kafka-broker-1 networks: kafka_net_interal: ipv4_address: 172.22.0.7 ports: - "9094:9094" environment: CLUSTER_ID: 'dev-kafka-cluster' KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093,PLAINTEXT_HOST://:9094' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9094' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui-1 networks: kafka_net_interal: ipv4_address: 172.22.0.8 ports: - "8080:8080" environment: KAFKA_CLUSTERS_0_NAME: 'dev-kafka-cluster' KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: 'kafka:9092' DYNAMIC_CONFIG_ENABLED: 'true' depends_on: - kafka stream-api-server: image: atlasplatformdocker/streaming-proto-server-host:latest container_name: stream-api-server networks: kafka_net_interal: ipv4_address: 172.22.0.9 ports: - 13579:13579 - 10010:10010 depends_on: - kafka environment: AUTO_START: true volumes: - ./configs:/app/Configs key-generator: image: atlasplatformdocker/keygenerator-proto-server:latest container_name: key-generator-server networks: kafka_net_interal: ipv4_address: 172.22.0.10 ports: - 15379:15379 - 10011:10010 depends_on: - kafka environment: AUTO_START: true volumes: - ./key-configs:/app/Configs networks: kafka_net_interal: driver: bridge ipam: config: - subnet: 172.22.0.0/16 gateway: 172.22.0.1 ``` ## Services ### Session Management Service Manages the lifecycle of data sessions, including creation, monitoring, and termination. #### Key Features - Session creation with metadata - Session info manipulation - Session information updates #### Example Usage ```csharp var sessionManager = streamingApiClient.GetSessionManagementClient(); // Create a new session var newSession = await sessionManager.CreateSessionAsync(new CreateSessionRequest { DataSource = "sensor-data", Type = "Test Session", Version = 1, Identifier = "test-001" }); // Update session details await sessionManager.UpdateSessionDetailsAsync(new UpdateSessionDetailsRequest { SessionKey = newSession.SessionKey, Details = { {"status", "active"}, {"participant", "driver-123"} } }); // End session await sessionManager.EndSessionAsync(new EndSessionRequest { DataSource = "sensor-data", SessionKey = newSession.SessionKey }); ``` ### Connection Management Service Handles client connections to data sessions with unique connection tracking. #### Key Features - Unique connection ID assignment - Connection state management #### Example Usage ```csharp var connectionManager = streamingApiClient.GetConnectionManagerClient(); // Create new connection var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "telemetry", SessionKey = "session-123", Streams = { "engine", "brakes", "suspension" }, StreamOffsets = { 0, 0, 0 }, ExcludeMainStream = false } }); // Retrieve connection details var connectionInfo = await connectionManager.GetConnectionAsync(new GetConnectionRequest { Connection = connection.Connection }); // Close connection var closeResult = await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = connection.Connection }); ``` ### Packet Writer Service Enables high-performance data publishing to Kafka topics with support for both individual packets and batch operations. #### Key Features - Single packet writing - Batch packet writing - Essential packet handling - Stream-based routing #### Example Usage ```csharp var packetWriter = streamingApiClient.GetPacketWriterClient(); // Write single packet await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "telemetry", Stream = "engine", SessionKey = "session-123", Message = new Packet { Content = ByteString.CopyFrom(sensorData), Type = "PeriodicData", IsEssential = false } } }); // Write batch of packets var batchStream = packetWriter.WriteDataPackets(); foreach (var data in dataPoints) { await batchStream.RequestStream.WriteAsync(new WriteDataPacketsRequest { Details = { new DataPacketDetails { DataSource = "telemetry", Stream = "engine", SessionKey = "session-123", Message = new Packet { Content = ByteString.CopyFrom(data), Type = "PeriodicData" } } } }); } await batchStream.RequestStream.CompleteAsync(); ``` ### Packet Reader Service Provides flexible data consumption with three reading modes: all packets, essential packets only, or filtered data packets. #### Key Features - **ReadPackets**: Continuous streaming of all packet types - **ReadEssentials**: Essential packets only (config, metadata) - **ReadDataPackets**: Filtered data packets using regex patterns #### Example Usage ```csharp var packetReader = streamingApiClient.GetPacketReaderClient(); // Read all packets var readStream = packetReader.ReadPackets(new ReadPacketsRequest { Connection = connection.Connection }); // Read essential packets only var essentialStream = packetReader.ReadEssentials(new ReadEssentialsRequest { Connection = connection.Connection }); // Read filtered data packets (supports regex patterns) var filteredStream = packetReader.ReadDataPackets(new ReadDataPacketsRequest { Request = new DataPacketRequest { Connection = connection.Connection, IncludeParameters = { "vCar.*", ".*Temp.*", "NGear.*" }, ExcludeParameters = { ".*Debug.*" } } }); // Process received data var cts = new CancellationTokenSource(); while (await readStream.ResponseStream.MoveNext(cts.Token)) { var response = readStream.ResponseStream.Current.Response; foreach (var packetResponse in response) { ProcessPacket(packetResponse.Packet, packetResponse.Stream); } } ``` ### Data Format Management Service Manages data schemas for parameters and events, enabling efficient data serialization and client-side filtering. #### Key Features - Parameter list registration - Event format registration - Data format retrieval - Automatic ID generation #### Example Usage ```csharp var dataFormatManager = streamingApiClient.GetDataFormatManagerClient(); // Register parameter data format var parameterFormat = await dataFormatManager.GetParameterDataFormatIdAsync( new GetParameterDataFormatIdRequest { DataSource = "telemetry", Parameters = { "EngineRPM", "Speed", "Throttle", "Brake" } }); // Register event data format var eventFormat = await dataFormatManager.GetEventDataFormatIdAsync( new GetEventDataFormatIdRequest { DataSource = "telemetry", Event = "LapCompleted" }); // Retrieve parameter list var parameters = await dataFormatManager.GetParametersListAsync( new GetParametersListRequest { DataSource = "telemetry", DataFormatIdentifier = parameterFormat.DataFormatIdentifier }); // Retrieve event name var eventInfo = await dataFormatManager.GetEventAsync(new GetEventRequest { DataSource = "telemetry", DataFormatIdentifier = eventFormat.DataFormatIdentifier }); ``` ## Configuration ### AppConfig.json Configure the Stream API server using the AppConfig.json file: ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "kafka:9092", "PartitionMappings": [], "StreamApiPort": 13579, "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": true, "RemoteKeyGeneratorServiceAddress": "key-generator-service:15379", "BatchingResponses": false, "InitialisationTimeoutSeconds": 5 } } ``` #### Configuration Options | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | `StreamCreationStrategy` | `int` | 1=Partition-based, 2=Topic-based | 1 | | `BrokerUrl` | `string` | Kafka broker connection string | Required | | `PartitionMappings` | `array` | Stream to partition mappings | `[]` | | `StreamApiPort` | `int` | gRPC server port | 13579 | | `IntegrateSessionManagement` | `bool` | Enable session management | true | | `IntegrateDataFormatManagement` | `bool` | Enable data format management | true | | `UseRemoteKeyGenerator` | `bool` | Use external key generator | false | | `BatchingResponses` | `bool` | Enable response batching | false | | `BatchingSizesKb` | `int` | Maximum batch size in KB | 4096 | | `BatchingTimeMs` | `int` | Maximum time before flushing a batch (ms) | 50 | | `InitialisationTimeoutSeconds` | `uint` | Service startup timeout | 3 | | `TerminationTimeoutSeconds` | `uint` | Graceful termination timeout | 1 | | `Domain` | `string` | Kafka topic domain prefix | "" | | `EnableGrpcReflection` | `bool` | Enable gRPC server reflection | false | !!! note "Prometheus Metrics Port" The Prometheus metrics port is configured via the `PROMETHEUS_METRIC_PORT` environment variable, not in `AppConfig.json`. ### Stream Creation Strategies #### Topic-Based Strategy (Strategy 2) Each stream maps to a separate Kafka topic, providing better isolation and scalability. PartitionMappings are not used in this mode. ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "PartitionMappings": [] } } ``` #### Partition-Based Strategy (Strategy 1) All streams use the same topic but different partitions, suitable for related data streams. This is the default strategy. Requires PartitionMappings configuration to specify which stream goes to which partition. ```json { "StreamApiConfig": { "StreamCreationStrategy": 1, "PartitionMappings": [ {"Stream": "Engine", "Partition": 1}, {"Stream": "Brakes", "Partition": 2}, {"Stream": "Suspension", "Partition": 3} ] } } ``` ## Examples ### Complete Telemetry Example ```csharp using MA.Streaming.API; using MA.Streaming.Proto.Client.Local; using Google.Protobuf; public class TelemetryExample { private PacketWriterService.PacketWriterServiceClient packetWriter; private PacketReaderService.PacketReaderServiceClient packetReader; private ConnectionManagerService.ConnectionManagerServiceClient connectionManager; private DataFormatManagerService.DataFormatManagerServiceClient dataFormatManager; public async Task RunExample() { // 1. Initialize the Stream API await InitializeStreamApi(); // 2. Create connection var connection = await CreateConnection(); // 3. Register data formats await RegisterDataFormats(); // 4. Start reading data in background var readingTask = StartReading(connection); // 5. Write telemetry data await WriteTelemetryData(); // 6. Wait for reading to complete await readingTask; // 7. Close connection await CloseConnection(connection); } private async Task InitializeStreamApi() { var configuration = new StreamingApiConfiguration( StreamCreationStrategy.TopicBased, "localhost:9092", [], integrateDataFormatManagement: true, integrateSessionManagement: true, batchingResponses: false); var tokenProvider = new CancellationTokenSourceProvider(); var brokerChecker = new KafkaBrokerAvailabilityChecker(); var loggingProvider = new LoggingDirectoryProvider(""); var streamingApiClient = StreamingApiClientFactory.Create( configuration, tokenProvider, brokerChecker, loggingProvider ) streamingApiClient.Initialise() // Get service clients packetWriter = streamingApiClient.GetPacketWriterClient(); packetReader = streamingApiClient.GetPacketReaderClient(); connectionManager = streamingApiClient.GetConnectionManagerClient(); dataFormatManager = streamingApiClient.GetDataFormatManagerClient(); } private async Task CreateConnection() { return await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "VehicleTelemetry", SessionKey = $"Session_{DateTime.UtcNow:yyyyMMdd_HHmmss}", Streams = { "Engine", "Brakes", "Suspension" }, StreamOffsets = { 0, 0, 0 }, ExcludeMainStream = false } }); } private async Task RegisterDataFormats() { // Register engine parameters await dataFormatManager.GetParameterDataFormatIdAsync(new GetParameterDataFormatIdRequest { DataSource = "VehicleTelemetry", Parameters = { "RPM", "Throttle", "FuelFlow", "OilPressure", "WaterTemp" } }); // Register brake parameters await dataFormatManager.GetParameterDataFormatIdAsync(new GetParameterDataFormatIdRequest { DataSource = "VehicleTelemetry", Parameters = { "BrakePressure", "BrakeTemp_FL", "BrakeTemp_FR", "BrakeTemp_RL", "BrakeTemp_RR" } }); // Register events await dataFormatManager.GetEventDataFormatIdAsync(new GetEventDataFormatIdRequest { DataSource = "VehicleTelemetry", Event = "LapCompleted" }); } private async Task StartReading(NewConnectionResponse connection) { var readStream = packetReader.ReadDataPackets(new ReadDataPacketsRequest { Request = new DataPacketRequest { Connection = connection.Connection, IncludeParameters = { "RPM", "Throttle", "BrakePressure" } } }); var cts = new CancellationTokenSource(); while (await readStream.ResponseStream.MoveNext(cts.Token)) { var response = readStream.ResponseStream.Current.Response; foreach (var packetResponse in response) { await ProcessTelemetryPacket(packetResponse); } } } private async Task WriteTelemetryData() { var sessionKey = $"Session_{DateTime.UtcNow:yyyyMMdd_HHmmss}"; var random = new Random(); // Simulate engine data for (int i = 0; i < 100; i++) { var engineData = CreateEngineData(random); await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "VehicleTelemetry", Stream = "Engine", SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(engineData), Type = "PeriodicData", IsEssential = false } } }); await Task.Delay(100); // 10Hz data rate } } private byte[] CreateEngineData(Random random) { // Create sample engine telemetry data var data = new { Timestamp = DateTime.UtcNow.Ticks, RPM = 6000 + random.Next(-500, 500), Throttle = random.NextDouble() * 100, FuelFlow = 15.5 + random.NextDouble() * 5, OilPressure = 4.2 + random.NextDouble() * 0.5, WaterTemp = 85 + random.Next(-5, 10) }; return System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(data); } private async Task ProcessTelemetryPacket(PacketResponse packetResponse) { Console.WriteLine($"Received packet from {packetResponse.Stream} at {packetResponse.SubmitTime}"); switch (packetResponse.Packet.Type) { case "PeriodicData": await ProcessPeriodicData(packetResponse); break; case "Event": await ProcessEvent(packetResponse); break; case "Marker": await ProcessMarker(packetResponse); break; } } private async Task CloseConnection(NewConnectionResponse connection) { await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = connection.Connection }); } } ``` ### Batch Writing Example ```csharp public async Task WriteBatchData() { var batchWriter = packetWriter.WriteDataPackets(); var sessionKey = "BatchSession_001"; try { // Write 1000 packets in batch for (int i = 0; i < 1000; i++) { await batchWriter.RequestStream.WriteAsync(new WriteDataPacketsRequest { Details = { new DataPacketDetails { DataSource = "HighFrequencyData", Stream = "Sensors", SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(GenerateSensorData(i)), Type = "PeriodicData" } } } }); } } finally { await batchWriter.RequestStream.CompleteAsync(); } } ``` ## API Reference For API reference, please refer to the [Stream API Reference](https://atlas.motionapplied.com/developer-resources/secu4/docs/) ### Error Handling The Stream API uses standard gRPC status codes for error reporting: ```csharp try { var result = await packetWriter.WriteDataPacketAsync(request); } catch (RpcException ex) { switch (ex.StatusCode) { case StatusCode.InvalidArgument: // Handle invalid request parameters break; case StatusCode.Unavailable: // Handle service unavailability break; case StatusCode.DeadlineExceeded: // Handle timeout break; default: // Handle other errors break; } } ``` ## Testing The Stream API includes comprehensive testing at multiple levels: ### Unit Tests Located in `MA.Streaming.UnitTests`, these test individual components: ```bash cd MA.Streaming.UnitTests dotnet test ``` ### Integration Tests Located in `MA.Streaming.IntegrationTests`, these test end-to-end scenarios: ```bash cd MA.Streaming.IntegrationTests dotnet test ``` Key integration test scenarios: - Topic-based packet writing and reading - Partition-based packet writing and reading - Batch operations - Data packet filtering - Session management - Connection management ### Sample Applications Example usage can be found in the [Sample Code](https://github.com/mat-docs/MA.Streaming.Api.UsageSample) repository. ## Deployment ### Standalone Deployment The Stream API can be deployed as a standalone executable or integrated into existing applications: ```csharp // Self-hosted deployment var host = Host.CreateDefaultBuilder(args) .ConfigureServices(services => { services.AddStreamingApi(configuration); }) .Build(); await host.RunAsync(); ``` ### Docker Deployment #### Production Docker Compose ```yaml version: '3.8' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_RETENTION_HOURS: 168 KAFKA_LOG_SEGMENT_BYTES: 1073741824 stream-api: image: atlasplatformdocker/streaming-proto-server-host:latest depends_on: - kafka ports: - "13579:13579"prot - "10010:10010" environment: CONFIG_PATH: /app/Configs/AppConfig.json AUTO_START: true volumes: - ./configs:/app/Configs ``` ### Monitoring The Stream API exposes Prometheus metrics on the configured port (default: 10010): ```yaml # Prometheus configuration scrape_configs: - job_name: 'stream-api' static_configs: - targets: ['stream-api:10010'] scrape_interval: 15s metrics_path: /metrics ``` Key metrics include: - `stream_api_connections_total`: Total number of active connections - `stream_api_packets_routed_total`: Total number of routed packets - `stream_api_sessions_active`: Number of active sessions - `stream_api_data_formats_registered`: Number of registered data formats ## Troubleshooting ### Common Issues #### Connection Failures ```csharp // Check Kafka broker connectivity var brokerChecker = new KafkaBrokerAvailabilityChecker(); bool isAvailable = await brokerChecker.CheckAsync("localhost:9092"); ``` #### Missing Data Formats ```csharp // Verify data format registration var formats = await dataFormatManager.GetParametersListAsync(new GetParametersListRequest { DataSource = "MyDataSource", DataFormatIdentifier = formatId }); ``` #### Performance Issues - Enable response batching for high-throughput scenarios - Use appropriate partition mapping for load distribution - Monitor Kafka broker performance and scaling #### Memory Management ```csharp // Properly dispose of streaming calls using var readStream = packetReader.ReadPackets(request); // Process data... ``` ## Support For technical support and questions: - **Documentation**: [Open Streaming Architecture Documentation](https://atlas.motionapplied.com/developer-resources/secu4/stream_api/) - **Sample Code**: [Usage Examples](https://github.com/mat-docs/MA.Streaming.Api.UsageSample) - **Protocol Definitions**: [MA.DataPlatforms.Protocol Repository](https://github.com/Software-Products/MA.DataPlatforms.Protocol) - **NuGet Packages**: [Motion Applied Packages](https://github.com/mat-docs/packages) --- © 2025 Motion Applied Ltd. All rights reserved. --- # Overview Welcome to the Motion Applied Stream API getting started guide. This section provides everything you need to begin working with our high-performance streaming platform. ## What You'll Learn - How to set up your development environment - Installing client libraries for your preferred language - Configuring the Stream API server - Creating your first streaming application ## Prerequisites Before getting started, ensure you have: - A supported development environment (.NET 8.0+, Python 3.8+, Node.js 16+, etc.) - Docker and Docker Compose for server deployment - Access to a Kafka broker (or use our Docker Compose setup) - Basic understanding of streaming data concepts ## Quick Navigation - **[Installation Guide](installation.md)**: Set up client libraries and dependencies - **[Quick Start](quick-start.md)**: Build your first streaming application - **[Server Configuration](../configuration/server-config.md)**: Configure the Stream API server - **[Examples](../examples/telemetry-example.md)**: Explore real-world examples ## Architecture Overview The Stream API provides five core services: 1. **Session Management**: Handle data session lifecycle 2. **Connection Management**: Manage client connections 3. **Packet Writer**: Publish data to streams 4. **Packet Reader**: Consume data from streams 5. **Data Format Management**: Define and manage schemas ## Next Steps Ready to get started? Follow our [installation guide](installation.md) to set up your development environment, then jump into the [quick start tutorial](quick-start.md) to build your first application. --- # Quick Start Get up and running with the Motion Applied Stream API in minutes. This guide walks you through setting up your first data streaming application. ## Prerequisites - .NET 8.0 or later (for C# clients) - Docker and Docker Compose (for server deployment) - Access to a Kafka broker ## Step 1: Start the Server ### Using Docker Compose Create a `docker-compose.yml` file: ```yaml services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ports: - 12181:2181 kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 stream-api-server: image: atlasplatformdocker/streaming-proto-server-host:latest ports: - 13579:13579 - 10010:10010 depends_on: - kafka restart: always environment: CONFIG_PATH: /Configs/AppConfig.json AUTO_START: true volumes: - ./Configs:/app/Configs ``` Create the configuration directory and file: ```bash mkdir Configs ``` Create `Configs/AppConfig.json`: ```json { "StreamCreationStrategy": 2, "BrokerUrl": "kafka:9092", "PartitionMappings": [], "StreamApiPort": 13579, "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "BatchingResponses": false, "PrometheusMetricPort": 10010 } ``` Start the services: ```bash docker-compose up -d ``` ## Step 2: Install Client Library ### For C# Applications ```xml ``` ### For Other Languages Download the proto files from the [Protocol Repository](https://github.com/Software-Products/MA.DataPlatforms.Protocol) and generate client code for your language. ## Step 3: Create Your First Application ### Simple Example Here's a minimal example to get you started: ```csharp using MA.Streaming.API; using MA.Streaming.Proto.Client.Local; using MA.Streaming.Core; using MA.Streaming.Core.Configs; using Google.Protobuf; // Initialize the client var config = new StreamingApiConfiguration( StreamCreationStrategy.TopicBased, "localhost:9092", []); var streamingApiClient = StreamingApiClientFactory.Create( config, new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider(@"C:\Temp")); streamingApiClient.Initialise(); // Get service clients var sessionManager = streamingApiClient.GetSessionManagementClient(); var connectionManager = streamingApiClient.GetConnectionManagerClient(); var packetWriter = streamingApiClient.GetPacketWriterClient(); var packetReader = streamingApiClient.GetPacketReaderClient(); const string DataSource = "QuickStartDemo"; // Create a session var sessionResponse = await sessionManager.CreateSessionAsync(new CreateSessionRequest { DataSource = DataSource, Type = "DemoSession", Version = 1, Identifier = "quick-start-001" }); string sessionKey = sessionResponse.SessionKey; Console.WriteLine($"Created session: {sessionKey}"); // Create a connection var connectionResponse = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = DataSource, SessionKey = sessionKey, Streams = { "demo-stream" }, StreamOffsets = { 0 } // Start from beginning } }); Console.WriteLine($"Connected with ID: {connectionResponse.Connection.Id}"); // Write a simple packet (using custom message - see Custom Messages example) var customData = new { Message = "Hello Stream API", Timestamp = DateTime.UtcNow }; string jsonData = System.Text.Json.JsonSerializer.Serialize(customData); byte[] jsonBytes = System.Text.Encoding.UTF8.GetBytes(jsonData); await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = DataSource, Stream = "demo-stream", SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(jsonBytes), Type = "CustomMessage", IsEssential = false, SessionKey = sessionKey } } }); Console.WriteLine("Sent packet successfully!"); // Read packets var stream = packetReader.ReadPackets(new ReadPacketsRequest { Connection = connectionResponse.Connection }); await foreach (var response in stream.ResponseStream.ReadAllAsync()) { foreach (var packet in response.Response) { Console.WriteLine($"Received: {packet.Packet.Type} from {packet.Stream}"); } } // Clean up await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = connectionResponse.Connection }); await sessionManager.EndSessionAsync(new EndSessionRequest { DataSource = DataSource, SessionKey = sessionKey }); Console.WriteLine("Demo completed!"); ``` ### Run the Application ```bash dotnet run ``` You should see output showing your session being created, packet sent, and data received. ## Complete Examples For more comprehensive examples with real-world scenarios: - **[Telemetry Example](../examples/telemetry-example.md)** - Complete vehicle telemetry streaming with Protocol Buffers - **[Custom Messages Example](../examples/custom-messages.md)** - Send and receive custom message types with JSON serialization ## Next Steps - [Explore the Connection Management Service](../services/connection-management.md) - [Learn about Session Management](../services/session-management.md) - [View more examples](../examples/telemetry-example.md) - [Docker Setup Guide](../configuration/docker-setup.md) --- # Installation This guide covers installing the Stream API client libraries and setting up your development environment. ## Client Libraries ### .NET / C# Install the NuGet package: ```bash dotnet add package MA.Streaming.Proto.Client.Local ``` Or add to your `.csproj` file: ```xml ``` ### Other Languages For other programming languages, download the Protocol Buffer definitions from our [Protocol Repository](https://github.com/Software-Products/MA.DataPlatforms.Protocol) and generate client code using the protoc compiler. Example for Python: ```bash # Install required packages pip install grpcio grpcio-tools # Clone the protocol repository git clone https://github.com/Software-Products/MA.DataPlatforms.Protocol.git cd MA.DataPlatforms.Protocol # Generate Python client code python -m grpc_tools.protoc --python_out=. --grpc_python_out=. --proto_path=. *.proto ``` ## Server Setup ### Docker Deployment (Recommended) The easiest way to run the Stream API server is using Docker: ```bash docker pull atlasplatformdocker/streaming-proto-server-host:latest ``` ### Self-Hosted Deployment Download the server binaries from our releases page or build from source: ```bash git clone https://github.com/Software-Products/MA.DataPlatforms.StreamApi.git cd MA.DataPlatforms.StreamApi/MA.Streaming dotnet build -c Release ``` ### Embedded Deployment If you're embedding the Stream API server directly into your application, you don't need to host the server separately on Docker or as a standalone service. Simply reference the server components in your project and initialize them within your application code. ```xml ``` When embedding the server, the Stream API runs as part of your application process, eliminating the need for external server deployment. ## Verification Verify your installation by running a simple test: ```csharp using MA.Streaming.Proto.Client.Remote; var configuration = new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []); var streamingApiClient = StreamingApiClientFactory.Create( configuration, new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider(@"C:\Temp")); streamingApiClient.Initialise(); var connectionManager = streamingApiClient.GetConnectionManagerClient(); // Test basic connectivity var connections = await connectionManager.GetConnection(); ``` ## Next Steps - [Quick Start Guide](quick-start.md): Build your first application - [Server Configuration](../configuration/server-config.md): Configure the server - [Docker Setup](../configuration/docker-setup.md): Docker deployment guide --- # Session Management The Session Management Service handles the complete lifecycle of data sessions, including creation, monitoring, and termination. ## Overview Sessions represent logical groupings of data streams with associated metadata. The service provides: - Session creation with custom metadata - Real-time session state tracking - Automatic session cleanup and resource management - Session information updates during runtime ## Key Features ### Session Lifecycle Management - **Creation**: Initialize new sessions with metadata - **Monitoring**: Track session state and activity - **Updates**: Modify session information during runtime - **Termination**: Clean shutdown with proper resource cleanup ### Metadata Support - Custom session types and versions - Associated participant information - Dynamic key-value properties - Hierarchical session relationships ### Integration - Automatic integration with other Stream API services - Event-driven architecture for real-time updates - Optional Prometheus metrics for monitoring (requires admin privileges) ## Usage Examples ### Basic Session Operations ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); // Create a new session var newSessionResponse = sessionManager.CreateSession(new CreateSessionRequest { DataSource = DataSource, Type = "Session", Version = 1, UtcOffset = new Google.Protobuf.WellKnownTypes.Duration { Seconds = 0 }, Identifier = "silverstone-gp-2024", Details = { {"track", "Silverstone Circuit"}, {"weather", "dry"}, {"temperature", "22°C"} } }); ``` ### Session Information Updates ```csharp // Update session information during runtime await sessionManager.UpdateSessionDetailsAsync(new UpdateSessionDetailsRequest { SessionKey = "race-session-2024-001", Details = { {"status", "active"}, {"participant", "driver-hamilton"}, {"lap_count", "15"}, {"sector_time", "1:23.456"} } }); ``` ### Session Termination ```csharp // End session with proper cleanup await sessionManager.EndSessionAsync(new EndSessionRequest { SessionKey = "race-session-2024-001", DataSource = "DataSource" }); ``` ## Advanced Features ### Session Hierarchies ```csharp // Create child sessions var associateSession = await sessionManager.CreateSessionAsync( new CreateSessionRequest { DataSource = "DataSource", Details = { { "track", "Silverstone Circuit" }, { "weather", "dry" }, { "temperature", "22°C" } }, Identifier = "associate_session", Type = "Session", UtcOffset = new Duration { Seconds = 0 }, Version = 1 }); // Create a parent session var parentSession = await sessionManager.CreateSessionAsync( new CreateSessionRequest { DataSource = "DataSource", Details = { { "track", "Silverstone Circuit" }, { "weather", "dry" }, { "temperature", "22°C" } }, Identifier = "main_sesssion", Type = "Session", UtcOffset = new Duration { Seconds = 0 }, Version = 1, AssociateSessionKey = { associateSession.SessionKey } }); ``` ### Session Queries ```csharp // Get session information var sessionInfo = await sessionManager.GetSessionInfoAsync( new GetSessionInfoRequest { SessionKey = "session_key" }); // List all sessions var activeSessions = await sessionManager.GetCurrentSessionsAsync(new GetCurrentSessionsRequest() { DataSource = "DataSource" }); ``` ### Session Notifications The Session Management Service provides real-time notification streams that push updates whenever sessions are created or stopped. These notification methods create persistent streams that remain open and deliver notifications as session events occur. #### Session Start Notifications Subscribe to receive notifications when new sessions are created for a specific data source: ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); // Create a streaming call to receive session start notifications var startNotificationStream = sessionManager.GetSessionStartNotification( new GetSessionStartNotificationRequest { DataSource = "DataSource" }); // Process notifications as they arrive await foreach (var notification in startNotificationStream.ResponseStream.ReadAllAsync()) { Console.WriteLine($"New session started:"); Console.WriteLine($" Data Source: {notification.DataSource}"); Console.WriteLine($" Session Key: {notification.SessionKey}"); // Handle the new session (e.g., create connections, start monitoring) await HandleNewSession(notification.SessionKey, notification.DataSource); } ``` #### Session Stop Notifications Subscribe to receive notifications when sessions are terminated: ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); // Create a streaming call to receive session stop notifications var stopNotificationStream = sessionManager.GetSessionStopNotification( new GetSessionStopNotificationRequest { DataSource = "DataSource" }); // Process notifications as they arrive await foreach (var notification in stopNotificationStream.ResponseStream.ReadAllAsync()) { Console.WriteLine($"Session stopped:"); Console.WriteLine($" Data Source: {notification.DataSource}"); Console.WriteLine($" Session Key: {notification.SessionKey}"); // Handle session termination (e.g., cleanup resources, close connections) await HandleSessionTermination(notification.SessionKey, notification.DataSource); } ``` #### Combined Notification Monitoring Monitor both session start and stop events concurrently: ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); var cancellationToken = new CancellationToken(); // Start monitoring session lifecycle var startTask = Task.Run(async () => { var stream = sessionManager.GetSessionStartNotification( new GetSessionStartNotificationRequest { DataSource = "DataSource" }); await foreach (var notification in stream.ResponseStream.ReadAllAsync(cancellationToken)) { await OnSessionStarted(notification.SessionKey, notification.DataSource); } }, cancellationToken); var stopTask = Task.Run(async () => { var stream = sessionManager.GetSessionStopNotification( new GetSessionStopNotificationRequest { DataSource = "DataSource" }); await foreach (var notification in stream.ResponseStream.ReadAllAsync(cancellationToken)) { await OnSessionStopped(notification.SessionKey, notification.DataSource); } }, cancellationToken); // Wait for both monitoring tasks await Task.WhenAll(startTask, stopTask); ``` #### Notification Behavior - **Persistent Streams**: Once established, notification streams remain open and continuously deliver events - **Real-time Delivery**: Notifications are pushed immediately when sessions are created or stopped - **Data Source Filtering**: Each notification stream is scoped to a specific data source - **No Historical Events**: Only new events that occur after the stream is established are delivered - **Automatic Reconnection**: If the stream is disconnected, the client must re-establish the connection to continue receiving notifications #### Use Cases Session notifications are particularly useful for: - **Session Discovery**: Automatically detect and connect to new sessions as they are created - **Resource Management**: Clean up resources when sessions terminate - **Monitoring Dashboards**: Display real-time session activity - **Automated Testing**: Coordinate test scenarios based on session lifecycle events - **Multi-client Coordination**: Synchronize multiple clients working with the same data sources ## Configuration ### Enable Session Management ```json { "IntegrateSessionManagement": true, } ``` ## Best Practices ### Session Key - Session Key is automatically allocated when the session is created. - The session key can be retreived by taking the result of the CreateNewSessionRequest call. ```csharp var newSessionResponse = sessionManager.CreateSession(new CreateSessionRequest { DataSource = "DataSource", Type = "Session", Version = 1, UtcOffset = new Google.Protobuf.WellKnownTypes.Duration { Seconds = 0 }, Identifier = "silverstone-gp-2024", Details = { {"track", "Silverstone Circuit"}, {"weather", "dry"}, {"temperature", "22°C"} } }); var sessionKey = newSession.SessionKey; ``` ## Monitoring (Optional) ### Metrics If Prometheus is enabled with admin privileges, the service exposes metrics: - `stream_api_sessions_total`: Number of sessions available ## See Also - [Connection Management Service](connection-management.md) - [Configuration Guide](../configuration/server-config.md) - [Telemetry Example](../examples/telemetry-example.md) --- # Connection Management The Connection Management Service handles client connections to sessions in the broker. ## Overview Connections represent individual client access points to data streams. The service provides: - Unique connection ID assignment for tracking - Connection validation and authentication - Real-time connection state management - Automatic cleanup of inactive connections ## Key Features ### Connection Lifecycle - **Creation**: Establish new connections with validation - **Tracking**: Monitor connection state and activity - **Validation**: Ensure connection integrity - **Cleanup**: Automatic resource deallocation ### Stream Management - Multiple stream subscription per connection - Flexible offset management for each stream - Main stream inclusion/exclusion options - Dynamic stream addition and removal ### Resource Optimization - Connection pooling for efficiency - Automatic timeout handling - Memory-efficient connection tracking - Optional Prometheus metrics integration (requires admin privileges) ## Usage Examples ### Basic Connection Operations ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var connectionManager = streamingApiClient.GetConnectionManagerClient(); // Create new connection var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "vehicle-telemetry", SessionKey = "race-session-2024-001", Streams = { "engine", "brakes", "suspension", "aerodynamics" }, StreamOffsets = { 0, 0, 0, 0 }, // Start from beginning ExcludeMainStream = false } }); Console.WriteLine($"Connection created with ID: {connection.Connection.Id}"); ``` ### Connection Information Retrieval ```csharp // Retrieve connection details var connectionInfo = await connectionManager.GetConnectionAsync(new GetConnectionRequest { Connection = new Connection { Id = connectionId } }); if (connectionInfo.Details != null) { Console.WriteLine($"Data Source: {connectionInfo.Details.DataSource}"); Console.WriteLine($"Session: {connectionInfo.Details.SessionKey}"); Console.WriteLine($"Streams: {string.Join(", ", connectionInfo.Details.Streams)}"); } ``` ### Connection Termination ```csharp // Close connection properly var closeResult = await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = new Connection { Id = connectionId } }); if (closeResult.Success) { Console.WriteLine("Connection closed successfully"); } else { Console.WriteLine("Failed to close connection"); } ``` ## Advanced Configuration ### Stream Offset Management ```csharp // Connect with specific offsets for each stream var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "historical-data", SessionKey = "replay-session-001", Streams = { "engine", "brakes" }, StreamOffsets = { 1000, 500 }, // Resume from specific positions ExcludeMainStream = true // Skip main data stream } }); ``` ### Multi-Stream Subscriptions ```csharp // Subscribe to multiple related streams var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "comprehensive-telemetry", SessionKey = "test-session-001", Streams = { "engine-rpm", "engine-temp", "engine-pressure", "brake-temp-fl", "brake-temp-fr", "brake-temp-rl", "brake-temp-rr", "suspension-travel-fl", "suspension-travel-fr" }, StreamOffsets = new long[9], // All zeros - start from beginning ExcludeMainStream = false } }); ``` ### Connection Validation ```csharp // Validate connection before use private async Task ValidateConnection(long connectionId) { try { var connectionInfo = await connectionManager.GetConnectionAsync(new GetConnectionRequest { Connection = new Connection { Id = connectionId } }); return connectionInfo.Details != null; } catch (RpcException ex) when (ex.StatusCode == StatusCode.NotFound) { return false; } } ``` ## Connection Patterns ### Single-Use Connections ```csharp // Pattern for short-lived connections public async Task ProcessQuickData() { NewConnectionResponse connection = null; try { connection = await connectionManager.NewConnectionAsync(request); // Process data quickly await ProcessData(connection.Connection); } finally { if (connection != null) { await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = connection.Connection }); } } } ``` ### Long-Running Connections ```csharp // Pattern for persistent connections public class TelemetryProcessor { private Connection? activeConnection; private readonly ConnectionManagerService.ConnectionManagerServiceClient connectionManager; public async Task StartProcessing() { var connectionResponse = await connectionManager.NewConnectionAsync(request); activeConnection = connectionResponse.Connection; // Long-running processing... } public async Task StopProcessing() { if (activeConnection != null) { await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = activeConnection }); activeConnection = null; } } } ``` ### Connection Pooling ```csharp // Connection pool implementation public class ConnectionPool { private readonly ConcurrentQueue availableConnections = new(); private readonly ConnectionManagerService.ConnectionManagerServiceClient connectionManager; public async Task GetConnection() { if (availableConnections.TryDequeue(out var connection)) { // Validate connection is still active if (await ValidateConnection(connection.Id)) { return connection; } } // Create new connection var newConnection = await connectionManager.NewConnectionAsync(request); return newConnection.Connection; } public void ReturnConnection(Connection connection) { availableConnections.Enqueue(connection); } } ``` ## Configuration (Optional) ### Connection Metrics If Prometheus is enabled with admin privileges, monitor connection health: - `stream_api_connections_total`: Number of active connections in the stream API - `stream_api_data_packets_read_total`: Number of packets read and ready to deliver (by connection, data source, stream) - `stream_api_data_packets_delivered_total`: Number of packets delivered (by connection, data source, stream) - `stream_api_essential_packets_read_total`: Number of essential packets read (by connection, data source) - `stream_api_essential_packets_delivered_total`: Number of essential packets delivered (by connection, data source) - `stream_api_router_messages_received_total`: Total messages received from router (by connection, data source, stream, session key) - `stream_api_router_messages_received_bytes_total`: Total message bytes received from router (by connection, data source, stream, session key) ## Error Handling ### Common Connection Errors ```csharp try { var connection = await connectionManager.NewConnectionAsync(request); } catch (RpcException ex) { switch (ex.StatusCode) { case StatusCode.InvalidArgument: // Invalid connection parameters Console.WriteLine("Invalid connection request"); break; case StatusCode.ResourceExhausted: // Too many connections Console.WriteLine("Connection limit reached"); break; case StatusCode.FailedPrecondition: // Session not found or invalid Console.WriteLine("Session validation failed"); break; case StatusCode.Unavailable: // Service temporarily unavailable Console.WriteLine("Service unavailable, retry later"); break; default: Console.WriteLine($"Unexpected error: {ex.Message}"); break; } } ``` ### Connection Recovery ```csharp // Automatic connection recovery public async Task GetOrCreateConnection() { if (currentConnection != null) { try { // Test connection await connectionManager.GetConnectionAsync(new GetConnectionRequest { Connection = currentConnection }); return currentConnection; // Connection is valid } catch (RpcException ex) when (ex.StatusCode == StatusCode.NotFound) { // Connection lost, create new one currentConnection = null; } } // Create new connection var newConnection = await connectionManager.NewConnectionAsync(connectionRequest); currentConnection = newConnection.Connection; return currentConnection; } ``` ## Best Practices ### Resource Management - Always close connections when done - Monitor connection count and duration ### Error Handling - Implement retry logic for transient failures - Validate connections before critical operations - Handle connection loss gracefully ### Performance - Reuse connections when possible - Use connection pooling for high-frequency access ## See Also - [Session Management Service](session-management.md) - [Packet Reader Service](packet-reader.md) - [Configuration Guide](../configuration/server-config.md) --- # Packet Writer The Packet Writer Service enables high-performance data publishing to Kafka topics with support for both individual packets and batch operations. ## Overview The service provides multiple writing modes optimized for different use cases: - **Single packet writing**: For low-frequency, critical data - **Batch operations**: For high-throughput scenarios - **Essential packet handling**: For critical data that requires special routing - **Stream-based routing**: Automatic topic/partition selection ## Key Features ### Writing Modes - Individual packet writing with immediate publishing - Batch writing for optimal throughput - Streaming writes for continuous data flows - Essential packet routing for critical data ### Performance Optimization - Asynchronous operations for non-blocking writes - Internal buffering for batch efficiency - Connection pooling and reuse - Optional Prometheus metrics for monitoring (requires admin privileges) ## Usage Examples ### Single Packet Writing ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var packetWriter = streamingApiClient.GetPacketWriterClient(); // Write individual telemetry packet await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "vehicle-telemetry", Stream = "engine", SessionKey = "race-session-2024-001", Message = new Packet { Content = ByteString.CopyFrom(engineData), Type = "PeriodicData", IsEssential = false } } }); ``` ### Batch Writing ```csharp // High-throughput batch writing var batchStream = packetWriter.WriteDataPackets(); try { for (int i = 0; i < 1000; i++) { await batchStream.RequestStream.WriteAsync(new WriteDataPacketsRequest { Details = { new DataPacketDetails { DataSource = "high-frequency-sensors", Stream = "accelerometers", SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(GenerateSensorData(i)), Type = "PeriodicData" } } } }); } } finally { await batchStream.RequestStream.CompleteAsync(); } ``` ### Essential Packet Writing ```csharp // Write critical system events await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "system-events", Stream = "", // Writes to the main stream SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(criticalEventData), Type = "Event", IsEssential = true // Ensures special routing and persistence } } }); ``` ## Advanced Usage ### Multi-Stream Writing ```csharp // Write to multiple streams simultaneously var tasks = new List(); // Engine data tasks.Add(packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "telemetry", Stream = "engine", SessionKey = sessionKey, Message = CreateEnginePacket(engineData) } })); // Brake data tasks.Add(packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "telemetry", Stream = "brakes", SessionKey = sessionKey, Message = CreateBrakePacket(brakeData) } })); // Wait for all writes to complete await Task.WhenAll(tasks); ``` ### Structured Data Writing ```csharp // Write structured telemetry data public async Task WriteVehicleState(VehicleState state) { var packet = new Packet { Content = ByteString.CopyFrom(JsonSerializer.SerializeToUtf8Bytes(new { timestamp = state.Timestamp.Ticks, vehicle_id = state.VehicleId, position = new { latitude = state.Position.Latitude, longitude = state.Position.Longitude, altitude = state.Position.Altitude }, dynamics = new { speed = state.Speed, heading = state.Heading, acceleration = state.Acceleration } })), Type = "VehicleState", SessionKey = state.SessionKey }; await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "vehicle-tracking", Stream = "position", SessionKey = state.SessionKey, Message = packet } }); } ``` ### High-Performance Streaming ```csharp // Continuous data streaming pattern public class ContinuousDataStreamer { private readonly PacketWriterService.PacketWriterServiceClient packetWriter; private AsyncClientStreamingCall? streamWriter; public async Task StartStreaming() { streamWriter = packetWriter.WriteDataPackets(); // Continuous streaming loop while (isStreaming) { var data = await GetNextDataPoint(); await streamWriter.RequestStream.WriteAsync(new WriteDataPacketsRequest { Details = { new DataPacketDetails { DataSource = "continuous-stream", Stream = "sensor-data", SessionKey = sessionKey, Message = new Packet { Content = ByteString.CopyFrom(data), Type = "PeriodicData" } } } }); } await streamWriter.RequestStream.CompleteAsync(); } } ``` ## Performance Optimization ### Batching Strategies ```csharp // Time-based batching public class TimeBatchedWriter { private readonly List batch = new(); private readonly Timer batchTimer; public TimeBatchedWriter() { batchTimer = new Timer(FlushBatch, null, TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(100)); } public void QueueWrite(WriteDataPacketsRequest request) { lock (batch) { batch.Add(request); // Size-based flushing if (batch.Count >= 100) { FlushBatch(null); } } } private async void FlushBatch(object? state) { List toFlush; lock (batch) { if (batch.Count == 0) return; toFlush = new List(batch); batch.Clear(); } var batchStream = packetWriter.WriteDataPackets(); try { foreach (var request in toFlush) { await batchStream.RequestStream.WriteAsync(request); } } finally { await batchStream.RequestStream.CompleteAsync(); } } } ``` ### Connection Reuse ```csharp // Reuse streaming connections for better performance public class OptimizedPacketWriter { private AsyncClientStreamingCall? currentStream; private readonly SemaphoreSlim streamLock = new(1, 1); public async Task WritePacketOptimized(WriteDataPacketsRequest request) { await streamLock.WaitAsync(); try { if (currentStream == null) { currentStream = packetWriter.WriteDataPackets(); } await currentStream.RequestStream.WriteAsync(request); } finally { streamLock.Release(); } } public async Task FlushAndClose() { await streamLock.WaitAsync(); try { if (currentStream != null) { await currentStream.RequestStream.CompleteAsync(); currentStream = null; } } finally { streamLock.Release(); } } } ``` ## Error Handling ### Retry Mechanisms ```csharp // Robust writing with retry logic public async Task WriteWithRetry(WriteDataPacketRequest request, int maxRetries = 3) { for (int attempt = 0; attempt <= maxRetries; attempt++) { try { await packetWriter.WriteDataPacketAsync(request); return; // Success } catch (RpcException ex) when (attempt < maxRetries) { switch (ex.StatusCode) { case StatusCode.Unavailable: case StatusCode.DeadlineExceeded: // Transient errors - retry await Task.Delay(TimeSpan.FromMilliseconds(100 * Math.Pow(2, attempt))); break; default: // Non-retryable error throw; } } } } ``` ### Error Recovery ```csharp // Handle streaming errors gracefully public async Task WriteStreamWithRecovery(IEnumerable requests) { var requestQueue = new Queue(requests); while (requestQueue.Count > 0) { var stream = packetWriter.WriteDataPackets(); try { // Write batch while (requestQueue.Count > 0) { var request = requestQueue.Dequeue(); await stream.RequestStream.WriteAsync(request); } await stream.RequestStream.CompleteAsync(); } catch (RpcException ex) { // Log error and retry remaining requests Console.WriteLine($"Stream error: {ex.Message}, retrying {requestQueue.Count} requests"); await Task.Delay(1000); // Wait before retry } } } ``` ## Monitoring (Optional) ### Performance Metrics If Prometheus is enabled with admin privileges, the following metrics are available: - `stream_api_data_packets_published_total`: Number of data packets published (by data source, stream) - `stream_api_info_packets_published_total`: Number of info packets published (by data source) - `stream_api_data_packets_routed_total`: Number of data packets routed by router (by data source, stream) - `stream_api_data_packets_routed_bytes_total`: Total bytes of data packets routed (by data source, stream) - `stream_api_info_packets_routed_total`: Number of info packets routed by router (by data source) - `stream_api_info_packets_routed_bytes_total`: Total bytes of info packets routed (by data source) Example usage: ```csharp // Metrics are automatically tracked by the service when Prometheus is enabled public class MetricsCollector { private readonly Counter packetsWritten = Metrics .CreateCounter("packets_written_total", "Total packets written"); private readonly Histogram writeLatency = Metrics .CreateHistogram("packet_write_duration_seconds", "Packet write latency"); public async Task WriteWithMetrics(WriteDataPacketRequest request) { using var timer = writeLatency.NewTimer(); try { await packetWriter.WriteDataPacketAsync(request); packetsWritten.Inc(); } catch { // Handle errors throw; } } } ``` ## Best Practices ### Data Organization - Use consistent packet types for similar data - Include timestamps in packet content - Validate data before writing - Use appropriate stream names for logical grouping ### Performance - Use batch writing for high-frequency data - Implement appropriate retry mechanisms - Monitor write latency and throughput - Consider data compression for large packets ### Reliability - Mark critical data as essential - Implement proper error handling - Use structured logging for debugging - Monitor system resources - Ensure to write all data before closing the session. ## See Also - [Packet Reader Service](packet-reader.md) - [Data Format Management](data-format-management.md) --- # Packet Reader The Packet Reader Service provides high-performance data consumption from Kafka topics with support for real-time streaming and historical data replay. ## Overview The service offers flexible data consumption modes: - **All packets**: Continuous consumption of all packet types via `ReadPackets` - **Essential packets**: Read only configuration and metadata via `ReadEssentials` - **Filtered data packets**: Consume specific parameters using regex patterns via `ReadDataPackets` ## Key Features ### Reading Modes - **ReadPackets**: Continuous streaming of all packet types - **ReadEssentials**: Essential packets only (config, metadata) - **ReadDataPackets**: Filtered data packets using regex patterns ### Performance - Low-latency data delivery - Efficient memory usage - Connection pooling - Backpressure handling ## Usage Examples ### Basic Stream Reading ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var packetReader = streamingApiClient.GetPacketReaderClient(); // Start reading all packets from a connection var readRequest = new ReadPacketsRequest { Connection = connection.Connection }; var stream = packetReader.ReadPackets(readRequest); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; foreach (var packet in response.Response) { Console.WriteLine($"Received: {packet.Packet.Type} from {packet.Stream}"); ProcessPacket(packet.Packet); } } ``` ### Reading Essential Packets ```csharp // Read only essential packets (configuration, metadata, etc.) var essentialsRequest = new ReadEssentialsRequest { Connection = connection.Connection }; var stream = packetReader.ReadEssentials(essentialsRequest); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; foreach (var packet in response.Response) { Console.WriteLine($"Essential packet: {packet.Packet.Type}"); ProcessEssentialPacket(packet.Packet); } } ``` ### Filtered Data Packet Reading ```csharp // Read data packets with specific parameters (supports regex patterns) var dataPacketRequest = new ReadDataPacketsRequest { Request = new DataPacketRequest { Connection = connection.Connection, IncludeParameters = { "vCar.*", ".*Temp.*", "NGear.*" }, ExcludeParameters = { ".*Debug.*" } } }; var stream = packetReader.ReadDataPackets(dataPacketRequest); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; foreach (var packet in response.Response) { // Only data packets matching the filter ProcessDataPacket(packet.Packet); } } ``` ## Advanced Features ### Parameter Filtering with Regex ```csharp // Include all engine-related parameters, exclude calibration data var dataRequest = new ReadDataPacketsRequest { Request = new DataPacketRequest { Connection = connection.Connection, IncludeParameters = { "NEngine.*", // All engine parameters "TEngine.*", // Engine temperature "PEngine.*" // Engine pressure }, ExcludeParameters = { ".*Calibration.*", // Exclude calibration ".*Raw.*" // Exclude raw data } } }; var stream = packetReader.ReadDataPackets(dataRequest); ``` ### Combined Reading Strategies ```csharp // Read essentials and filtered data packets concurrently var essentialsTask = Task.Run(async () => { var stream = packetReader.ReadEssentials(new ReadEssentialsRequest { Connection = connection.Connection }); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; foreach (var packet in response.Response) { ProcessEssentialPacket(packet.Packet); } } }); var dataTask = Task.Run(async () => { var stream = packetReader.ReadDataPackets(new ReadDataPacketsRequest { Request = new DataPacketRequest { Connection = connection.Connection, IncludeParameters = { "vCar.*", "NGear.*" } } }); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; foreach (var packet in response.Response) { ProcessDataPacket(packet.Packet); } } }); await Task.WhenAll(essentialsTask, dataTask); ``` ## Best Practices - Handle backpressure appropriately - Implement proper error recovery - Monitor consumer lag - Use appropriate buffer sizes ## See Also - [Packet Writer Service](packet-writer.md) - [Connection Management](connection-management.md) - [Examples](../examples/telemetry-example.md) --- # Data Format Management The Data Format Management Service handles registering and getting data formats for parameter and event packets. ## Overview This service provides: - **Data Format Registration** Register new data formats. - **Get Data Format Definitions** Allows to get data format definitions when needed. ## Key Features ### Manage Data Formats. - Register new data formats. - Get defined data formats based on the data format identifier. ## Usage Examples ### Register Data Format #### Parameter List ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var formatManager = streamingApiClient.GetDataFormatManagerClient(); var parameterListFormatRequest = new GetParameterDataFormatIdRequest { DataSource = DataSource, Parameters = { "vCar:Chassis", "NGear:Chassis", "NEngine:FIA" } }; var response = await formatManager.GetParameterDataFormatIdAsync(parameterListFormatRequest); var dataFormatIdentifier = response.DataFormatIdentifier; ``` #### Event Identifier ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var formatManager = streamingApiClient.GetDataFormatManagerClient(); var formatRequest = new GetEventDataFormatIdRequest { DataSource = DataSource, Event = "RaceEvent" }; var response = await formatManager.GetEventDataFormatIdAsync(formatRequest); var dataFormatIdentifier = response.DataFormatIdentifier; ``` ### Get Data Format Definition #### Parameter List ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var formatManager = streamingApiClient.GetDataFormatManagerClient(); var dataFormatIdentifier = 123456UL; var dataFormatResponse = await formatManager.GetParametersListAsync(new GetParametersListRequest { DataSource = DataSource, DataFormatIdentifier = dataFormatIdentifier }); var parameterList = dataFormatResponse.Parameters; ``` #### Event Identifier ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var formatManager = streamingApiClient.GetDataFormatManagerClient(); var dataFormatIdentifier = 123456UL; var dataFormatResponse = await formatManager.GetEventAsync(new GetEventRequest { DataFormatIdentifier = dataFormatIdentifier, DataSource = DataSource }); var eventIdentifier = dataFormatResponse.Event; ``` ## Best Practices - Cache responses as much as you can. Helps reduce grpc calls. - Try to reuse as much data formats as you can. ## See Also - [Packet Writer Service](packet-writer.md) - [Packet Reader Service](packet-reader.md) - [Configuration Guide](../configuration/server-config.md) --- # Server Configuration The MA Streaming API provides a flexible way to stream data using Kafka as the messaging backbone. The API supports two streaming strategies: **Topic-Based** and **Partition-Based**, each suited for different use cases and scalability requirements. ## Configuration The Streaming API is configured through the `IStreamingApiConfiguration` interface. ### Interface Definition ```csharp public interface IStreamingApiConfiguration { StreamCreationStrategy StreamCreationStrategy { get; } string BrokerUrl { get; } IReadOnlyList? PartitionMappings { get; } int StreamApiPort { get; } bool IntegrateSessionManagement { get; } bool IntegrateDataFormatManagement { get; } bool UseRemoteKeyGenerator { get; } string RemoteKeyGeneratorServiceAddress { get; } bool BatchingResponses { get; } uint InitialisationTimeoutSeconds { get; } uint TerminationTimeoutSeconds { get; } string Domain { get; } bool EnableRouterPublishMetrics { get; } bool EnableRouterConsumeMetrics { get; } string KafkaBrokerPublishingConfigFilePath { get; } string KafkaBrokerConsumingConfigFilePath { get; } string KafkaBrokerConfigWhitelistFilePath { get; } KafkaSecurityConfiguration? Security { get; } } ``` ### Configuration Properties | Property | Description | Required | Default Value | |----------|-------------|----------|---------------| | `StreamCreationStrategy` | Topic-Based or Partition-Based streaming | Yes | TopicBased | | `BrokerUrl` | Kafka broker connection string | Yes | "localhost:9092" | | `PartitionMappings` | Partition mapping (Partition-Based only) | Yes* | null | | `StreamApiPort` | Port number for the Streaming API server | No | 5000 | | `IntegrateSessionManagement` | Enable/disable session management | No | true | | `IntegrateDataFormatManagement` | Enable/disable data format management | No | true | | `UseRemoteKeyGenerator` | Use remote key generator service | No | false | | `RemoteKeyGeneratorServiceAddress` | Address of remote key generator | No | "" | | `BatchingResponses` | Enable response batching | No | true | | `InitialisationTimeoutSeconds` | Timeout for initialization operations | No | 30 | | `TerminationTimeoutSeconds` | Timeout for graceful termination | No | 1 | | `Domain` | Domain prefix for Kafka topics | No | "" | | `EnableRouterPublishMetrics` | Enable Prometheus metrics on the publish side | No | false | | `EnableRouterConsumeMetrics` | Enable Prometheus metrics on the consume side | No | false | | `KafkaBrokerPublishingConfigFilePath` | Path to the publisher broker override file | No | "kafka-publisher-broker.json" | | `KafkaBrokerConsumingConfigFilePath` | Path to the consumer broker override file | No | "kafka-consumer-broker.json" | | `KafkaBrokerConfigWhitelistFilePath` | Path to the broker override whitelist file | No | "kafka-broker-whitelist.json" | | `Security` | Kafka SASL/SSL credentials | No | null (unsecured) | See [Kafka Security](kafka-security.md) for `Security`, and [Kafka Broker Tuning](kafka-broker-tuning.md) for the broker override and whitelist file paths. !!! warning "Required Conditionally" `PartitionMappings` is required when using Partition-Based streaming strategy. ## Stream Creation Strategies The Streaming API supports two distinct strategies for organizing data streams. The choice between them depends on your scalability needs, topic management preferences, and use case requirements. ### Topic-Based Streaming In Topic-Based streaming, each data stream is created as a separate Kafka topic. This is the simpler approach and is suitable for most use cases. #### When to Use Topic-Based - You have a moderate number of streams - You want clear separation between different data streams - You prefer easier debugging and monitoring (each stream has its own topic) - You don't need to worry about Kafka topic limits - Simpler configuration and management #### How It Works ``` Stream 1 → Kafka Topic: {Domain}.Data.Stream1 Stream 2 → Kafka Topic: {Domain}.Data.Stream2 Stream 3 → Kafka Topic: {Domain}.Data.Stream3 ``` #### Configuration Example ```json { "StreamApiConfig": { "StreamCreationStrategy": "TopicBased", "BrokerUrl": "kafka-broker.company.com:9092", "Domain": "Production", "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true } } ``` #### Benefits ✅ Simple to configure ✅ Clear separation between streams ✅ Easy debugging and monitoring ✅ Natural Kafka topic organization ✅ Recommended for most use cases #### Limitations ⚠️ Each stream creates a new Kafka topic ⚠️ May hit Kafka topic limits with thousands of streams ⚠️ More topics to manage in Kafka ### Partition-Based Streaming In Partition-Based streaming, multiple streams can share the same Kafka topic, with data distributed across different partitions. This is more complex but allows for greater scalability. #### When to Use Partition-Based - You have a very large number of streams (hundreds to thousands) - You want to minimize the number of Kafka topics - You need fine-grained control over partition distribution - You're working with Kafka cluster topic limits - You understand Kafka partition management #### How It Works ``` Stream 1 → Kafka Topic: DataStream, Partition 0 Stream 2 → Kafka Topic: DataStream, Partition 1 Stream 3 → Kafka Topic: DataStream, Partition 0 Stream 4 → Kafka Topic: DataStream, Partition 2 ``` Multiple streams can share the same topic but are distributed across different partitions. #### Configuration Example ```json { "StreamApiConfig": { "StreamCreationStrategy": 1, "BrokerUrl": "kafka-broker.company.com:9092", "Domain": "Production", "PartitionMappings": [ { "Stream": "SensorGroup1", "Partition": 1 }, { "Stream": "SensorGroup2", "Partition": 2 } ] } } ``` #### Benefits ✅ Scales to thousands of streams ✅ Minimizes number of Kafka topics ✅ Fine-grained partition control ✅ Efficient use of Kafka resources ✅ Better for high-cardinality data sources #### Limitations ⚠️ More complex configuration ⚠️ Requires careful partition planning ⚠️ More difficult to debug individual streams ⚠️ Requires understanding of Kafka partitions ## Partition Mappings When using **Partition-Based Streaming**, you must define partition mappings that assign each stream to a specific Kafka partition number. ### PartitionMapping Structure ```csharp public class PartitionMapping { public string Stream { get; } public int Partition { get; } } ``` Each `PartitionMapping` maps a **stream name** to a **Kafka partition number**. Every stream used with Partition-Based strategy must have a corresponding mapping. ### Example Configuration ```json { "StreamApiConfig": { "StreamCreationStrategy": 1, "PartitionMappings": [ { "Stream": "EngineData", "Partition": 1 }, { "Stream": "ChassisData", "Partition": 2 }, { "Stream": "AerodynamicsData", "Partition": 3 } ] } } ``` ### Best Practices !!! tip "Partition Assignment" - Do **not** use partition 0 - Assign each stream a unique partition number - Plan partition numbers upfront — they identify where data is stored - Keep a clear mapping document for your stream-to-partition assignments !!! warning "Planning Considerations" - Every stream used in Partition-Based mode must have an entry in `PartitionMappings` - Partition numbers must be within the configured partition count for the Kafka topic - Streams without a mapping will not route correctly ## Domain Configuration The **Domain** property allows you to organize Kafka topics with a meaningful prefix. ### Purpose - Organize topics by environment (Production, Staging, Development) - Separate topics by team or application - Support multi-tenant architectures - Make topic identification easier ### Examples ```json { "Domain": "Production" } ``` **Results in topics like:** - `Production.Data.SensorData` - `Production.Sys.SessionInfo` - `Production.Essential.CriticalData` ```json { "Domain": "" } ``` **Results in topics like:** - `Data.SensorData` - `Sys.SessionInfo` - `Essential.CriticalData` !!! tip "Naming Conventions" Use clear, consistent domain names: - ✅ `Production`, `Staging`, `Development` - ✅ `TeamA`, `TeamB` - ✅ `F1`, `IndyCar`, `LeMans` - ❌ `prod123`, `test`, `my-domain` ## Logging Configuration The Streaming API uses **Serilog** for structured logging, giving you full control over log output. ### Configure Logging ```json { "Serilog": { "MinimumLevel": { "Default": "Information", "Override": { "Microsoft": "Warning", "System": "Warning" } }, "WriteTo": [ { "Name": "Console", "Args": { "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj}{NewLine}{Exception}" } }, { "Name": "File", "Args": { "path": "logs/streaming-api-.log", "rollingInterval": "Day", "retainedFileCountLimit": 7 } } ] } } ``` ## Router Metrics `EnableRouterPublishMetrics` and `EnableRouterConsumeMetrics` turn on the Kafka producer/consumer statistics callback (librdkafka `statistics.interval.ms`) that feeds the router's Prometheus metrics for each side. Both default to `false`, in which case the callback never fires and there is zero overhead — it is not simply set to a small interval in the background. ```json title="AppConfig.json (excerpt)" linenums="1" { "StreamApiConfig": { "EnableRouterPublishMetrics": true, "EnableRouterConsumeMetrics": true } } ``` !!! note "Timer-based, not per-message" The statistics callback fires on a fixed timer (every 5 seconds when enabled), not once per message, so enabling it does not scale with message volume. ## Kafka Security and Broker Tuning Two related, optional areas are documented on their own pages: - [Kafka Security](kafka-security.md) — SASL/SSL credentials via the `Security` section. - [Kafka Broker Tuning](kafka-broker-tuning.md) — the `KafkaBrokerPublishingConfigFilePath` / `KafkaBrokerConsumingConfigFilePath` / `KafkaBrokerConfigWhitelistFilePath` override files. ## Best Practices ### Choosing a Strategy | Use Topic-Based If... | Use Partition-Based If... | |------------------------|---------------------------| | You have < 100 streams | You have > 100 streams | | Simple is better | You need maximum scalability | | Easy debugging is important | You're hitting Kafka topic limits | | Standard use case | High-cardinality data sources | ### Configuration Tips !!! success "Recommended Settings" ```json { "StreamCreationStrategy": "TopicBased", "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "BatchingResponses": true, "InitialisationTimeoutSeconds": 30, "TerminationTimeoutSeconds": 1 } ``` ### Performance Optimization - **Enable Batching**: Set `BatchingResponses: true` for better throughput - **Tune Timeouts**: Adjust `InitialisationTimeoutSeconds` based on your network - **Monitor Partition Count**: Keep partitions per topic under 100 for optimal performance - **Use Domains**: Organize topics with meaningful domain prefixes ### Troubleshooting | Issue | Solution | |-------|----------| | Connection timeout | Increase `InitialisationTimeoutSeconds` | | Too many topics | Switch to Partition-Based strategy | | Slow startup | Check network connectivity to Kafka broker | | Stream not appearing | Verify `BrokerUrl` and domain configuration | --- # Kafka Security Configure the Stream API to connect to a secured Kafka broker — SASL authentication, TLS encryption, or both. If your broker accepts unauthenticated plaintext connections, you can skip this page entirely: the server runs without security by default. ## Overview Kafka credentials are described by a `Security` section on the Stream API configuration. The server resolves it from two sources, merged in this order (later wins): | Order | Source | Best for | |-------|--------|----------| | 1 | `AppConfig.json` — `StreamApiConfig.Security` section | Shared baseline / dev defaults | | 2 | Environment variables (`StreamApiConfig__Security__*`) | Docker, Kubernetes, CI | !!! note "When embedding the server in-process" Code that hosts `Server` or `StreamingApiClientFactory` directly can instead supply a custom `IKafkaSecurityConfigurationProvider` and bypass both of the sources above. When no credentials are set, the connection is unsecured — existing deployments are unaffected. ## When is security activated? Security is applied when **any** of `SaslUsername`, `SslCaLocation`, or `SslCertificateLocation` is non-empty. All other fields are optional refinements. Once activated, unset fields fall back to: | Field | Default when activated | |-------|-------------------------| | `Protocol` | `SaslPlaintext` | | `Mechanism` | `Plain` | ## Configuration Fields | Field | Description | Values | |-------|-------------|--------| | `SaslUsername` | SASL username — setting this activates security | any string | | `SaslPassword` | SASL password | any string | | `Protocol` | Kafka security protocol | `Plaintext`, `Ssl`, `SaslPlaintext`, `SaslSsl` | | `Mechanism` | SASL mechanism | `Plain`, `ScramSha256`, `ScramSha512` | | `SslCaLocation` | Path to CA certificate (PEM) for broker verification | file path | | `SslCertificateLocation` | Path to client certificate (PEM) for mutual TLS | file path | | `SslKeyLocation` | Path to client private key (PEM) for mutual TLS | file path | | `SslKeyPassword` | Password for the client private key | any string | All fields are optional strings — there is no required combination beyond the activation trigger above. ### AppConfig.json ```json title="AppConfig.json" linenums="1" { "StreamApiConfig": { "BrokerUrl": "kafka-broker:9094", "Security": { "SaslUsername": "myuser", "SaslPassword": "mypassword", "Protocol": "SaslPlaintext", "Mechanism": "Plain" } } } ``` !!! warning "Avoid committing secrets" Don't commit real credentials in `AppConfig.json`. Prefer environment variables for production and keep only non-sensitive defaults in the file. ### Environment Variables Environment variables use the `StreamApiConfig__Security__` prefix and always override `AppConfig.json`. | Variable | Description | |----------|-------------| | `StreamApiConfig__Security__SaslUsername` | SASL username — activates security when set | | `StreamApiConfig__Security__SaslPassword` | SASL password | | `StreamApiConfig__Security__Protocol` | Security protocol | | `StreamApiConfig__Security__Mechanism` | SASL mechanism | | `StreamApiConfig__Security__SslCaLocation` | CA certificate path | | `StreamApiConfig__Security__SslCertificateLocation` | Client certificate path | | `StreamApiConfig__Security__SslKeyLocation` | Client private key path | | `StreamApiConfig__Security__SslKeyPassword` | Client private key password | ## Security Scenarios ### SASL/PLAIN === "JSON" ```json title="AppConfig.json (excerpt)" linenums="1" "Security": { "SaslUsername": "myuser", "SaslPassword": "mypassword", "Protocol": "SaslPlaintext", "Mechanism": "Plain" } ``` === "Environment variables" ```bash StreamApiConfig__Security__SaslUsername=myuser StreamApiConfig__Security__SaslPassword=mypassword StreamApiConfig__Security__Protocol=SaslPlaintext StreamApiConfig__Security__Mechanism=Plain ``` ### SASL/SCRAM-SHA-256 ```json title="AppConfig.json (excerpt)" linenums="1" "Security": { "SaslUsername": "myuser", "SaslPassword": "mypassword", "Protocol": "SaslPlaintext", "Mechanism": "ScramSha256" } ``` ### SSL — broker verification only (one-way TLS) ```json title="AppConfig.json (excerpt)" linenums="1" "Security": { "Protocol": "Ssl", "SslCaLocation": "/certs/ca.pem" } ``` ### SSL — mutual TLS (client certificate) ```json title="AppConfig.json (excerpt)" linenums="1" "Security": { "Protocol": "Ssl", "SslCaLocation": "/certs/ca.pem", "SslCertificateLocation": "/certs/client-cert.pem", "SslKeyLocation": "/certs/client-key.pem", "SslKeyPassword": "keypassword" } ``` ### SASL over TLS (SCRAM with SSL transport) ```json title="AppConfig.json (excerpt)" linenums="1" "Security": { "Protocol": "SaslSsl", "Mechanism": "ScramSha512", "SaslUsername": "myuser", "SaslPassword": "mypassword", "SslCaLocation": "/certs/ca.pem" } ``` ## Docker ```bash docker run -p 13579:13579 \ -e StreamApiConfig__Security__SaslUsername=myuser \ -e StreamApiConfig__Security__SaslPassword=mypassword \ -e StreamApiConfig__Security__Protocol=SaslPlaintext \ -e StreamApiConfig__Security__Mechanism=Plain \ atlasplatformdocker/streaming-proto-server-host:latest ``` Or in `docker-compose.yml`: ```yaml title="docker-compose.yml" linenums="1" services: stream-api: image: atlasplatformdocker/streaming-proto-server-host:latest ports: - "13579:13579" environment: StreamApiConfig__Security__SaslUsername: myuser StreamApiConfig__Security__SaslPassword: mypassword StreamApiConfig__Security__Protocol: SaslPlaintext StreamApiConfig__Security__Mechanism: Plain ``` ## Troubleshooting See [Troubleshooting → Kafka SASL/SSL handshake exception during group coordination](../reference/troubleshooting.md#issue-saslssl-handshake-exception-during-group-coordination) if metadata queries against a secured broker throw a handshake or group-coordination error. ## See Also - [Server Configuration](server-config.md) — full list of non-security config fields - [Kafka Broker Tuning](kafka-broker-tuning.md) — producer/consumer performance overrides - [Docker Setup](docker-setup.md) — container deployment guide --- # Kafka Broker Tuning Optional JSON files let you tune the underlying Kafka producer and consumer the Stream API uses, without touching code. Most deployments never need this page — the built-in defaults below are sensible for typical telemetry workloads. For Kafka security and credentials, see [Kafka Security](kafka-security.md). ## File Locations Three optional files, resolved relative to the server's working directory: | Setting | Default filename | |---------|-------------------| | `KafkaBrokerPublishingConfigFilePath` | `kafka-publisher-broker.json` | | `KafkaBrokerConsumingConfigFilePath` | `kafka-consumer-broker.json` | | `KafkaBrokerConfigWhitelistFilePath` | `kafka-broker-whitelist.json` | Each is set via `StreamApiConfig` in `AppConfig.json` (or the equivalent constructor argument when embedding the server). If a file doesn't exist, its section of this page simply doesn't apply — the server falls back to the defaults below. ## File Format Broker override files are a flat JSON object of raw **librdkafka property names** — dot-separated lowercase strings as documented by the [librdkafka configuration reference](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md). Values are always strings (quote numbers). ```json title="kafka-publisher-broker.json" linenums="1" { "compression.type": "lz4", "linger.ms": "50", "batch.size": "8388608" } ``` Only the keys present in the file override the built-in default; everything else keeps the Stream API default (and ultimately the Confluent/librdkafka default where the Stream API has no opinion). ## Stream API Defaults ### Publisher | librdkafka key | Default | |---------------|---------| | `compression.type` | `zstd` | | `linger.ms` | `20` | | `batch.size` | `4194304` | | `queue.buffering.max.messages` | `2000000` | | `queue.buffering.max.kbytes` | `2097152` | | `message.max.bytes` | `4194304` | | `message.timeout.ms` | `600000` | | `acks` | `1` | | `max.in.flight.requests.per.connection` | `1000000` | | `enable.idempotence` | `false` | ### Consumer | librdkafka key | Default | |---------------|---------| | `session.timeout.ms` | `45000` (Confluent default is `30000`) | All other consumer settings use the Confluent/librdkafka default unless set in the override file. !!! warning "Delivery-semantics keys — override with care" `acks`, `enable.idempotence`, and `max.in.flight.requests.per.connection` can be changed via the override file, but they're not independently validated at startup. Changing one without the others can produce an invalid or contradictory producer configuration (for example, `acks=all` combined with `enable.idempotence=false` is a common misconfiguration). Change these only if you understand the interaction between them. ## Always-Excluded Keys A small set of keys are always taken from Stream API's own routing configuration, never from the override file, regardless of what the file contains: | File | Excluded key | Reason | |------|-------------|--------| | Publisher | `bootstrap.servers` | Always from `BrokerUrl` | | Consumer | `bootstrap.servers` | Always from `BrokerUrl` | | Consumer | `group.id` | Always managed internally | | Consumer | `auto.offset.reset` | Set by the consuming code | Any other key present in the file is applied, subject to the whitelist below. ## Whitelist File The whitelist file (`kafka-broker-whitelist.json` by default) is an optional second gate that controls which of the remaining keys in a broker override file are actually applied. ### File Structure ```json title="kafka-broker-whitelist.json" linenums="1" { "Publish": [ { "Pattern": "linger.ms", "Allow": true }, { "Pattern": "compression.type", "Allow": true }, { "Pattern": "*", "Allow": false } ], "Consume": [ { "Pattern": "session.timeout.ms", "Allow": true }, { "Pattern": "*", "Allow": false } ] } ``` - `Publish` rules gate the publisher broker override file; `Consume` rules gate the consumer broker override file. - Rules are evaluated **in order — first match wins**. - `Pattern` supports `*` (any sequence of characters) and `?` (single character), matched case-insensitively. !!! warning "Unmatched keys are denied once a section has rules" - If the whitelist file doesn't exist, **no filtering is applied** — every key in the override file takes effect (subject only to the always-excluded keys above). This is the default, and is identical to the behavior before whitelisting existed. - If the whitelist file exists and a section (`Publish` or `Consume`) has an **empty** rule list, that section is also unrestricted. - If a section has **one or more rules**, any key that matches none of them is **denied** — you need an explicit catch-all (`{ "Pattern": "*", "Allow": true }`) if you want everything else to pass through. ### Explicit allow-all To make the open-by-default behavior explicit without restricting anything: ```json title="kafka-broker-whitelist.json" linenums="1" { "Publish": [ { "Pattern": "*", "Allow": true } ], "Consume": [ { "Pattern": "*", "Allow": true } ] } ``` ### Strict whitelist example Allow only a handful of throughput-related keys; deny everything else: ```json title="kafka-broker-whitelist.json" linenums="1" { "Publish": [ { "Pattern": "linger.ms", "Allow": true }, { "Pattern": "batch.size", "Allow": true }, { "Pattern": "compression.type", "Allow": true }, { "Pattern": "queue.buffering.max.messages", "Allow": true }, { "Pattern": "queue.buffering.max.kbytes", "Allow": true }, { "Pattern": "*", "Allow": false } ], "Consume": [ { "Pattern": "fetch.min.bytes", "Allow": true }, { "Pattern": "fetch.wait.max.ms", "Allow": true }, { "Pattern": "*", "Allow": false } ] } ``` !!! note "Docker image ships a strict default" The `streaming-proto-server-host` container image bundles a `kafka-broker-whitelist.json` at its default location, and it is a **strict** list (only `compression.type`, `linger.ms`, `batch.size`, `queue.buffering.max.messages`, `queue.buffering.max.kbytes`, `message.max.bytes`, `message.timeout.ms` on publish, and `session.timeout.ms` on consume — everything else denied). If you deploy the image as-is and drop in your own broker override file, keys outside that list are silently ignored unless you replace or extend the whitelist file. ## Tuning Guidance - **Throughput vs latency** — raise `fetch.min.bytes` together with `fetch.wait.max.ms` to let the broker batch more data per fetch; improves consumer throughput at the cost of latency. Lower both for latency-sensitive reads. - **Producer batching** — `linger.ms` (how long the producer waits before sending a batch) and `batch.size` (max bytes per batch) work together; larger values improve throughput, smaller values reduce end-to-end latency. - **Compression** — defaults to `zstd` for best ratio. Use `lz4` for lower CPU cost at the expense of slightly larger messages, or `none` to disable. - **Diagnostics** — `statistics.interval.ms` is not part of the Stream API defaults above; it's driven separately by the router metrics toggles (see [Server Configuration → Router Metrics](server-config.md#router-metrics)). ## See Also - [Kafka Security](kafka-security.md) — Kafka security model, credentials, and deployment patterns - [Server Configuration](server-config.md) — full list of non-security config fields - [Docker Setup](docker-setup.md) — container deployment guide - [librdkafka Configuration Reference](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) — authoritative list of all valid librdkafka keys --- # Docker Setup Deploy the Stream API using Docker containers for development and production environments. ## Quick Start with Docker Compose ### Complete Stack Create `docker-compose.yml`: ```yaml version: '3.8' services: kafka: image: apache/kafka:latest hostname: kafka container_name: kafka-broker ports: - "9092:9092" environment: CLUSTER_ID: 'stream-api-cluster' KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://localhost:9092' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' volumes: - kafka-data:/tmp/kraft-kafka-logs stream-api: image: atlasplatformdocker/streaming-proto-server-host:latest depends_on: - kafka ports: - "13579:13579" - "10010:10010" environment: CONFIG_PATH: /app/Configs/AppConfig.json AUTO_START: true volumes: - ./configs:/app/Configs restart: unless-stopped volumes: kafka-data: ``` ### Configuration Create `configs/AppConfig.json`: ```json { "StreamCreationStrategy": 2, "BrokerUrl": "kafka:9092", "StreamApiPort": 13579, "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "BatchingResponses": false, "PrometheusMetricPort": 10010, "PartitionMappings": [] } ``` ### Start the Stack ```bash # Create config directory mkdir -p configs # Start all services docker-compose up -d # Check service status docker-compose ps # View logs docker-compose logs -f stream-api ``` ## Production Deployment ### Multi-Node Kafka Cluster ```yaml version: '3.8' services: kafka-1: image: apache/kafka:latest hostname: kafka-1 container_name: kafka-broker-1 ports: - "9092:9092" environment: CLUSTER_ID: 'stream-api-prod-cluster' KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-1:9092' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' volumes: - kafka-1-data:/tmp/kraft-kafka-logs kafka-2: image: apache/kafka:latest hostname: kafka-2 container_name: kafka-broker-2 ports: - "9192:9092" environment: CLUSTER_ID: 'stream-api-prod-cluster' KAFKA_NODE_ID: 2 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-2:9092' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' volumes: - kafka-2-data:/tmp/kraft-kafka-logs kafka-3: image: apache/kafka:latest hostname: kafka-3 container_name: kafka-broker-3 ports: - "9292:9092" environment: CLUSTER_ID: 'stream-api-prod-cluster' KAFKA_NODE_ID: 3 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka-3:9092' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' volumes: - kafka-3-data:/tmp/kraft-kafka-logs stream-api-1: image: atlasplatformdocker/streaming-proto-server-host:latest environment: CONFIG_PATH: /app/Configs/AppConfig.json INSTANCE_ID: stream-api-1 volumes: - ./configs:/app/Configs depends_on: - kafka-1 - kafka-2 - kafka-3 volumes: kafka-1-data: kafka-2-data: kafka-3-data: ``` ## Container Configuration ### Environment Variables | Variable | Description | Default | |----------|-------------|---------| | `CONFIG_PATH` | Path to configuration file | /app/Configs/AppConfig.json | | `AUTO_START` | Auto-start server on container start | true | | `LOG_LEVEL` | Logging level | Information | | `METRICS_PORT` | Prometheus metrics port (requires admin privileges) | 10010 | ### Volume Mounts ```yaml volumes: # Configuration files - ./configs:/app/Configs:ro # Logs - ./logs:/app/logs ``` ## Development Setup ### Local Development ```yaml version: '3.8' services: kafka: image: apache/kafka:latest hostname: kafka container_name: kafka-dev ports: - "9092:9092" environment: CLUSTER_ID: 'dev-cluster' KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: 'broker,controller' KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093' KAFKA_LISTENERS: 'PLAINTEXT://:9092,CONTROLLER://:9093' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://localhost:9092' KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT' KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT' KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER' KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_LOG_DIRS: '/tmp/kraft-kafka-logs' stream-api-dev: build: context: . dockerfile: Dockerfile.dev ports: - "13579:13579" - "10010:10010" volumes: - .:/app - /app/bin - /app/obj environment: ASPNETCORE_ENVIRONMENT: Development CONFIG_PATH: /app/Configs/AppConfig.Development.json depends_on: - kafka ``` ### Development Dockerfile ```dockerfile FROM mcr.microsoft.com/dotnet/sdk:8.0 AS dev WORKDIR /app COPY . . # Install development tools RUN dotnet tool install --global dotnet-ef RUN dotnet restore # Development configuration EXPOSE 13579 10010 ENTRYPOINT ["dotnet", "watch", "run", "--project", "MA.Streaming.Proto.ServerComponent.Host"] ``` ## Container Monitoring (Optional) ### Prometheus Configuration If using Prometheus with admin privileges for monitoring: ```yaml prometheus: image: prom/prometheus:latest ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--web.console.templates=/etc/prometheus/consoles' ``` `prometheus.yml`: ```yaml global: scrape_interval: 15s scrape_configs: - job_name: 'stream-api' static_configs: - targets: ['stream-api:10010'] ``` ### Grafana Dashboard ```yaml grafana: image: grafana/grafana:latest ports: - "3000:3000" environment: GF_SECURITY_ADMIN_PASSWORD: admin volumes: - grafana-data:/var/lib/grafana - ./grafana/dashboards:/etc/grafana/provisioning/dashboards - ./grafana/datasources:/etc/grafana/provisioning/datasources ``` ## Troubleshooting ### Common Issues 1. **Port conflicts**: Ensure ports 9092, 13579, and 10010 are available 2. **Memory issues**: Increase Docker memory allocation for Kafka 3. **Network connectivity**: Verify container networking configuration 4. **Kafka startup**: Kafka may take 20-30 seconds to fully initialize ### Debugging ```bash # Check container logs docker-compose logs -f stream-api # Execute shell in container docker-compose exec stream-api bash # Check container resource usage docker stats # Inspect container configuration docker inspect stream-api_stream-api_1 ``` ## See Also - [Server Configuration](server-config.md) - [Monitoring](../reference/troubleshooting.md) --- # Telemetry Example This example demonstrates how to build a complete telemetry data streaming application using the Stream API. ## Overview We'll create a system that: - Simulates vehicle telemetry data from multiple sensors (mocked data generation) - Streams the data in real-time to Kafka - Processes and analyzes the streaming data - Provides real-time dashboards and alerts **Note:** This example uses simulated sensor data for demonstration purposes. In a production environment, you would replace the mock data generators with actual sensor readings from vehicle systems. ## Prerequisites - Stream API server running (see [Docker Setup](../configuration/docker-setup.md)) - .NET 8.0+ SDK for the client application - Basic understanding of telemetry data concepts ## Architecture ![Telemetry Architecture Diagram](./telemetry-architecture.svg) The architecture consists of three main components: 1. **Simulated Vehicle Data**: Mock data generators simulate real-time sensor readings from vehicle systems 2. **Stream API Server**: Handles data ingestion, processing, and distribution via Kafka 3. **Analytics Dashboard**: Processes and visualizes streaming telemetry data in real-time ## Data Producer ### Telemetry Configuration First, define the telemetry parameters using Protocol Buffers: ```csharp // Create configuration packet with all telemetry parameters private readonly ConfigurationPacket telemetryConfig = new() { ConfigId = "vehicle-telemetry-v1", ParameterDefinitions = { // Engine parameters new ParameterDefinition { Identifier = "Engine.Rpm", Name = "Engine RPM", ApplicationName = "VehicleTelemetry", Description = "Engine revolutions per minute", Units = "RPM", DataType = DataType.Float64, FormatString = "%5.0f", MinValue = 0.0, MaxValue = 12000.0, Groups = { "Engine" }, Frequencies = { 10.0 }, // 10 Hz IncludesRowData = true }, new ParameterDefinition { Identifier = "Engine.Temperature", Name = "Engine Temperature", ApplicationName = "VehicleTelemetry", Description = "Engine coolant temperature", Units = "°C", DataType = DataType.Float64, FormatString = "%5.1f", Groups = { "Engine" }, MinValue = -40.0, MaxValue = 150.0, Frequencies = { 1.0 } // 1 Hz }, new ParameterDefinition { Identifier = "Engine.OilPressure", Name = "Oil Pressure", ApplicationName = "VehicleTelemetry", Description = "Engine oil pressure", Units = "bar", DataType = DataType.Float64, FormatString = "%5.1f", Groups = { "Engine" }, MinValue = 0.0, MaxValue = 10.0, Frequencies = { 2.0 } // 2 Hz }, // Brake parameters new ParameterDefinition { Identifier = "Brakes.FrontLeft.Temperature", Name = "Front Left Brake Temperature", ApplicationName = "VehicleTelemetry", Description = "Front left brake disc temperature", Units = "°C", DataType = DataType.Float64, FormatString = "%5.1f", MinValue = 0.0, MaxValue = 800.0, Groups = { "Brakes" }, WarningMaxValue = 700.0, Frequencies = { 5.0 } // 5 Hz }, // Position parameters new ParameterDefinition { Identifier = "Position.Latitude", Name = "Latitude", ApplicationName = "VehicleTelemetry", Description = "GPS latitude coordinate", Units = "degrees", DataType = DataType.Float64, FormatString = "%5.6f", Groups = { "Position" }, MinValue = -90.0, MaxValue = 90.0, IncludesRowData = true }, new ParameterDefinition { Identifier = "Position.Longitude", Name = "Longitude", ApplicationName = "VehicleTelemetry", Description = "GPS longitude coordinate", Units = "degrees", DataType = DataType.Float64, FormatString = "%5.6f", Groups = { "Position" }, MinValue = -180.0, MaxValue = 180.0, IncludesRowData = true }, new ParameterDefinition { Identifier = "Position.Speed", Name = "Vehicle Speed", ApplicationName = "VehicleTelemetry", Description = "Vehicle ground speed", Units = "km/h", DataType = DataType.Float64, FormatString = "%5.1f", MinValue = 0.0, MaxValue = 400.0, Groups = { "Position" }, IncludesRowData = true } }, EventDefinitions = { new EventDefinition { Identifier = "Engine.OverTemp", Name = "Engine Overtemperature", ApplicationName = "VehicleTelemetry", Description = "Engine temperature exceeds safe operating limits", Priority = EventPriority.Critical, Groups = { "Engine" }, DataTypes = { DataType.Float64 }, // Temperature value Units = { "°C" } }, new EventDefinition { Identifier = "Brakes.Overheat", Name = "Brake Overheating", ApplicationName = "VehicleTelemetry", Description = "Brake temperature exceeds safe limits", Priority = EventPriority.High, Groups = { "Brakes" }, DataTypes = { DataType.String, DataType.Float64 }, // Location, Temperature Units = { "", "°C" } } }, GroupDefinitions = { new GroupDefinition { Identifier = "VehicleTelemetry", ApplicationName = "VehicleTelemetry", Name = "Vehicle Telemetry", Description = "Vehicle Telemetry", Groups = { new GroupDefinition { Identifier = "Brakes", ApplicationName = "VehicleTelemetry", Name = "Brakes", Description = "Brake system telemetry" }, new GroupDefinition { Identifier = "Engine", ApplicationName = "VehicleTelemetry", Name = "Engine", Description = "Engine telemetry" }, new GroupDefinition { Identifier = "Position", ApplicationName = "VehicleTelemetry", Name = "Position", Description = "Vehicle position telemetry" } } } } }; ``` ### Telemetry Producer ```csharp public class TelemetryProducer { private readonly PacketWriterService.PacketWriterServiceClient packetWriter; public readonly string SessionKey; private readonly string vehicleId; // Create configuration packet with all telemetry parameters private readonly ConfigurationPacket telemetryConfig = new() { ConfigId = "vehicle-telemetry-v1", ParameterDefinitions = { // Engine parameters new ParameterDefinition { Identifier = "Engine.Rpm", Name = "Engine RPM", ApplicationName = "VehicleTelemetry", Description = "Engine revolutions per minute", Units = "RPM", DataType = DataType.Float64, FormatString = "%5.0f", MinValue = 0.0, MaxValue = 12000.0, Groups = { "Engine" }, Frequencies = { 10.0 }, // 10 Hz IncludesRowData = true }, new ParameterDefinition { Identifier = "Engine.Temperature", Name = "Engine Temperature", ApplicationName = "VehicleTelemetry", Description = "Engine coolant temperature", Units = "°C", DataType = DataType.Float64, FormatString = "%5.1f", Groups = { "Engine" }, MinValue = -40.0, MaxValue = 150.0, Frequencies = { 1.0 } // 1 Hz }, new ParameterDefinition { Identifier = "Engine.OilPressure", Name = "Oil Pressure", ApplicationName = "VehicleTelemetry", Description = "Engine oil pressure", Units = "bar", DataType = DataType.Float64, FormatString = "%5.1f", Groups = { "Engine" }, MinValue = 0.0, MaxValue = 10.0, Frequencies = { 2.0 } // 2 Hz }, // Brake parameters new ParameterDefinition { Identifier = "Brakes.FrontLeft.Temperature", Name = "Front Left Brake Temperature", ApplicationName = "VehicleTelemetry", Description = "Front left brake disc temperature", Units = "°C", DataType = DataType.Float64, FormatString = "%5.1f", MinValue = 0.0, MaxValue = 800.0, Groups = { "Brakes" }, WarningMaxValue = 700.0, Frequencies = { 5.0 } // 5 Hz }, // Position parameters new ParameterDefinition { Identifier = "Position.Latitude", Name = "Latitude", ApplicationName = "VehicleTelemetry", Description = "GPS latitude coordinate", Units = "degrees", DataType = DataType.Float64, FormatString = "%5.6f", Groups = { "Position" }, MinValue = -90.0, MaxValue = 90.0, IncludesRowData = true }, new ParameterDefinition { Identifier = "Position.Longitude", Name = "Longitude", ApplicationName = "VehicleTelemetry", Description = "GPS longitude coordinate", Units = "degrees", DataType = DataType.Float64, FormatString = "%5.6f", Groups = { "Position" }, MinValue = -180.0, MaxValue = 180.0, IncludesRowData = true }, new ParameterDefinition { Identifier = "Position.Speed", Name = "Vehicle Speed", ApplicationName = "VehicleTelemetry", Description = "Vehicle ground speed", Units = "km/h", DataType = DataType.Float64, FormatString = "%5.1f", MinValue = 0.0, MaxValue = 400.0, Groups = { "Position" }, IncludesRowData = true } }, EventDefinitions = { new EventDefinition { Identifier = "Engine.OverTemp", Name = "Engine Overtemperature", ApplicationName = "VehicleTelemetry", Description = "Engine temperature exceeds safe operating limits", Priority = EventPriority.Critical, Groups = { "Engine" }, DataTypes = { DataType.Float64 }, // Temperature value Units = { "°C" } }, new EventDefinition { Identifier = "Brakes.Overheat", Name = "Brake Overheating", ApplicationName = "VehicleTelemetry", Description = "Brake temperature exceeds safe limits", Priority = EventPriority.High, Groups = { "Brakes" }, DataTypes = { DataType.String, DataType.Float64 }, // Location, Temperature Units = { "", "°C" } } }, GroupDefinitions = { new GroupDefinition { Identifier = "VehicleTelemetry", ApplicationName = "VehicleTelemetry", Name = "Vehicle Telemetry", Description = "Vehicle Telemetry", Groups = { new GroupDefinition { Identifier = "Brakes", ApplicationName = "VehicleTelemetry", Name = "Brakes", Description = "Brake system telemetry" }, new GroupDefinition { Identifier = "Engine", ApplicationName = "VehicleTelemetry", Name = "Engine", Description = "Engine telemetry" }, new GroupDefinition { Identifier = "Position", ApplicationName = "VehicleTelemetry", Name = "Position", Description = "Vehicle position telemetry" } } } } }; public readonly string DataSource; public TelemetryProducer(StreamingApiConfiguration configuration, string vehicleId) { var streamingApiClient = StreamingApiClientFactory.Create( configuration, new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider(@"C:\Temp")); streamingApiClient.Initialise(); var sessionManagement = streamingApiClient.GetSessionManagementClient(); var response = sessionManagement.CreateSession( new CreateSessionRequest { DataSource = vehicleId, Type = "Session", Version = 1, UtcOffset = new Duration { Seconds = 0 }, Identifier = "sample-session", Details = { { "Description", $"Telemetry session for vehicle {vehicleId}" }, { "weather", "dry" }, { "temperature", "22°C" } } }); Console.WriteLine($"Session {response.SessionKey} created."); this.packetWriter = streamingApiClient.GetPacketWriterClient(); this.vehicleId = vehicleId; this.SessionKey = response.SessionKey; this.DataSource = vehicleId; } public async Task StartCollecting(CancellationToken cancellationToken = default) { Console.WriteLine($"Starting telemetry collection for vehicle {this.vehicleId}"); // Send configuration first await this.SendConfiguration(); while (!cancellationToken.IsCancellationRequested) { var timestamp = (ulong)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000000; // Convert to nanoseconds // Send telemetry data as periodic packets await Task.WhenAll( this.StreamEngineData(timestamp), this.StreamPositionData(timestamp), this.CheckForEvents(timestamp) ); await Task.Delay(100, cancellationToken); // 10Hz data rate } } private async Task SendConfiguration() { // Send the telemetry configuration (as defined above) await this.packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = this.DataSource, SessionKey = this.SessionKey, Message = new Packet { Content = this.telemetryConfig.ToByteString(), Type = "Configuration", IsEssential = true, SessionKey = this.SessionKey } } }); } private async Task StreamEngineData(ulong timestamp) { // Simulate engine sensor readings var engineData = new PeriodicDataPacket { DataFormat = new SampleDataFormat { ParameterIdentifiers = new ParameterList { ParameterIdentifiers = { "Engine.Rpm", "Engine.Temperature", "Engine.OilPressure" } } }, StartTime = timestamp, Interval = 100000000, // 100ms in nanoseconds Columns = { new SampleColumn // RPM { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = this.GenerateRpm(), Status = DataStatus.Valid } } } }, new SampleColumn // Temperature { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = this.GenerateEngineTemp(), Status = DataStatus.Valid } } } }, new SampleColumn // Oil Pressure { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = this.GenerateOilPressure(), Status = DataStatus.Valid } } } } } }; await this.packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = this.DataSource, Stream = "engine", SessionKey = this.SessionKey, Message = new Packet { Content = engineData.ToByteString(), Type = "PeriodicData", IsEssential = false, SessionKey = this.SessionKey } } }); } private async Task StreamPositionData(ulong timestamp) { // GPS position as row data (timestamped samples) var positionData = new RowDataPacket { DataFormat = new SampleDataFormat { ParameterIdentifiers = new ParameterList { ParameterIdentifiers = { "Position.Latitude", "Position.Longitude", "Position.Speed" } } }, Timestamps = { timestamp }, Rows = { new SampleRow { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = this.GenerateLatitude(), Status = DataStatus.Valid }, new DoubleSample { Value = this.GenerateLongitude(), Status = DataStatus.Valid }, new DoubleSample { Value = this.GenerateSpeed(), Status = DataStatus.Valid } } } } } }; await this.packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = this.DataSource, Stream = "position", SessionKey = this.SessionKey, Message = new Packet { Content = positionData.ToByteString(), Type = "RowData", IsEssential = false, SessionKey = this.SessionKey } } }); } private async Task CheckForEvents(ulong timestamp) { var engineTemp = this.GenerateEngineTemp(); // Check for overtemperature event if (engineTemp > 120.0) { var eventPacket = new EventPacket { DataFormat = new EventDataFormat { EventIdentifier = "Engine.OverTemp" }, Timestamp = timestamp, RawValues = { engineTemp } }; await this.packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = this.DataSource, Stream = "events", SessionKey = this.SessionKey, Message = new Packet { Content = eventPacket.ToByteString(), Type = "Event", IsEssential = false, SessionKey = this.SessionKey } } }); } } // Simulation methods private double GenerateRpm() { return 2000 + Random.Shared.NextDouble() * 4000; } private double GenerateEngineTemp() { return 85 + Random.Shared.NextDouble() * 50; } private double GenerateOilPressure() { return 4.5 + Random.Shared.NextDouble() * 2.0; } private double GenerateLatitude() { return 52.0747 + Random.Shared.NextDouble() * 0.001; } private double GenerateLongitude() { return -1.0174 + Random.Shared.NextDouble() * 0.001; } private double GenerateSpeed() { return Random.Shared.NextDouble() * 200; } } ``` ## Data Consumer ### Real-time Processor ```csharp public class TelemetryProcessor { private readonly string dataSource; private readonly string sessionKey; private readonly PacketReaderService.PacketReaderServiceClient packetReader; private readonly PacketWriterService.PacketWriterServiceClient packetWriter; private readonly ConnectionManagerService.ConnectionManagerServiceClient connectionManager; public TelemetryProcessor(StreamingApiConfiguration configuration, string dataSource, string sessionKey) { this.dataSource = dataSource; this.sessionKey = sessionKey; var streamingApiClient = StreamingApiClientFactory.Create( configuration, new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider(@"C:\Temp")); streamingApiClient.Initialise(); this.packetReader = streamingApiClient.GetPacketReaderClient(); this.packetWriter = streamingApiClient.GetPacketWriterClient(); this.connectionManager = streamingApiClient.GetConnectionManagerClient(); } public async Task StartProcessing(CancellationToken cancellationToken = default) { // Create connection for all telemetry streams var connection = await this.connectionManager.NewConnectionAsync( new NewConnectionRequest { Details = new ConnectionDetails { DataSource = this.dataSource, SessionKey = this.sessionKey, Streams = { "engine", "brakes", "suspension", "position" }, StreamOffsets = { 0, 0, 0, 0 } // Start from beginning } }); Console.WriteLine($"Connected for processing with ID: {connection.Connection.Id}"); // Start processing stream var readRequest = new ReadPacketsRequest { Connection = connection.Connection }; var stream = this.packetReader.ReadPackets(readRequest); while (await stream.ResponseStream.MoveNext(cancellationToken)) { var response = stream.ResponseStream.Current; foreach (var data in response.Response) { await this.ProcessTelemetryPacket(data.Packet); } } } private async Task ProcessTelemetryPacket(Packet packet) { try { // Process based on packet type switch (packet.Type) { case "PeriodicData": var periodicData = PeriodicDataPacket.Parser.ParseFrom(packet.Content); await this.ProcessPeriodicData(periodicData); break; case "RowData": var rowData = RowDataPacket.Parser.ParseFrom(packet.Content); await this.ProcessRowData(rowData); break; case "Event": var eventData = EventPacket.Parser.ParseFrom(packet.Content); await this.ProcessEventData(eventData); break; default: Console.WriteLine($"Unknown packet type: {packet.Type}"); break; } } catch (Exception ex) { Console.WriteLine($"Error processing packet from {packet.SessionKey}: {ex.Message}"); } } private async Task ProcessPeriodicData(PeriodicDataPacket periodicData) { // Extract parameter values based on the data format var parameterIds = periodicData.DataFormat.ParameterIdentifiers.ParameterIdentifiers ?? []; for (var columnIndex = 0; columnIndex < periodicData.Columns.Count; columnIndex++) { if (columnIndex >= parameterIds.Count) { break; } var parameterId = parameterIds[columnIndex]; var column = periodicData.Columns[columnIndex]; if (column.DoubleSamples != null) { foreach (var sample in column.DoubleSamples.Samples) { await this.ProcessParameterValue(parameterId, sample.Value, sample.Status); } } } } private async Task ProcessRowData(RowDataPacket rowData) { var parameterIds = rowData.DataFormat.ParameterIdentifiers?.ParameterIdentifiers ?? []; for (var rowIndex = 0; rowIndex < rowData.Rows.Count; rowIndex++) { var row = rowData.Rows[rowIndex]; if (row.DoubleSamples != null) { for (var sampleIndex = 0; sampleIndex < row.DoubleSamples.Samples.Count && sampleIndex < parameterIds.Count; sampleIndex++) { var parameterId = parameterIds[sampleIndex]; var sample = row.DoubleSamples.Samples[sampleIndex]; await this.ProcessParameterValue(parameterId, sample.Value, sample.Status); } } } } private async Task ProcessParameterValue(string parameterId, double value, DataStatus status) { if (status != DataStatus.Valid) { Console.WriteLine($"Invalid data for {parameterId}: {status}"); return; } // Process based on parameter identifier switch (parameterId) { case "Engine.Rpm": await this.ProcessEngineRpm(value); break; case "Engine.Temperature": await this.ProcessEngineTemperature(value); break; case "Engine.OilPressure": await this.ProcessOilPressure(value); break; case "Brakes.FrontLeft.Temperature": case "Brakes.FrontRight.Temperature": case "Brakes.RearLeft.Temperature": case "Brakes.RearRight.Temperature": await this.ProcessBrakeTemperature(parameterId, value); break; case "Position.Latitude": case "Position.Longitude": case "Position.Speed": await this.ProcessPositionData(parameterId, value); break; default: // Log unknown parameter for debugging Console.WriteLine($"Unknown parameter: {parameterId} = {value}"); break; } } private async Task ProcessEngineRpm(double rpm) { if (rpm > 6500) { await this.RaiseAlert("Engine Over-rev", $"Engine RPM: {rpm:F0}"); } await this.UpdateDashboard( "engine", new { rpm }); } private async Task ProcessEngineTemperature(double temperature) { if (temperature > 95) { await this.RaiseAlert("Engine Overheat", $"Engine temperature: {temperature:F1}°C"); } await this.UpdateDashboard( "engine", new { temperature }); } private async Task ProcessOilPressure(double pressure) { if (pressure < 2.0) { await this.RaiseAlert("Low Oil Pressure", $"Oil pressure: {pressure:F1} bar"); } await this.UpdateDashboard( "engine", new { oil_pressure = pressure }); } private async Task ProcessBrakeTemperature(string location, double temperature) { if (temperature > 350) { await this.RaiseAlert("Brake Overheat", $"{location} temperature: {temperature:F1}°C"); } await this.UpdateDashboard( "brakes", new { location, temperature }); } private async Task ProcessPositionData(string parameter, double value) { // Store position data for combined processing await this.UpdateDashboard( "position", new { parameter, value }); } private async Task ProcessEventData(EventPacket eventData) { var eventId = eventData.DataFormat.EventIdentifier; var timestamp = eventData.Timestamp; var values = eventData.RawValues; switch (eventId) { case "Engine.OverTemp": var temperature = values.Count > 0 ? values[0] : 0; await this.RaiseAlert("Engine Overtemperature Event", $"Temperature: {temperature:F1}°C"); break; case "Brakes.Overheat": var brakeLocation = values.Count > 1 ? $"Temperature: {values[1]:F1}°C" : "Unknown"; await this.RaiseAlert("Brake Overheating Event", brakeLocation); break; default: Console.WriteLine($"Received event: {eventId} at {timestamp} with {values.Count} values"); break; } } private async Task RaiseAlert(string alertType, string message) { Console.WriteLine($"🚨 ALERT: {alertType} - {message}"); // Send alert as an event packet var eventPacket = new EventPacket { DataFormat = new EventDataFormat { EventIdentifier = "System.Alert" }, Timestamp = (ulong)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000000 }; var packetWriter = this.packetWriter; await packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "vehicle-telemetry", Stream = "alerts", SessionKey = "alert-system", Message = new Packet { Content = eventPacket.ToByteString(), Type = "Event", IsEssential = true } } }); } private async Task UpdateDashboard(string component, object data) { // Send to dashboard via WebSocket or similar await Task.Run(() => { Console.WriteLine($"📊 Dashboard Update: {component} = {data}"); }); } } ``` ## Main Application ### Program Entry Point ```csharp class Program { private static async Task Main(string[] args) { var port = IsPortAvailable(13579) ? 13579 : FindFreePortAfter(13579); var configuration = new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", [], port); var cancellationTokenSource = new CancellationTokenSource(); Console.CancelKeyPress += (_, e) => { e.Cancel = true; cancellationTokenSource.Cancel(); cancellationTokenSource.Dispose(); }; var tasks = new List(); var producer = new TelemetryProducer(configuration, "vehicle-001"); tasks.Add(producer.StartCollecting(cancellationTokenSource.Token)); var processor = new TelemetryProcessor(configuration, producer.DataSource, producer.SessionKey); tasks.Add(processor.StartProcessing(cancellationTokenSource.Token)); Console.WriteLine("Telemetry application started. Press Ctrl+C to stop."); try { await Task.WhenAll(tasks); } catch (OperationCanceledException) { Console.WriteLine("Application stopped."); } } private static int FindFreePortAfter(int startPort, int maxPort = 65535) { for (var port = startPort + 1; port <= maxPort; port++) { if (IsPortAvailable(port)) { return port; } } throw new Exception("No free port found in the specified range."); } private static bool IsPortAvailable(int port) { try { using var listener = new TcpListener(IPAddress.Loopback, port); listener.Start(); listener.Stop(); return true; } catch (SocketException) { return false; } } } ``` ## Running the Example ### Build and Run ```bash # Create new console project dotnet new console -n TelemetryExample cd TelemetryExample # Replace Program.cs with the example code above # Build the project dotnet build # Run producer only dotnet run ``` ### Expected Output ``` Telemetry application started in both mode. Press Ctrl+C to stop. Starting telemetry collection for vehicle vehicle-001 Connected for processing with ID: 12345 📊 Dashboard Update: engine = {"rpm":4523.2,"temperature":87.3} 📊 Dashboard Update: brakes = {"max_temp":245.7} 📊 Dashboard Update: position = {"speed":125.4,"latitude":52.0234,"longitude":-0.9876} 🚨 ALERT: Engine Overheat - Engine temperature: 96.2°C ``` ## Next Steps - [Docker Setup Guide](../configuration/docker-setup.md) - [Monitoring and Metrics](../reference/troubleshooting.md) --- # Custom Messages This guide demonstrates how to implement and use custom message types with the Stream API. While the API uses Protocol Buffers internally for the packet structure, you can serialize your custom message content using any serialization method (JSON, XML, MessagePack, etc.). ## Overview The Stream API `Packet` structure supports custom message types: ```protobuf message Packet { bytes content = 1; // Your serialized custom message string type = 2; // Your custom message type name bool is_essential = 3; // Whether this is critical data string session_key = 4; // Session identifier } ``` You can put **any serialized data** into the `content` field and identify it with a custom `type` name. ## Simple Custom Message Example ### Step 1: Create Session ```csharp using MA.Streaming.API; const string DataSource = "sensors"; // Create session first var sessionManager = StreamingApiClient.GetSessionManagerClient(); var sessionResponse = await sessionManager.CreateSessionAsync(new CreateSessionRequest { DataSource = DataSource, Type = "SensorSession", Version = 1, Identifier = "temperature-monitoring", Details = { ["location"] = "Lab A" } }); string sessionKey = sessionResponse.SessionKey; Console.WriteLine($"Created session: {sessionKey}"); ``` ### Step 2: Define Your Custom Message Class ```csharp using System; // Simple custom message class public class CustomType { public string MessageId { get; set; } public DateTime Timestamp { get; set; } public string Source { get; set; } public double Value { get; set; } public string Unit { get; set; } } ``` ### Step 3: Serialize and Send ```csharp using System.Text; using System.Text.Json; using Google.Protobuf; using MA.Streaming.API; // Create your custom message var customMessage = new CustomType { MessageId = "msg-001", Timestamp = DateTime.UtcNow, Source = "temperature-sensor", Value = 85.5, Unit = "celsius" }; // Serialize to JSON string jsonString = JsonSerializer.Serialize(customMessage); byte[] jsonBytes = Encoding.UTF8.GetBytes(jsonString); // Create packet with custom message var packet = new Packet { Content = ByteString.CopyFrom(jsonBytes), Type = "CustomType", // Your custom type identifier IsEssential = false, SessionKey = sessionKey }; // Send the packet (use same data source as session) var packetWriter = StreamingApiClient.GetPacketWriterClient(); await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = DataSource, // Same as session Stream = "temperature", SessionKey = sessionKey, // Session key from Step 1 Message = packet } }); ``` ### Step 4: Receive and Deserialize ```csharp using System.Text; using System.Text.Json; // Create connection with same data source and session key var connectionManager = StreamingApiClient.GetConnectionManagerClient(); var connectionResponse = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = DataSource, // Same data source SessionKey = sessionKey, // Same session key Streams = { "temperature" }, StreamOffsets = { 0 } // Start from beginning } }); // Read packets var packetReader = StreamingApiClient.GetPacketReaderClient(); var stream = packetReader.ReadPackets(new ReadPacketsRequest { Connection = connectionResponse.Connection }); while (await stream.ResponseStream.MoveNext()) { foreach (var response in stream.ResponseStream.Current.Response) { var packet = response.Packet; // Check if it's your custom type if (packet.Type == "CustomType") { // Deserialize from JSON string jsonString = Encoding.UTF8.GetString(packet.Content.ToByteArray()); var customMessage = JsonSerializer.Deserialize(jsonString); Console.WriteLine($"Received custom message:"); Console.WriteLine($" ID: {customMessage.MessageId}"); Console.WriteLine($" Timestamp: {customMessage.Timestamp}"); Console.WriteLine($" Source: {customMessage.Source}"); Console.WriteLine($" Value: {customMessage.Value} {customMessage.Unit}"); } } } ``` ## Complete Working Example ```csharp using System; using System.Text; using System.Text.Json; using System.Threading.Tasks; using Google.Protobuf; using MA.Streaming.API; using MA.Streaming.Proto.Client; public class CustomType { public string MessageId { get; set; } public DateTime Timestamp { get; set; } public string Source { get; set; } public double Value { get; set; } public string Unit { get; set; } } public class CustomMessageExample { private readonly StreamingApiClient _apiClient; private const string DataSource = "sensors"; public CustomMessageExample(string serverAddress) { _apiClient = new StreamingApiClient(serverAddress); } public async Task CreateSessionAsync() { // Create session first var sessionManager = _apiClient.GetSessionManagerClient(); var sessionResponse = await sessionManager.CreateSessionAsync(new CreateSessionRequest { DataSource = DataSource, Type = "SensorSession", Version = 1, Identifier = "temperature-monitoring", Details = { ["location"] = "Lab A", ["purpose"] = "Temperature monitoring" } }); Console.WriteLine($"Created session: {sessionResponse.SessionKey}"); return sessionResponse.SessionKey; } public async Task SendCustomMessageAsync(string sessionKey) { // Create custom message var customMessage = new CustomType { MessageId = Guid.NewGuid().ToString(), Timestamp = DateTime.UtcNow, Source = "temperature-sensor-01", Value = 85.5, Unit = "celsius" }; // Serialize to JSON bytes string jsonString = JsonSerializer.Serialize(customMessage); byte[] jsonBytes = Encoding.UTF8.GetBytes(jsonString); // Wrap in Packet var packet = new Packet { Content = ByteString.CopyFrom(jsonBytes), Type = "CustomType", IsEssential = false, SessionKey = sessionKey }; // Send packet var packetWriter = _apiClient.GetPacketWriterClient(); await packetWriter.WriteDataPacketAsync(new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = DataSource, Stream = "temperature", SessionKey = sessionKey, Message = packet } }); Console.WriteLine($"Sent custom message: {customMessage.MessageId}"); } public async Task ReceiveCustomMessagesAsync(string sessionKey) { // Create connection with same data source and session key var connectionManager = _apiClient.GetConnectionManagerClient(); var connectionResponse = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = DataSource, SessionKey = sessionKey, Streams = { "temperature" }, StreamOffsets = { 0 } // Start from beginning } }); Console.WriteLine($"Created connection: {connectionResponse.Connection.Id}"); // Read packets var packetReader = _apiClient.GetPacketReaderClient(); var stream = packetReader.ReadPackets(new ReadPacketsRequest { Connection = connectionResponse.Connection }); while (await stream.ResponseStream.MoveNext()) { foreach (var response in stream.ResponseStream.Current.Response) { var packet = response.Packet; // Handle custom message type if (packet.Type == "CustomType") { ProcessCustomMessage(packet); } } } } private void ProcessCustomMessage(Packet packet) { try { // Deserialize from JSON string jsonString = Encoding.UTF8.GetString(packet.Content.ToByteArray()); var customMessage = JsonSerializer.Deserialize(jsonString); Console.WriteLine($"Received CustomType message:"); Console.WriteLine($" ID: {customMessage.MessageId}"); Console.WriteLine($" Timestamp: {customMessage.Timestamp:yyyy-MM-dd HH:mm:ss}"); Console.WriteLine($" Source: {customMessage.Source}"); Console.WriteLine($" Value: {customMessage.Value} {customMessage.Unit}"); } catch (JsonException ex) { Console.WriteLine($"Failed to deserialize custom message: {ex.Message}"); } } } // Usage public static async Task Main() { var example = new CustomMessageExample("localhost:13579"); // Step 1: Create session string sessionKey = await example.CreateSessionAsync(); // Step 2: Send custom messages await example.SendCustomMessageAsync(sessionKey); // Step 3: Read custom messages (use same data source and session key) await example.ReceiveCustomMessagesAsync(sessionKey); } ``` ## Using Different Serializers ### MessagePack Serialization ```csharp using MessagePack; [MessagePackObject] public class CustomType { [Key(0)] public string MessageId { get; set; } [Key(1)] public DateTime Timestamp { get; set; } [Key(2)] public string Source { get; set; } [Key(3)] public double Value { get; set; } [Key(4)] public string Unit { get; set; } } // Serialize byte[] msgPackBytes = MessagePackSerializer.Serialize(customMessage); var packet = new Packet { Content = ByteString.CopyFrom(msgPackBytes), Type = "CustomType", // ... }; // Deserialize var customMessage = MessagePackSerializer.Deserialize(packet.Content.ToByteArray()); ``` ### Binary Serialization ```csharp using System.Runtime.Serialization.Formatters.Binary; using System.IO; // Serialize using var ms = new MemoryStream(); var formatter = new BinaryFormatter(); formatter.Serialize(ms, customMessage); byte[] binaryBytes = ms.ToArray(); var packet = new Packet { Content = ByteString.CopyFrom(binaryBytes), Type = "CustomType", // ... }; // Deserialize using var ms = new MemoryStream(packet.Content.ToByteArray()); var formatter = new BinaryFormatter(); var customMessage = (CustomType)formatter.Deserialize(ms); ``` ### XML Serialization ```csharp using System.Xml.Serialization; // Serialize var xmlSerializer = new XmlSerializer(typeof(CustomType)); using var stringWriter = new StringWriter(); xmlSerializer.Serialize(stringWriter, customMessage); byte[] xmlBytes = Encoding.UTF8.GetBytes(stringWriter.ToString()); var packet = new Packet { Content = ByteString.CopyFrom(xmlBytes), Type = "CustomType", // ... }; // Deserialize string xmlString = Encoding.UTF8.GetString(packet.Content.ToByteArray()); using var stringReader = new StringReader(xmlString); var customMessage = (CustomType)xmlSerializer.Deserialize(stringReader); ``` ## Message Type Registry Pattern For larger applications, implement a message type registry: ```csharp public interface ICustomMessageHandler { string MessageType { get; } void Process(byte[] content); } public class CustomTypeHandler : ICustomMessageHandler { public string MessageType => "CustomType"; public void Process(byte[] content) { string jsonString = Encoding.UTF8.GetString(content); var message = JsonSerializer.Deserialize(jsonString); // Process your custom message Console.WriteLine($"Processing {message.Source}: {message.Value} {message.Unit}"); } } public class MessageRegistry { private readonly Dictionary _handlers = new(); public void Register(ICustomMessageHandler handler) { _handlers[handler.MessageType] = handler; } public void ProcessPacket(Packet packet) { if (_handlers.TryGetValue(packet.Type, out var handler)) { handler.Process(packet.Content.ToByteArray()); } else { Console.WriteLine($"Unknown message type: {packet.Type}"); } } } // Usage var registry = new MessageRegistry(); registry.Register(new CustomTypeHandler()); // Process incoming packets foreach (var response in packetResponses) { registry.ProcessPacket(response.Packet); } ``` ## Best Practices ### 1. **Use Consistent Type Names** ```csharp public static class MessageTypes { public const string CustomType = "CustomType"; public const string SensorData = "SensorData"; public const string AlertMessage = "AlertMessage"; } var packet = new Packet { Type = MessageTypes.CustomType, // ... }; ``` ### 2. **Handle Deserialization Errors** ```csharp private CustomType DeserializeCustomMessage(byte[] content) { try { string jsonString = Encoding.UTF8.GetString(content); return JsonSerializer.Deserialize(jsonString); } catch (JsonException ex) { Console.WriteLine($"Deserialization failed: {ex.Message}"); return null; } } ``` ## Key Takeaways 1. **Flexible Content**: The `Packet.Content` field accepts any byte array, allowing you to use any serialization format 2. **Type Identification**: Use the `Packet.Type` field to identify your custom message types 3. **Any Serializer**: Choose JSON for readability, MessagePack for performance, or any other serialization library 4. **Standard Protocol**: The packet structure itself uses Protocol Buffers, but your message content is completely flexible 5. **Type Safety**: Implement a message registry pattern for handling multiple custom types in a type-safe manner ## See Also - [API Reference](../reference/api-reference.md) - [Packet Writer Service](../services/packet-writer.md) - [Packet Reader Service](../services/packet-reader.md) - [Telemetry Example](telemetry-example.md) --- # API Reference Complete reference for the Stream API gRPC services and message types based on `ma.streaming.api.v1`. ## Service Overview The Stream API consists of four main gRPC services: 1. **[Connection Manager Service](#connection-manager-service)**: Manage client connections and data source access 2. **[Session Management Service](#session-management-service)**: Handle data sessions and session lifecycle 3. **[Packet Writer Service](#packet-writer-service)**: Publish data and info packets to streams 4. **[Packet Reader Service](#packet-reader-service)**: Consume data packets with filtering capabilities 5. **[Data Format Manager Service](#data-format-manager-service)**: Manage data format identifiers for parameters and events ## Connection Manager Service Manages client connections to data sources and streams. Connections maintain current offset positions and provide access to streaming data. ### Methods #### NewConnection Creates a new connection to data streams. ```protobuf rpc NewConnection(NewConnectionRequest) returns (NewConnectionResponse); ``` **Request:** ```protobuf message NewConnectionRequest { ConnectionDetails details = 1; } message ConnectionDetails { string data_source = 1; // Data source to read from string session_key = 2; // Session key (optional if sessions not used) repeated string streams = 3; // Streams to read (optional for all streams) repeated int64 stream_offsets = 4; // Start offset for each stream (-1=Latest, 0=Earliest) int64 main_offset = 5; // Offset for main data source topic int64 essentials_offset = 6; // Offset for essentials topic bool exclude_main_stream = 7; // Exclude main stream from reading } ``` **Response:** ```protobuf message NewConnectionResponse { Connection connection = 1; // Connection identifier } message Connection { int64 id = 1; // Internal connection ID (do not modify) } ``` #### GetConnection Retrieves details of an existing connection. ```protobuf rpc GetConnection(GetConnectionRequest) returns (GetConnectionResponse); ``` **Request:** ```protobuf message GetConnectionRequest { Connection connection = 1; // Connection identifier } ``` **Response:** ```protobuf message GetConnectionResponse { ConnectionDetails details = 1; // The connection details } ``` #### CloseConnection Closes an existing connection. ```protobuf rpc CloseConnection(CloseConnectionRequest) returns (CloseConnectionResponse); ``` **Request:** ```protobuf message CloseConnectionRequest { Connection connection = 1; // Connection identifier } ``` **Response:** ```protobuf message CloseConnectionResponse { bool success = 1; // Whether the close connection request succeeded } ``` ### Usage Examples ```csharp // Initialize client var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); // Create connection var connectionManager = streamingApiClient.GetConnectionManagerClient(); var connection = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "telemetry", SessionKey = "session-001", Streams = { "engine", "brakes" }, StreamOffsets = { 0, 0 } } }); // Get connection info var info = await connectionManager.GetConnectionAsync(new GetConnectionRequest { Connection = connection.Connection }); // Close connection await connectionManager.CloseConnectionAsync(new CloseConnectionRequest { Connection = connection.Connection }); ``` ## Session Management Service Manages data sessions and session lifecycle. Sessions group related data streams and provide metadata context. ### Methods #### CreateSession Creates a new data session with metadata. ```protobuf rpc CreateSession(CreateSessionRequest) returns (CreateSessionResponse); ``` **Request:** ```protobuf message CreateSessionRequest { string data_source = 1; // Data source name string type = 2; // Session type (defaults to "Session") uint32 version = 3; // Version (defaults to 1) google.protobuf.Duration utc_offset = 4; // UTC offset for timezone string identifier = 5; // Session identifier repeated string associate_session_key = 6; // Associated session keys map details = 7; // Session details (key-value pairs) } ``` **Response:** ```protobuf message CreateSessionResponse { string session_key = 1; // Unique session key ma.streaming.open_data.v1.NewSessionPacket new_session = 2; // New session packet bool success = 3; // Whether creation succeeded } ``` #### EndSession Ends an existing session. ```protobuf rpc EndSession(EndSessionRequest) returns (EndSessionResponse); ``` **Request:** ```protobuf message EndSessionRequest { string data_source = 1; // Data source name string session_key = 2; // Unique session key } ``` **Response:** ```protobuf message EndSessionResponse { ma.streaming.open_data.v1.EndOfSessionPacket end_session = 2; // End session packet bool success = 3; // Whether end succeeded } ``` #### GetCurrentSessions Lists all active sessions for a data source. ```protobuf rpc GetCurrentSessions(GetCurrentSessionsRequest) returns (GetCurrentSessionsResponse); ``` **Request:** ```protobuf message GetCurrentSessionsRequest { string data_source = 1; // Data source name } ``` **Response:** ```protobuf message GetCurrentSessionsResponse { repeated string session_keys = 1; // List of active session keys bool success = 2; // Whether request succeeded } ``` #### GetSessionInfo Retrieves detailed information about a session. ```protobuf rpc GetSessionInfo(GetSessionInfoRequest) returns (GetSessionInfoResponse); ``` **Request:** ```protobuf message GetSessionInfoRequest { string session_key = 1; // Unique session key } ``` **Response:** ```protobuf message GetSessionInfoResponse { string data_source = 1; // Data source name string identifier = 2; // Session identifier string type = 3; // Session type uint32 version = 4; // Session version repeated string associate_session_keys = 5; // Associated sessions bool is_complete = 6; // Whether session is completed repeated string streams = 7; // Available streams map topic_partition_offsets = 8; // Topic/partition offsets int64 main_offset = 9; // Main topic offset int64 essentials_offset = 10; // Essentials topic offset map details = 11; // Session details google.protobuf.Duration utc_offset = 12; // UTC offset bool success = 13; // Whether request succeeded } ``` #### GetSessionStartNotification Continuously receives notifications when sessions start. ```protobuf rpc GetSessionStartNotification(GetSessionStartNotificationRequest) returns (stream GetSessionStartNotificationResponse); ``` **Request:** ```protobuf message GetSessionStartNotificationRequest { string data_source = 1; // Data source name } ``` **Response:** ```protobuf message GetSessionStartNotificationResponse { string session_key = 1; // Unique session key of the started session string data_source = 2; // Data source name } ``` #### GetSessionStopNotification Continuously receives notifications when sessions end. ```protobuf rpc GetSessionStopNotification(GetSessionStopNotificationRequest) returns (stream GetSessionStopNotificationResponse); ``` **Request:** ```protobuf message GetSessionStopNotificationRequest { string data_source = 1; // Data source name } ``` **Response:** ```protobuf message GetSessionStopNotificationResponse { string session_key = 1; // Unique session key of the ended session string data_source = 2; // Data source name } ``` #### UpdateSessionIdentifier Updates the identifier of an existing session. ```protobuf rpc UpdateSessionIdentifier(UpdateSessionIdentifierRequest) returns (UpdateSessionIdentifierResponse); ``` **Request:** ```protobuf message UpdateSessionIdentifierRequest { string session_key = 1; // Unique session key string identifier = 2; // Session identifier } ``` **Response:** ```protobuf message UpdateSessionIdentifierResponse { bool success = 1; // Whether the update session identifier request succeeded } ``` #### AddAssociateSession Associates one session with another. ```protobuf rpc AddAssociateSession(AddAssociateSessionRequest) returns (AddAssociateSessionResponse); ``` **Request:** ```protobuf message AddAssociateSessionRequest { string session_key = 1; // Unique session key of the parent session string associate_session_key = 2; // Unique session key of the associate session } ``` **Response:** ```protobuf message AddAssociateSessionResponse { bool success = 1; // Whether the add associate session request succeeded } ``` #### UpdateSessionDetails Updates the details of an existing session. ```protobuf rpc UpdateSessionDetails(UpdateSessionDetailsRequest) returns (UpdateSessionDetailsResponse); ``` **Request:** ```protobuf message UpdateSessionDetailsRequest { string session_key = 1; // Unique session key map details = 2; // Session details to update (detail name, detail value) } ``` **Response:** ```protobuf message UpdateSessionDetailsResponse { bool success = 1; // Whether the update session details request succeeded } ``` ### Usage Examples ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); // Create session var session = await sessionManager.CreateSessionAsync(new CreateSessionRequest { SessionKey = "race-session-001", DataSource = "telemetry", SessionInfo = new SessionInfoPacket { Type = "Race Session", Version = 1, Identifier = "silverstone-2024", Details = { ["track"] = "Silverstone Circuit", ["weather"] = "dry" } } }); // Update session await sessionManager.UpdateSessionInfoAsync(new UpdateSessionInfoRequest { SessionKey = "race-session-001", SessionInfo = new SessionInfoPacket { Details = { ["status"] = "active" } } }); ``` ## Packet Writer Service Provides methods to write data packets to streams. Supports both single packet and batch operations. ### Methods #### WriteDataPacket Writes a single data packet to the stream. ```protobuf rpc WriteDataPacket(WriteDataPacketRequest) returns (WriteDataPacketResponse); ``` **Request:** ```protobuf message WriteDataPacketRequest { DataPacketDetails detail = 1; } message DataPacketDetails { ma.streaming.open_data.v1.Packet message = 1; // Packet to write string data_source = 2; // Data source for this packet string stream = 3; // Stream name to which to write string session_key = 4; // Unique session key } ``` **Response:** ```protobuf message WriteDataPacketResponse { } ``` #### WriteDataPackets Writes multiple data packets in a batch/streaming operation. ```protobuf rpc WriteDataPackets(stream WriteDataPacketsRequest) returns (WriteDataPacketsResponse); ``` **Request:** ```protobuf message WriteDataPacketsRequest { repeated DataPacketDetails details = 1; } ``` **Response:** ```protobuf message WriteDataPacketsResponse { } ``` #### WriteInfoPacket Writes a single info packet to the stream. ```protobuf rpc WriteInfoPacket(WriteInfoPacketRequest) returns (WriteInfoPacketResponse); ``` **Request:** ```protobuf message WriteInfoPacketRequest { ma.streaming.open_data.v1.Packet message = 1; // Packet to write InfoType type = 2; // Type for this packet } enum InfoType { INFO_TYPE_UNSPECIFIED = 0; // Unspecified INFO_TYPE_SESSION_INFO = 1; // Session info INFO_TYPE_SYSTEM_STATUS = 2; // System status } ``` **Response:** ```protobuf message WriteInfoPacketResponse { } ``` #### WriteInfoPackets Writes multiple info packets in a streaming operation. ```protobuf rpc WriteInfoPackets(stream WriteInfoPacketsRequest) returns (WriteInfoPacketsResponse); ``` **Request:** ```protobuf message WriteInfoPacketsRequest { ma.streaming.open_data.v1.Packet message = 1; // Packet to write InfoType type = 2; // Type for this packet } ``` **Response:** ```protobuf message WriteInfoPacketsResponse { } ``` ### Usage Examples ```csharp using MA.Streaming.OpenData; // Initialize client var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var packetWriter = streamingApiClient.GetPacketWriterClient(); // Create periodic data packet with parameter samples var timestamp = (ulong)DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000000; // nanoseconds var periodicData = new PeriodicDataPacket { DataFormat = new SampleDataFormat { ParameterIdentifiers = new ParameterList { ParameterIdentifiers = { "Engine.Rpm", "Engine.Temperature", "Engine.OilPressure" } } }, StartTime = timestamp, Interval = 100000000, // 100ms in nanoseconds Columns = { new SampleColumn // RPM { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = 5000.0, Status = DataStatus.Valid } } } }, new SampleColumn // Temperature { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = 85.5, Status = DataStatus.Valid } } } }, new SampleColumn // Oil Pressure { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample { Value = 4.2, Status = DataStatus.Valid } } } } } }; // Write single packet await packetWriter.WriteDataPacketAsync( new WriteDataPacketRequest { Detail = new DataPacketDetails { DataSource = "telemetry", Stream = "engine", SessionKey = "session-001", Message = new Packet { Content = periodicData.ToByteString(), Type = "PeriodicData", IsEssential = false, SessionKey = "session-001" } } }); // Write multiple packets in batch var packets = new List { new Packet { Content = periodicData.ToByteString(), Type = "PeriodicData", SessionKey = "session-001" } }; var batchResponse = await packetWriter.WriteDataPacketsAsync(new WriteDataPacketsRequest { Details = { new DataPacketDetails { DataSource = "telemetry", Stream = "engine", SessionKey = "session-001", Message = packets[0] } } }); ``` ## Packet Reader Service Provides methods to read data packets from streams with filtering and streaming capabilities. ### Methods #### ReadPackets Reads data packets from streams with optional filtering. ```protobuf rpc ReadPackets(ReadPacketsRequest) returns (stream ReadPacketsResponse); ``` **Request:** ```protobuf message ReadPacketsRequest { Connection connection = 1; } ``` **Response:** ```protobuf message ReadPacketsResponse { repeated PacketResponse response = 1; } message PacketResponse { ma.streaming.open_data.v1.Packet packet = 1; // The packet that was read string stream = 2; // The stream the packet was read from google.protobuf.Timestamp submit_time = 3; // Time that packet was submitted to broker } ``` #### ReadEssentials Reads only essential packets from the stream. ```protobuf rpc ReadEssentials(ReadEssentialsRequest) returns (stream ReadEssentialsResponse); ``` **Request:** ```protobuf message ReadEssentialsRequest { Connection connection = 1; } ``` **Response:** ```protobuf message ReadEssentialsResponse { repeated PacketResponse response = 1; } ``` #### ReadDataPackets Reads data packets with advanced filtering capabilities. ```protobuf rpc ReadDataPackets(ReadDataPacketsRequest) returns (stream ReadDataPacketsResponse); ``` **Request:** ```protobuf message ReadDataPacketsRequest { DataPacketRequest request = 1; } message DataPacketRequest { Connection connection = 1; // Connection to read repeated string include_parameters = 2; // Parameters to include (regex) repeated string exclude_parameters = 3; // Parameters to exclude (regex) repeated string include_events = 4; // Events to include (regex) repeated string exclude_events = 5; // Events to exclude (regex) bool include_markers = 6; // Include markers? } ``` **Response:** ```protobuf message ReadDataPacketsResponse { repeated PacketResponse response = 1; } ``` ### Usage Examples ```csharp using MA.Streaming.OpenData; // Initialize client var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var packetReader = streamingApiClient.GetPacketReaderClient(); // Get connection details first var connectionManager = streamingApiClient.GetConnectionManagerClient(); var connectionResponse = await connectionManager.NewConnectionAsync(new NewConnectionRequest { Details = new ConnectionDetails { DataSource = "telemetry", SessionKey = "session-001", Streams = { "engine", "brakes", "suspension", "position" }, StreamOffsets = { 0, 0, 0, 0 } // Start from beginning } }); // Read all packets var readRequest = new ReadPacketsRequest { Connection = connectionResponse.Connection }; var stream = packetReader.ReadPackets(readRequest); while (await stream.ResponseStream.MoveNext()) { var response = stream.ResponseStream.Current; if (response.Success) { var packet = response.Packet; // Process different packet types switch (packet.Type) { case "PeriodicData": ProcessPeriodicData(packet); break; case "Event": ProcessEvent(packet); break; case "Configuration": ProcessConfiguration(packet); break; } } } // Read only essential packets var essentialsRequest = new ReadEssentialsRequest { Connection = connectionResponse.Connection }; var essentialsStream = packetReader.ReadEssentials(essentialsRequest); while (await essentialsStream.ResponseStream.MoveNext()) { var response = essentialsStream.ResponseStream.Current; if (response.Success) { ProcessEssentialPacket(response.Packet); } } ``` ## Data Format Manager Service Manages data format definitions and provides access to parameter and event schemas. ### Methods #### GetParameterDataFormatId Retrieves the data format ID for parameter data. ```protobuf rpc GetParameterDataFormatId(GetParameterDataFormatIdRequest) returns (GetParameterDataFormatIdResponse); ``` **Request:** ```protobuf message GetParameterDataFormatIdRequest { string data_source = 1; // Data source name repeated string parameters = 2; // Ordered list of parameter identifiers } ``` **Response:** ```protobuf message GetParameterDataFormatIdResponse { uint64 data_format_identifier = 1; // Data format identifier } ``` #### GetEventDataFormatId Retrieves the data format ID for event data. ```protobuf rpc GetEventDataFormatId(GetEventDataFormatIdRequest) returns (GetEventDataFormatIdResponse); ``` **Request:** ```protobuf message GetEventDataFormatIdRequest { string data_source = 1; // Data source name string event = 2; // Event identifier } ``` **Response:** ```protobuf message GetEventDataFormatIdResponse { uint64 data_format_identifier = 1; // Data format identifier } ``` #### GetParametersList Retrieves the list of available parameters for a data source. ```protobuf rpc GetParametersList(GetParametersListRequest) returns (GetParametersListResponse); ``` **Request:** ```protobuf message GetParametersListRequest { string data_source = 1; // Data source name uint64 data_format_identifier = 2; // Data format identifier } ``` **Response:** ```protobuf message GetParametersListResponse { repeated string parameters = 1; // List of parameters } ``` #### GetEvent Retrieves information about a specific event. ```protobuf rpc GetEvent(GetEventRequest) returns (GetEventResponse); ``` **Request:** ```protobuf message GetEventRequest { string data_source = 1; // Data source name uint64 data_format_identifier = 2; // Data format identifier } ``` **Response:** ```protobuf message GetEventResponse { string event = 1; // Event identifier } ``` ### Usage Examples ```csharp var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var dataFormatManager = streamingApiClient.GetDataFormatManagerClient(); // Get parameter data format ID var parameterFormatResponse = await dataFormatManager.GetParameterDataFormatIdAsync( new GetParameterDataFormatIdRequest { DataSource = "telemetry" }); if (parameterFormatResponse.Success) { var formatId = parameterFormatResponse.DataFormatId; Console.WriteLine($"Parameter data format ID: {formatId}"); } // Get event data format ID var eventFormatResponse = await dataFormatManager.GetEventDataFormatIdAsync( new GetEventDataFormatIdRequest { DataSource = "telemetry" }); // Get parameters list var parametersResponse = await dataFormatManager.GetParametersListAsync( new GetParametersListRequest { DataSource = "telemetry" }); if (parametersResponse.Success) { foreach (var parameter in parametersResponse.Parameters) { Console.WriteLine($"Parameter: {parameter.Name} (ID: {parameter.Id})"); } } // Get specific event information var eventResponse = await dataFormatManager.GetEventAsync( new GetEventRequest { DataSource = "telemetry", EventId = 1001 }); if (eventResponse.Success) { var eventInfo = eventResponse.Event; Console.WriteLine($"Event: {eventInfo.Name} - {eventInfo.Description}"); } ``` ## Common Message Types ### Error Handling All services use standard gRPC status codes: - `OK` (0): Success - `INVALID_ARGUMENT` (3): Invalid request parameters - `NOT_FOUND` (5): Resource not found - `ALREADY_EXISTS` (6): Resource already exists - `RESOURCE_EXHAUSTED` (8): Resource limits exceeded - `UNAVAILABLE` (14): Service temporarily unavailable ### Status Codes ```csharp try { var result = await client.SomeOperation(); } catch (RpcException ex) { switch (ex.StatusCode) { case StatusCode.InvalidArgument: // Handle invalid parameters break; case StatusCode.NotFound: // Handle missing resource break; case StatusCode.Unavailable: // Handle service unavailable break; default: // Handle other errors break; } } ``` ## Monitoring and Metrics (Optional) ### Prometheus Metrics If Prometheus is enabled with admin privileges, metrics are available at `/metrics` endpoint (default port 10010): **Write Metrics:** - `stream_api_data_packets_published_total`: Number of data packets published - `stream_api_info_packets_published_total`: Number of info packets published - `stream_api_data_packets_routed_total`: Number of data packets routed - `stream_api_data_packets_routed_bytes_total`: Total bytes of data packets routed - `stream_api_info_packets_routed_total`: Number of info packets routed - `stream_api_info_packets_routed_bytes_total`: Total bytes of info packets routed **Read Metrics:** - `stream_api_data_packets_read_total`: Number of data packets read - `stream_api_data_packets_delivered_total`: Number of data packets delivered - `stream_api_essential_packets_read_total`: Number of essential packets read - `stream_api_essential_packets_delivered_total`: Number of essential packets delivered - `stream_api_router_messages_received_total`: Total messages received from router - `stream_api_router_messages_received_bytes_total`: Total message bytes received from router **Resource Metrics:** - `stream_api_connections_total`: Number of active connections - `stream_api_sessions_total`: Number of available sessions - `stream_api_data_formats_total`: Number of data formats stored ## See Also - [Service Documentation](../services/connection-management.md) - [Examples](../examples/telemetry-example.md) - [Error Handling](error-handling.md) --- # Error Handling Guide to handling errors in the Stream API. ## gRPC Status Codes The Stream API uses standard gRPC status codes: | Status Code | Name | Description | |-------------|------|-------------| | 0 | OK | Success | | 3 | INVALID_ARGUMENT | Invalid request parameters | | 5 | NOT_FOUND | Resource not found | | 6 | ALREADY_EXISTS | Resource already exists | | 14 | UNAVAILABLE | Service unavailable | ## Error Handling Example ```csharp try { var connection = await connectionManager.NewConnectionAsync(request); } catch (RpcException ex) { switch (ex.StatusCode) { case StatusCode.InvalidArgument: Console.WriteLine("Invalid parameters"); break; case StatusCode.NotFound: Console.WriteLine("Resource not found"); break; case StatusCode.Unavailable: Console.WriteLine("Service unavailable"); break; } } ``` ## See Also - [API Reference](api-reference.md) - [Troubleshooting](troubleshooting.md) --- # Troubleshooting Common issues, diagnostic steps, and solutions for the Stream API. ## Quick Diagnostics ### Check Metrics ```bash # Check metrics endpoint (if Prometheus is enabled) curl http://localhost:10010/metrics | grep stream_api ``` ### Service Status ```bash # Docker deployment docker-compose ps docker-compose logs -f stream-api # Local deployment ps aux | grep StreamApi netstat -tlnp | grep 13579 ``` ## Monitoring (Optional) ### Key Metrics to Monitor If Prometheus metrics are enabled with admin privileges: ```bash # Write metrics stream_api_data_packets_published_total stream_api_info_packets_published_total stream_api_data_packets_routed_total stream_api_data_packets_routed_bytes_total stream_api_info_packets_routed_total stream_api_info_packets_routed_bytes_total # Read metrics stream_api_data_packets_read_total stream_api_data_packets_delivered_total stream_api_essential_packets_read_total stream_api_essential_packets_delivered_total stream_api_router_messages_received_total stream_api_router_messages_received_bytes_total # Connection and session metrics stream_api_connections_total stream_api_sessions_total stream_api_data_formats_total ``` ### Alerting Rules Example Prometheus alert rules (if using Prometheus): ```yaml # Prometheus alert rules groups: - name: stream-api-alerts rules: - alert: StreamApiDown expr: up{job="stream-api"} == 0 for: 30s - alert: NoPacketsDelivered expr: rate(stream_api_data_packets_delivered_total[5m]) == 0 for: 2m - alert: HighPacketReadLatency expr: stream_api_data_packets_read_total - stream_api_data_packets_delivered_total > 1000 for: 5m ``` ## Common Issues ### Connection Problems #### Issue: "Connection refused" **Symptoms:** - Client cannot connect to server - Error: `UNAVAILABLE: Connection refused` **Diagnosis:** ```bash # Check if server is running curl http://localhost:13579 # Check port availability netstat -an | grep 13579 # Check firewall sudo ufw status ``` **Solutions:** 1. Start the server: `docker-compose up -d` 2. Check port configuration in `AppConfig.json` 3. Verify firewall rules allow port 13579 4. Ensure correct server address in client #### Issue: "Session not found" **Symptoms:** - Error: `NOT_FOUND: Session key 'xyz' not found` - Connection creation fails **Diagnosis:** ```csharp // Check available sessions var streamingApiClient = StreamingApiClientFactory.Create( new StreamingApiConfiguration(StreamCreationStrategy.TopicBased, "localhost:9092", []), new CancellationTokenSourceProvider(), new KafkaBrokerAvailabilityChecker(), new LoggingDirectoryProvider("")); streamingApiClient.Initialise(); var sessionManager = streamingApiClient.GetSessionManagementClient(); var sessions = await sessionManager.ListActiveSessionsAsync(new ListActiveSessionsRequest()); ``` **Solutions:** 1. Create session before creating connections 2. Verify session key spelling 3. Check session hasn't expired 4. Ensure session management is enabled ### Performance Issues #### Issue: High latency **Symptoms:** - Slow packet writing/reading - Timeouts on operations **Diagnosis:** ```bash # Check system resources top free -h iostat -x 1 # Check network ping kafka-broker iperf3 -c kafka-broker # Check metrics (if Prometheus is enabled) curl http://localhost:10010/metrics | grep duration ``` **Solutions:** 1. Increase batch sizes for bulk operations 2. Optimize network configuration 3. Scale up resources (CPU/memory) 4. Use connection pooling 5. Enable response batching #### Issue: Memory leaks **Symptoms:** - Increasing memory usage over time - Out of memory errors **Diagnosis:** ```bash # Monitor memory usage docker stats stream-api # Check GC metrics (if Prometheus is enabled) curl http://localhost:10010/metrics | grep gc ``` **Solutions:** 1. Close connections properly 2. Dispose of clients when done 3. Implement connection pooling 4. Review packet sizes 5. Tune GC settings ### Data Issues #### Issue: Packet loss **Symptoms:** - Missing data packets - Gaps in sequence numbers **Diagnosis:** ```bash # Check Kafka lag docker exec kafka kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups # Check error rates (if Prometheus is enabled) curl http://localhost:10010/metrics | grep error ``` **Solutions:** 1. Check Kafka configuration 2. Verify network stability 3. Implement retry mechanisms 4. Use essential packets for critical data 5. Monitor consumer lag #### Issue: Data corruption **Symptoms:** - Invalid packet content - Deserialization errors **Diagnosis:** ```csharp // Validate packet content using Protocol Buffers try { // Attempt to parse as known protobuf message type var periodicData = PeriodicDataPacket.Parser.ParseFrom(packet.Content); Console.WriteLine($"Successfully parsed PeriodicData with {periodicData.Columns.Count} columns"); } catch (InvalidProtocolBufferException ex) { // Content is not valid protobuf or wrong message type Console.WriteLine($"Failed to parse packet: {ex.Message}"); // Try other message types try { var eventData = EventPacket.Parser.ParseFrom(packet.Content); Console.WriteLine($"Successfully parsed Event: {eventData.DataFormat}"); } catch (InvalidProtocolBufferException) { Console.WriteLine("Packet content is not a recognized protobuf message"); } } ``` **Solutions:** 1. Implement protobuf message validation 2. Use proper message types from open_data.proto 3. Check message serialization using ToByteString() 4. Verify packet type matches content structure 5. Use protobuf reflection to inspect unknown messages ## Kafka-Specific Issues ### Issue: Kafka connection failed **Symptoms:** - Stream API can't connect to Kafka - Error: `Connection to kafka:9092 failed` **Diagnosis:** ```bash # Test Kafka connectivity docker exec stream-api ping kafka # Check Kafka logs docker-compose logs kafka # Test Kafka directly docker exec kafka kafka-topics.sh --bootstrap-server localhost:9092 --list ``` **Solutions:** 1. Verify Kafka is running 2. Check network connectivity 3. Verify broker URLs in configuration 4. Check Kafka authentication settings ### Issue: Topic not found **Symptoms:** - Error: `Topic 'stream-name' does not exist` **Diagnosis:** ```bash # List available topics docker exec kafka kafka-topics.sh --bootstrap-server localhost:9092 --list # Check topic configuration docker exec kafka kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic stream-name ``` **Solutions:** 1. Enable auto-topic creation 2. Create topics manually 3. Check topic naming conventions 4. Verify topic permissions ## Configuration Issues ### Issue: Invalid configuration **Symptoms:** - Server startup fails - Configuration validation errors **Diagnosis:** ```bash # Validate configuration ./StreamApiServer --validate-config --config AppConfig.json # Check JSON syntax python -m json.tool AppConfig.json ``` **Solutions:** 1. Fix JSON syntax errors 2. Verify all required settings 3. Check file permissions 4. Validate configuration values ### Issue: Port conflicts **Symptoms:** - Server fails to start - Error: `Address already in use` **Diagnosis:** ```bash # Check what's using the port lsof -i :13579 netstat -tlnp | grep 13579 ``` **Solutions:** 1. Stop conflicting services 2. Change port configuration 3. Use different port mapping in Docker ## Debug Mode ### Enable Debug Logging ```json // AppConfig.json { "Logging": { "LogLevel": { "Default": "Debug", "MA.Streaming": "Trace" } } } ``` ### Capture Network Traffic ```bash # Capture gRPC traffic tcpdump -i any -s 0 -w stream-api.pcap port 13579 # Analyze with Wireshark wireshark stream-api.pcap ``` ### Enable gRPC Logging ```csharp // Client-side logging Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "DEBUG"); Environment.SetEnvironmentVariable("GRPC_TRACE", "all"); ``` ## Performance Tuning ### Server Optimization ```json { "PerformanceSettings": { "MaxConcurrentConnections": 5000, "ThreadPoolSize": 50, "BatchSize": 1000, "FlushIntervalMs": 50, "MaxMemoryUsageMB": 4096 } } ``` ### Client Optimization ```csharp var channelOptions = new ChannelOption[] { new ChannelOption(ChannelOptions.MaxReceiveMessageLength, 64 * 1024 * 1024), new ChannelOption(ChannelOptions.MaxSendMessageLength, 64 * 1024 * 1024), new ChannelOption("grpc.keepalive_time_ms", 30000), new ChannelOption("grpc.max_connection_idle_ms", 300000) }; ``` ### Kafka Tuning ```properties # server.properties num.network.threads=8 num.io.threads=16 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 ``` ## Recovery Procedures ### Service Recovery ```bash # Restart services docker-compose restart stream-api # Full reset (loses data) docker-compose down -v docker-compose up -d ``` ### Data Recovery ```bash # Restore from backup docker-compose down docker volume rm $(docker volume ls -q) tar -xzf backup.tar.gz docker-compose up -d ``` ## Support Information ### Log Collection ```bash #!/bin/bash # collect-logs.sh DATE=$(date +%Y%m%d_%H%M%S) LOG_DIR="support-logs-$DATE" mkdir -p $LOG_DIR # Collect application logs docker-compose logs --no-color > $LOG_DIR/docker-compose.log docker-compose logs stream-api --no-color > $LOG_DIR/stream-api.log docker-compose logs kafka --no-color > $LOG_DIR/kafka.log # Collect system information docker version > $LOG_DIR/docker-version.txt docker-compose version > $LOG_DIR/compose-version.txt docker stats --no-stream > $LOG_DIR/container-stats.txt docker ps -a > $LOG_DIR/containers.txt # Collect configuration cp -r ./configs $LOG_DIR/ cp docker-compose.yml $LOG_DIR/ # Create archive tar -czf $LOG_DIR.tar.gz $LOG_DIR rm -rf $LOG_DIR echo "Support logs collected: $LOG_DIR.tar.gz" ``` ### Contact Information For additional support: - Documentation: [Stream API Docs](https://atlas.motionapplied.com/stream-api/) - Issues: [GitHub Issues](https://github.com/Software-Products/MA.DataPlatforms.StreamApi/issues) - Support: atlas10@motionapplied.com ## See Also - [Error Handling](error-handling.md) - [API Reference](api-reference.md) - [Performance Tuning](../configuration/server-config.md) --- # Protocol Documentation ## Table of Contents - [ma/streaming/api/v1/api.proto](#ma_streaming_api_v1_api-proto) - [AddAssociateSessionRequest](#ma-streaming-api-v1-AddAssociateSessionRequest) - [AddAssociateSessionResponse](#ma-streaming-api-v1-AddAssociateSessionResponse) - [CloseConnectionRequest](#ma-streaming-api-v1-CloseConnectionRequest) - [CloseConnectionResponse](#ma-streaming-api-v1-CloseConnectionResponse) - [Connection](#ma-streaming-api-v1-Connection) - [ConnectionDetails](#ma-streaming-api-v1-ConnectionDetails) - [CreateSessionRequest](#ma-streaming-api-v1-CreateSessionRequest) - [CreateSessionRequest.DetailsEntry](#ma-streaming-api-v1-CreateSessionRequest-DetailsEntry) - [CreateSessionResponse](#ma-streaming-api-v1-CreateSessionResponse) - [DataPacketDetails](#ma-streaming-api-v1-DataPacketDetails) - [DataPacketRequest](#ma-streaming-api-v1-DataPacketRequest) - [EndSessionRequest](#ma-streaming-api-v1-EndSessionRequest) - [EndSessionResponse](#ma-streaming-api-v1-EndSessionResponse) - [GetConnectionRequest](#ma-streaming-api-v1-GetConnectionRequest) - [GetConnectionResponse](#ma-streaming-api-v1-GetConnectionResponse) - [GetCurrentSessionsRequest](#ma-streaming-api-v1-GetCurrentSessionsRequest) - [GetCurrentSessionsResponse](#ma-streaming-api-v1-GetCurrentSessionsResponse) - [GetEventDataFormatIdRequest](#ma-streaming-api-v1-GetEventDataFormatIdRequest) - [GetEventDataFormatIdResponse](#ma-streaming-api-v1-GetEventDataFormatIdResponse) - [GetEventRequest](#ma-streaming-api-v1-GetEventRequest) - [GetEventResponse](#ma-streaming-api-v1-GetEventResponse) - [GetParameterDataFormatIdRequest](#ma-streaming-api-v1-GetParameterDataFormatIdRequest) - [GetParameterDataFormatIdResponse](#ma-streaming-api-v1-GetParameterDataFormatIdResponse) - [GetParametersListRequest](#ma-streaming-api-v1-GetParametersListRequest) - [GetParametersListResponse](#ma-streaming-api-v1-GetParametersListResponse) - [GetSessionInfoRequest](#ma-streaming-api-v1-GetSessionInfoRequest) - [GetSessionInfoResponse](#ma-streaming-api-v1-GetSessionInfoResponse) - [GetSessionInfoResponse.DetailsEntry](#ma-streaming-api-v1-GetSessionInfoResponse-DetailsEntry) - [GetSessionInfoResponse.TopicPartitionOffsetsEntry](#ma-streaming-api-v1-GetSessionInfoResponse-TopicPartitionOffsetsEntry) - [GetSessionStartNotificationRequest](#ma-streaming-api-v1-GetSessionStartNotificationRequest) - [GetSessionStartNotificationResponse](#ma-streaming-api-v1-GetSessionStartNotificationResponse) - [GetSessionStopNotificationRequest](#ma-streaming-api-v1-GetSessionStopNotificationRequest) - [GetSessionStopNotificationResponse](#ma-streaming-api-v1-GetSessionStopNotificationResponse) - [NewConnectionRequest](#ma-streaming-api-v1-NewConnectionRequest) - [NewConnectionResponse](#ma-streaming-api-v1-NewConnectionResponse) - [PacketResponse](#ma-streaming-api-v1-PacketResponse) - [ReadDataPacketsRequest](#ma-streaming-api-v1-ReadDataPacketsRequest) - [ReadDataPacketsResponse](#ma-streaming-api-v1-ReadDataPacketsResponse) - [ReadEssentialsRequest](#ma-streaming-api-v1-ReadEssentialsRequest) - [ReadEssentialsResponse](#ma-streaming-api-v1-ReadEssentialsResponse) - [ReadPacketsRequest](#ma-streaming-api-v1-ReadPacketsRequest) - [ReadPacketsResponse](#ma-streaming-api-v1-ReadPacketsResponse) - [UpdateSessionDetailsRequest](#ma-streaming-api-v1-UpdateSessionDetailsRequest) - [UpdateSessionDetailsRequest.DetailsEntry](#ma-streaming-api-v1-UpdateSessionDetailsRequest-DetailsEntry) - [UpdateSessionDetailsResponse](#ma-streaming-api-v1-UpdateSessionDetailsResponse) - [UpdateSessionIdentifierRequest](#ma-streaming-api-v1-UpdateSessionIdentifierRequest) - [UpdateSessionIdentifierResponse](#ma-streaming-api-v1-UpdateSessionIdentifierResponse) - [WriteDataPacketRequest](#ma-streaming-api-v1-WriteDataPacketRequest) - [WriteDataPacketResponse](#ma-streaming-api-v1-WriteDataPacketResponse) - [WriteDataPacketsRequest](#ma-streaming-api-v1-WriteDataPacketsRequest) - [WriteDataPacketsResponse](#ma-streaming-api-v1-WriteDataPacketsResponse) - [WriteInfoPacketRequest](#ma-streaming-api-v1-WriteInfoPacketRequest) - [WriteInfoPacketResponse](#ma-streaming-api-v1-WriteInfoPacketResponse) - [WriteInfoPacketsRequest](#ma-streaming-api-v1-WriteInfoPacketsRequest) - [WriteInfoPacketsResponse](#ma-streaming-api-v1-WriteInfoPacketsResponse) - [InfoType](#ma-streaming-api-v1-InfoType) - [ConnectionManagerService](#ma-streaming-api-v1-ConnectionManagerService) - [DataFormatManagerService](#ma-streaming-api-v1-DataFormatManagerService) - [PacketReaderService](#ma-streaming-api-v1-PacketReaderService) - [PacketWriterService](#ma-streaming-api-v1-PacketWriterService) - [SessionManagementService](#ma-streaming-api-v1-SessionManagementService) - [ma/streaming/open_data/v1/open_data.proto](#ma_streaming_open_data_v1_open_data-proto) - [AxisDataPacket](#ma-streaming-open_data-v1-AxisDataPacket) - [BoolSample](#ma-streaming-open_data-v1-BoolSample) - [BoolSampleList](#ma-streaming-open_data-v1-BoolSampleList) - [ConfigurationPacket](#ma-streaming-open_data-v1-ConfigurationPacket) - [CoverageCursorInfoPacket](#ma-streaming-open_data-v1-CoverageCursorInfoPacket) - [DataFormatConfigurationPacket](#ma-streaming-open_data-v1-DataFormatConfigurationPacket) - [DataFormatDefinitionPacket](#ma-streaming-open_data-v1-DataFormatDefinitionPacket) - [DoubleSample](#ma-streaming-open_data-v1-DoubleSample) - [DoubleSampleList](#ma-streaming-open_data-v1-DoubleSampleList) - [EndOfSessionPacket](#ma-streaming-open_data-v1-EndOfSessionPacket) - [EndOfSessionPacket.TopicPartitionOffsetsEntry](#ma-streaming-open_data-v1-EndOfSessionPacket-TopicPartitionOffsetsEntry) - [ErrorPacket](#ma-streaming-open_data-v1-ErrorPacket) - [EventDataFormat](#ma-streaming-open_data-v1-EventDataFormat) - [EventDefinition](#ma-streaming-open_data-v1-EventDefinition) - [EventPacket](#ma-streaming-open_data-v1-EventPacket) - [FormulaDefinition](#ma-streaming-open_data-v1-FormulaDefinition) - [GroupDefinition](#ma-streaming-open_data-v1-GroupDefinition) - [Int32Sample](#ma-streaming-open_data-v1-Int32Sample) - [Int32SampleList](#ma-streaming-open_data-v1-Int32SampleList) - [MapDataPacket](#ma-streaming-open_data-v1-MapDataPacket) - [MarkerPacket](#ma-streaming-open_data-v1-MarkerPacket) - [MetadataPacket](#ma-streaming-open_data-v1-MetadataPacket) - [MetadataPacket.MetadataEntry](#ma-streaming-open_data-v1-MetadataPacket-MetadataEntry) - [NewSessionPacket](#ma-streaming-open_data-v1-NewSessionPacket) - [NewSessionPacket.TopicPartitionOffsetsEntry](#ma-streaming-open_data-v1-NewSessionPacket-TopicPartitionOffsetsEntry) - [Packet](#ma-streaming-open_data-v1-Packet) - [ParameterDefinition](#ma-streaming-open_data-v1-ParameterDefinition) - [ParameterList](#ma-streaming-open_data-v1-ParameterList) - [PeriodicDataPacket](#ma-streaming-open_data-v1-PeriodicDataPacket) - [RawCANDataPacket](#ma-streaming-open_data-v1-RawCANDataPacket) - [RowDataPacket](#ma-streaming-open_data-v1-RowDataPacket) - [SampleColumn](#ma-streaming-open_data-v1-SampleColumn) - [SampleDataFormat](#ma-streaming-open_data-v1-SampleDataFormat) - [SampleRow](#ma-streaming-open_data-v1-SampleRow) - [SessionInfoPacket](#ma-streaming-open_data-v1-SessionInfoPacket) - [SessionInfoPacket.DetailsEntry](#ma-streaming-open_data-v1-SessionInfoPacket-DetailsEntry) - [StringSample](#ma-streaming-open_data-v1-StringSample) - [StringSampleList](#ma-streaming-open_data-v1-StringSampleList) - [SynchroDataPacket](#ma-streaming-open_data-v1-SynchroDataPacket) - [SystemStatusMessage](#ma-streaming-open_data-v1-SystemStatusMessage) - [TextConversionDefinition](#ma-streaming-open_data-v1-TextConversionDefinition) - [ValueArray](#ma-streaming-open_data-v1-ValueArray) - [CanType](#ma-streaming-open_data-v1-CanType) - [DataFormatType](#ma-streaming-open_data-v1-DataFormatType) - [DataStatus](#ma-streaming-open_data-v1-DataStatus) - [DataType](#ma-streaming-open_data-v1-DataType) - [ErrorStatus](#ma-streaming-open_data-v1-ErrorStatus) - [ErrorType](#ma-streaming-open_data-v1-ErrorType) - [EventPriority](#ma-streaming-open_data-v1-EventPriority) - [FormulaType](#ma-streaming-open_data-v1-FormulaType) - [SystemStatusType](#ma-streaming-open_data-v1-SystemStatusType) - [ma/streaming/key_generator/v1/key_generator.proto](#ma_streaming_key_generator_v1_key_generator-proto) - [GenerateUniqueKeyRequest](#ma-streaming-key_generator-v1-GenerateUniqueKeyRequest) - [GenerateUniqueKeyResponse](#ma-streaming-key_generator-v1-GenerateUniqueKeyResponse) - [KeyType](#ma-streaming-key_generator-v1-KeyType) - [UniqueKeyGeneratorService](#ma-streaming-key_generator-v1-UniqueKeyGeneratorService) - [Scalar Value Types](#scalar-value-types)

Top

## ma/streaming/api/v1/api.proto ### AddAssociateSessionRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key of the parent session | | associate_session_key | [string](#string) | | Unique session key of the associate session | ### AddAssociateSessionResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | success | [bool](#bool) | | Whether the add associate session request succeeded | ### CloseConnectionRequest Request to close an existing connection | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | Identifier for the connection | ### CloseConnectionResponse Response to a close connection request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | success | [bool](#bool) | | Whether the close connection request succeeded | ### Connection A connection identifier | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | id | [int64](#int64) | | For internal use only - do not modify this | ### ConnectionDetails Details of a connection | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source to read from | | session_key | [string](#string) | | Session key of the session to read (can be ommitted if sessions are not being used) | | streams | [string](#string) | repeated | Streams to read (can be ommitted if all streams are required) | | stream_offsets | [int64](#int64) | repeated | Offset from which to start for each stream -1 = Latest, 0 = Earliest (default) | | main_offset | [int64](#int64) | | Offset from which to start for the main data source topic | | essentials_offset | [int64](#int64) | | Offset from which to start for the data source essentials topic | | exclude_main_stream | [bool](#bool) | | to specify exclusion of the main stream from reading | ### CreateSessionRequest Request for the creation of a new session | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | type | [string](#string) | | Session type (defaults to "Session") | | version | [uint32](#uint32) | | Version (defaults to 1) | | utc_offset | [google.protobuf.Duration](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Duration.html) | | Difference between UTC time and local standard time in the timezone in which the data is recorded (negative for negative longitudes, positive for positive longitudes, defaults to 0) | | identifier | [string](#string) | | Session identifier | | associate_session_key | [string](#string) | repeated | List of unique session key to associate to the session | | details | [CreateSessionRequest.DetailsEntry](#ma-streaming-api-v1-CreateSessionRequest-DetailsEntry) | repeated | Session details to update (detail name, detail value) | ### CreateSessionRequest.DetailsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [string](#string) | | | ### CreateSessionResponse Response to a session creation request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key | | new_session | [ma.streaming.open_data.v1.NewSessionPacket](#ma-streaming-open_data-v1-NewSessionPacket) | | New session packet | | success | [bool](#bool) | | Whether the create session request succeeded | ### DataPacketDetails Details for the packet being written | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | message | [ma.streaming.open_data.v1.Packet](#ma-streaming-open_data-v1-Packet) | | Packet to write | | data_source | [string](#string) | | Data source for this packet | | stream | [string](#string) | | Stream name to which to write | | session_key | [string](#string) | | Unique session key of the session to which to write | ### DataPacketRequest Request for reading parameters | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | Connection to Read | | include_parameters | [string](#string) | repeated | Parameters to include (regular expressions) | | exclude_parameters | [string](#string) | repeated | Parameters to exclude (regular expressions) | | include_events | [string](#string) | repeated | Events to include (regular expressions) | | exclude_events | [string](#string) | repeated | Events to exclude (regular expressions) | | include_markers | [bool](#bool) | | Include markers? | ### EndSessionRequest Request to end an existing session | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source name | | session_key | [string](#string) | | Unique session key | ### EndSessionResponse Response to the termination of a session | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | end_session | [ma.streaming.open_data.v1.EndOfSessionPacket](#ma-streaming-open_data-v1-EndOfSessionPacket) | | End of session packet | | success | [bool](#bool) | | Whether the end session request succeeded | ### GetConnectionRequest Request for details of an existing connection | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | Identifier for the connection | ### GetConnectionResponse Response to a get connection request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | details | [ConnectionDetails](#ma-streaming-api-v1-ConnectionDetails) | | The connection details | ### GetCurrentSessionsRequest Request to return all sessions present on the broker for a given data source | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source name | ### GetCurrentSessionsResponse Response to the listing of all available sessions on the broker request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_keys | [string](#string) | repeated | List of unique session keys present on the broker for the specified data source | | success | [bool](#bool) | | Whether the get current session request succeeded | ### GetEventDataFormatIdRequest Request for a data format ID for a given event | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source for which this format is to be used | | event | [string](#string) | | Event identifier | ### GetEventDataFormatIdResponse Response for a data format request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format_identifier | [uint64](#uint64) | | Data format identifier | ### GetEventRequest Request for an event from a data format ID | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source for which this format is used | | data_format_identifier | [uint64](#uint64) | | Data format identifier | ### GetEventResponse Response for an event request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | event | [string](#string) | | Event identifier | ### GetParameterDataFormatIdRequest Request for a data format ID for a given set of parameters | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source for which this format is to be used | | parameters | [string](#string) | repeated | Ordered of parameter identifiers (the same parameters in a different order is a different id) | ### GetParameterDataFormatIdResponse Response for a data format request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format_identifier | [uint64](#uint64) | | Data format identifier | ### GetParametersListRequest Request for a list of parameters from a data format ID | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source for which this format is used | | data_format_identifier | [uint64](#uint64) | | Data format identifier | ### GetParametersListResponse Response for a parameter list request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameters | [string](#string) | repeated | List of parameters | ### GetSessionInfoRequest Request to return session info of a given session | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key | ### GetSessionInfoResponse Response to the session info request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | identifier | [string](#string) | | Identifier | | type | [string](#string) | | Type | | version | [uint32](#uint32) | | Version | | associate_session_keys | [string](#string) | repeated | Associate session keys | | is_complete | [bool](#bool) | | Shows whether the session is completed or not | | streams | [string](#string) | repeated | Available streams | | topic_partition_offsets | [GetSessionInfoResponse.TopicPartitionOffsetsEntry](#ma-streaming-api-v1-GetSessionInfoResponse-TopicPartitionOffsetsEntry) | repeated | The offsets into each topic / partition (key is topic name, optionally appended with a ':' followed by the partition number) | | main_offset | [int64](#int64) | | Offset of the main data source topic | | essentials_offset | [int64](#int64) | | Offset of the data source essentials topic | | details | [GetSessionInfoResponse.DetailsEntry](#ma-streaming-api-v1-GetSessionInfoResponse-DetailsEntry) | repeated | Session details (detail name, detail value) | | utc_offset | [google.protobuf.Duration](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Duration.html) | | Difference between UTC time and local standard time in the timezone in which the data is recorded (negative for negative longitudes, positive for positive longitudes) | | success | [bool](#bool) | | Whether the get session info request succeeded | ### GetSessionInfoResponse.DetailsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [string](#string) | | | ### GetSessionInfoResponse.TopicPartitionOffsetsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [int64](#int64) | | | ### GetSessionStartNotificationRequest Request for obtaining a session start notification | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source name | ### GetSessionStartNotificationResponse Response to a session start notification request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key of the started session | | data_source | [string](#string) | | Data source name | ### GetSessionStopNotificationRequest Request for obtaining a session end notification | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data source name | ### GetSessionStopNotificationResponse Response to a session end notification request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key of the ended session | | data_source | [string](#string) | | Data source name | ### NewConnectionRequest Request for a new (read) connection | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | details | [ConnectionDetails](#ma-streaming-api-v1-ConnectionDetails) | | The connection details | ### NewConnectionResponse Response to a new connection request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | Identifier for the connection | ### PacketResponse The return from a packet read | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | packet | [ma.streaming.open_data.v1.Packet](#ma-streaming-open_data-v1-Packet) | | The packet that was read | | stream | [string](#string) | | The stream the packet was read from | | submit_time | [google.protobuf.Timestamp](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Timestamp.html) | | the time that packet submit in the broker topic partition | ### ReadDataPacketsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | request | [DataPacketRequest](#ma-streaming-api-v1-DataPacketRequest) | | | ### ReadDataPacketsResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | response | [PacketResponse](#ma-streaming-api-v1-PacketResponse) | repeated | | ### ReadEssentialsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | | ### ReadEssentialsResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | response | [PacketResponse](#ma-streaming-api-v1-PacketResponse) | repeated | | ### ReadPacketsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | connection | [Connection](#ma-streaming-api-v1-Connection) | | | ### ReadPacketsResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | response | [PacketResponse](#ma-streaming-api-v1-PacketResponse) | repeated | | ### UpdateSessionDetailsRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key | | details | [UpdateSessionDetailsRequest.DetailsEntry](#ma-streaming-api-v1-UpdateSessionDetailsRequest-DetailsEntry) | repeated | Session details to update (detail name, detail value) | ### UpdateSessionDetailsRequest.DetailsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [string](#string) | | | ### UpdateSessionDetailsResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | success | [bool](#bool) | | Whether the update session details request succeeded | ### UpdateSessionIdentifierRequest | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | session_key | [string](#string) | | Unique session key | | identifier | [string](#string) | | Session identifier | ### UpdateSessionIdentifierResponse | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | success | [bool](#bool) | | Whether the update session identifier request succeeded | ### WriteDataPacketRequest Request for writing a data packet to the stream | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | detail | [DataPacketDetails](#ma-streaming-api-v1-DataPacketDetails) | | | ### WriteDataPacketResponse Response to a request for writing a data packet to the stream ### WriteDataPacketsRequest Request for writing a stream of data packets | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | details | [DataPacketDetails](#ma-streaming-api-v1-DataPacketDetails) | repeated | | ### WriteDataPacketsResponse Response to a request for writing a stream of data packets ### WriteInfoPacketRequest Request for writing an info packet to the stream | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | message | [ma.streaming.open_data.v1.Packet](#ma-streaming-open_data-v1-Packet) | | Packet to write | | type | [InfoType](#ma-streaming-api-v1-InfoType) | | Type for this packet | ### WriteInfoPacketResponse Response to a request for writing an info packet to the stream ### WriteInfoPacketsRequest Request for writing a stream of info packets | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | message | [ma.streaming.open_data.v1.Packet](#ma-streaming-open_data-v1-Packet) | | Packet to write | | type | [InfoType](#ma-streaming-api-v1-InfoType) | | Type for this packet | ### WriteInfoPacketsResponse Response to a request for writing a stream of info packets ### InfoType Info type | Name | Number | Description | | ---- | ------ | ----------- | | INFO_TYPE_UNSPECIFIED | 0 | Unspecified | | INFO_TYPE_SESSION_INFO | 1 | Session info | | INFO_TYPE_SYSTEM_STATUS | 2 | System status | ### ConnectionManagerService Connection manager service Open and close Connections Connections maintain the current offset position | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | NewConnection | [NewConnectionRequest](#ma-streaming-api-v1-NewConnectionRequest) | [NewConnectionResponse](#ma-streaming-api-v1-NewConnectionResponse) | Create a new connection | | GetConnection | [GetConnectionRequest](#ma-streaming-api-v1-GetConnectionRequest) | [GetConnectionResponse](#ma-streaming-api-v1-GetConnectionResponse) | Get an existing connection | | CloseConnection | [CloseConnectionRequest](#ma-streaming-api-v1-CloseConnectionRequest) | [CloseConnectionResponse](#ma-streaming-api-v1-CloseConnectionResponse) | Close connection | ### DataFormatManagerService Data format manager service | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | GetParameterDataFormatId | [GetParameterDataFormatIdRequest](#ma-streaming-api-v1-GetParameterDataFormatIdRequest) | [GetParameterDataFormatIdResponse](#ma-streaming-api-v1-GetParameterDataFormatIdResponse) | Request a data format ID for a set of parameters | | GetEventDataFormatId | [GetEventDataFormatIdRequest](#ma-streaming-api-v1-GetEventDataFormatIdRequest) | [GetEventDataFormatIdResponse](#ma-streaming-api-v1-GetEventDataFormatIdResponse) | Request a data format ID for an event | | GetParametersList | [GetParametersListRequest](#ma-streaming-api-v1-GetParametersListRequest) | [GetParametersListResponse](#ma-streaming-api-v1-GetParametersListResponse) | Request a list of paramters for a given data format ID | | GetEvent | [GetEventRequest](#ma-streaming-api-v1-GetEventRequest) | [GetEventResponse](#ma-streaming-api-v1-GetEventResponse) | Request an event for a given data format ID | ### PacketReaderService Read packets | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | ReadPackets | [ReadPacketsRequest](#ma-streaming-api-v1-ReadPacketsRequest) | [ReadPacketsResponse](#ma-streaming-api-v1-ReadPacketsResponse) stream | Continuously read all packets from the specified connection | | ReadEssentials | [ReadEssentialsRequest](#ma-streaming-api-v1-ReadEssentialsRequest) | [ReadEssentialsResponse](#ma-streaming-api-v1-ReadEssentialsResponse) stream | Continuously read all of the essential packets from this connection | | ReadDataPackets | [ReadDataPacketsRequest](#ma-streaming-api-v1-ReadDataPacketsRequest) | [ReadDataPacketsResponse](#ma-streaming-api-v1-ReadDataPacketsResponse) stream | Continuously read all data packets containing any of the specified parameters or events Only data packets will be returned (i.e. no config, metadata etc) | ### PacketWriterService Packet writer service. | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | WriteDataPacket | [WriteDataPacketRequest](#ma-streaming-api-v1-WriteDataPacketRequest) | [WriteDataPacketResponse](#ma-streaming-api-v1-WriteDataPacketResponse) | Write a packet for a specified data source | | WriteDataPackets | [WriteDataPacketsRequest](#ma-streaming-api-v1-WriteDataPacketsRequest) stream | [WriteDataPacketsResponse](#ma-streaming-api-v1-WriteDataPacketsResponse) | Continuously write a stream of data packets | | WriteInfoPacket | [WriteInfoPacketRequest](#ma-streaming-api-v1-WriteInfoPacketRequest) | [WriteInfoPacketResponse](#ma-streaming-api-v1-WriteInfoPacketResponse) | Write an info packet of a specified type | | WriteInfoPackets | [WriteInfoPacketsRequest](#ma-streaming-api-v1-WriteInfoPacketsRequest) stream | [WriteInfoPacketsResponse](#ma-streaming-api-v1-WriteInfoPacketsResponse) | Continuously write a stream of info packets | ### SessionManagementService Manage sessions | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | CreateSession | [CreateSessionRequest](#ma-streaming-api-v1-CreateSessionRequest) | [CreateSessionResponse](#ma-streaming-api-v1-CreateSessionResponse) | Create a new session. Returns the unique session key | | EndSession | [EndSessionRequest](#ma-streaming-api-v1-EndSessionRequest) | [EndSessionResponse](#ma-streaming-api-v1-EndSessionResponse) | End the session for a given unique session key | | GetCurrentSessions | [GetCurrentSessionsRequest](#ma-streaming-api-v1-GetCurrentSessionsRequest) | [GetCurrentSessionsResponse](#ma-streaming-api-v1-GetCurrentSessionsResponse) | Get a list of the current sessions within the broker | | GetSessionInfo | [GetSessionInfoRequest](#ma-streaming-api-v1-GetSessionInfoRequest) | [GetSessionInfoResponse](#ma-streaming-api-v1-GetSessionInfoResponse) | Get the info for a session | | GetSessionStartNotification | [GetSessionStartNotificationRequest](#ma-streaming-api-v1-GetSessionStartNotificationRequest) | [GetSessionStartNotificationResponse](#ma-streaming-api-v1-GetSessionStartNotificationResponse) stream | Continuously notifies the client when a session starts | | GetSessionStopNotification | [GetSessionStopNotificationRequest](#ma-streaming-api-v1-GetSessionStopNotificationRequest) | [GetSessionStopNotificationResponse](#ma-streaming-api-v1-GetSessionStopNotificationResponse) stream | Continuously notifies the client when a session ends | | UpdateSessionIdentifier | [UpdateSessionIdentifierRequest](#ma-streaming-api-v1-UpdateSessionIdentifierRequest) | [UpdateSessionIdentifierResponse](#ma-streaming-api-v1-UpdateSessionIdentifierResponse) | Update the session identifier given its unique session key | | AddAssociateSession | [AddAssociateSessionRequest](#ma-streaming-api-v1-AddAssociateSessionRequest) | [AddAssociateSessionResponse](#ma-streaming-api-v1-AddAssociateSessionResponse) | Add associate session to a session given its unique session key | | UpdateSessionDetails | [UpdateSessionDetailsRequest](#ma-streaming-api-v1-UpdateSessionDetailsRequest) | [UpdateSessionDetailsResponse](#ma-streaming-api-v1-UpdateSessionDetailsResponse) | Update the session details for a session given its unique session key |

Top

## ma/streaming/open_data/v1/open_data.proto ### AxisDataPacket Values corresponsponding to points on an axis for map parameters | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | axis_identifier | [string](#string) | | Axis identifier | | timestamp | [fixed64](#fixed64) | | Timestamp | | values | [ValueArray](#ma-streaming-open_data-v1-ValueArray) | | The data for the axis | ### BoolSample | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | value | [bool](#bool) | | | | status | [DataStatus](#ma-streaming-open_data-v1-DataStatus) | | | ### BoolSampleList | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | samples | [BoolSample](#ma-streaming-open_data-v1-BoolSample) | repeated | | ### ConfigurationPacket Data configuration | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | config_id | [string](#string) | | Configuration ID | | parameter_definitions | [ParameterDefinition](#ma-streaming-open_data-v1-ParameterDefinition) | repeated | The parameter definitions | | event_definitions | [EventDefinition](#ma-streaming-open_data-v1-EventDefinition) | repeated | The event definitions | | group_definitions | [GroupDefinition](#ma-streaming-open_data-v1-GroupDefinition) | repeated | The group definitions | | partial_packet | [bool](#bool) | | Flag to indicate if this packet is part of a larger config. | | packet_number | [uint32](#uint32) | | The packet number out of the total packets sent if partial packet was set to true. | | total_packet_number | [uint32](#uint32) | | The total packet number that make up the config id if partial packet was set to true. | ### CoverageCursorInfoPacket Coverage cursor position for a session has been updated | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | coverage_cursor_time | [fixed64](#fixed64) | | Coverage cursor time (nanoseconds since the UNIX epoch of 1st January 1970, UTC) | ### DataFormatConfigurationPacket Data format | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format_id | [string](#string) | | Data format ID | | data_formats | [DataFormatDefinitionPacket](#ma-streaming-open_data-v1-DataFormatDefinitionPacket) | repeated | The data format definitions | ### DataFormatDefinitionPacket Data Format Definition | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | identifier | [uint64](#uint64) | | Data format identifier | | type | [DataFormatType](#ma-streaming-open_data-v1-DataFormatType) | | Data format type | | parameter_identifiers | [ParameterList](#ma-streaming-open_data-v1-ParameterList) | | | | event_identifier | [string](#string) | | | ### DoubleSample | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | value | [double](#double) | | | | status | [DataStatus](#ma-streaming-open_data-v1-DataStatus) | | | ### DoubleSampleList | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | samples | [DoubleSample](#ma-streaming-open_data-v1-DoubleSample) | repeated | | ### EndOfSessionPacket A session has ended | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | topic_partition_offsets | [EndOfSessionPacket.TopicPartitionOffsetsEntry](#ma-streaming-open_data-v1-EndOfSessionPacket-TopicPartitionOffsetsEntry) | repeated | The offsets into each topic / partition (key is topic name, optionally appended with a ':' followed by the partition number) | ### EndOfSessionPacket.TopicPartitionOffsetsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [int64](#int64) | | | ### ErrorPacket Error Packet | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | error_identifier | [string](#string) | | Error Identifier | | timestamp | [fixed64](#fixed64) | | Time the error occurred | | application_name | [string](#string) | | The application containing the error | | group | [string](#string) | | The error group | | name | [string](#string) | | The error name | | description | [string](#string) | | The error description | | type | [ErrorType](#ma-streaming-open_data-v1-ErrorType) | | The error type | | status | [ErrorStatus](#ma-streaming-open_data-v1-ErrorStatus) | | The error status | ### EventDataFormat Desribes the event in this data packet | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format_identifier | [uint64](#uint64) | | | | event_identifier | [string](#string) | | | ### EventDefinition Defines the metadata for a given event | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | identifier | [string](#string) | | The full identifier for the event | | name | [string](#string) | | The name of the event | | definition_id | [uint32](#uint32) | | A numeric identifier for the event | | application_name | [string](#string) | | The application containing the event | | description | [string](#string) | | The event description | | priority | [EventPriority](#ma-streaming-open_data-v1-EventPriority) | | Event priority | | groups | [string](#string) | repeated | List of group identifiers in which the event should appear | | data_types | [DataType](#ma-streaming-open_data-v1-DataType) | repeated | List of native data types for each value | | format_strings | [string](#string) | repeated | List of format strings, applied to event values for display | | conversions | [TextConversionDefinition](#ma-streaming-open_data-v1-TextConversionDefinition) | repeated | Text conversion rules to be applied to event values if applicable | | units | [string](#string) | repeated | List of units applied to event values for display | ### EventPacket Event | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format | [EventDataFormat](#ma-streaming-open_data-v1-EventDataFormat) | | Data format | | timestamp | [fixed64](#fixed64) | | Time the event occurred | | raw_values | [double](#double) | repeated | Raw data (list of doubles, event specific meaning) | ### FormulaDefinition A formula definition | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | type | [FormulaType](#ma-streaming-open_data-v1-FormulaType) | | Type of formula | | formula | [string](#string) | | The formula code | ### GroupDefinition Defines the group | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | identifier | [string](#string) | | The full identifier for the group | | name | [string](#string) | | The name of the group | | application_name | [string](#string) | | The applicaion that the group is contained within | | description | [string](#string) | | Description for the group | | groups | [GroupDefinition](#ma-streaming-open_data-v1-GroupDefinition) | repeated | Any subgroups nested within the group | ### Int32Sample | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | value | [int32](#int32) | | | | status | [DataStatus](#ma-streaming-open_data-v1-DataStatus) | | | ### Int32SampleList | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | samples | [Int32Sample](#ma-streaming-open_data-v1-Int32Sample) | repeated | | ### MapDataPacket Values corresponsponding to the map | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | map_identifier | [string](#string) | | Map identifier | | timestamp | [fixed64](#fixed64) | | Timestamp | | x_axis_identifier | [string](#string) | | The identifier of the X axis | | x_index_identifier | [string](#string) | | The identifier of the X axis index parameter | | y_axis_identifier | [string](#string) | | The name of the Y axis (only for 2D maps) | | y_index_identifier | [string](#string) | | The identifier of the Y axis index parameter (only for 2D maps) | | values | [ValueArray](#ma-streaming-open_data-v1-ValueArray) | repeated | The data for the map | ### MarkerPacket Marker | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | timestamp | [fixed64](#fixed64) | | Start time of marker | | end_time | [fixed64](#fixed64) | | End time of marker (optional) | | label | [string](#string) | | Text label | | type | [string](#string) | | Type of marker | | description | [string](#string) | | Text Description | | source | [string](#string) | | Source | | value | [int64](#int64) | | Value of the marker if applicable | ### MetadataPacket General metadata | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | metadata | [MetadataPacket.MetadataEntry](#ma-streaming-open_data-v1-MetadataPacket-MetadataEntry) | repeated | Metadata | ### MetadataPacket.MetadataEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [google.protobuf.Any](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Any.html) | | | ### NewSessionPacket New session has started | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | topic_partition_offsets | [NewSessionPacket.TopicPartitionOffsetsEntry](#ma-streaming-open_data-v1-NewSessionPacket-TopicPartitionOffsetsEntry) | repeated | The offsets into each topic / partition (key is topic name, optionally appended with a ':' followed by the partition number) | | utc_offset | [google.protobuf.Duration](https://protobuf.dev/reference/php/api-docs/Google/Protobuf/Duration.html) | | Difference between UTC time and local standard time in the timezone in which the data is recorded (negative for negative longitudes, positive for positive longitudes) | | session_info | [SessionInfoPacket](#ma-streaming-open_data-v1-SessionInfoPacket) | optional | Session info | ### NewSessionPacket.TopicPartitionOffsetsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [int64](#int64) | | | ### Packet Wrapper for all packets | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | type | [string](#string) | | Message type (unique name) | | session_key | [string](#string) | | Unique session key | | is_essential | [bool](#bool) | | Is this packet essential? | | content | [bytes](#bytes) | | Content | | id | [uint64](#uint64) | | Id (optional if needed by default is 0) | ### ParameterDefinition Defines the metadata for a given parameter | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | identifier | [string](#string) | | The full identifier for the parameter | | name | [string](#string) | | The name of the parameter | | application_name | [string](#string) | | The application containing the parameter | | description | [string](#string) | | The parameter description | | groups | [string](#string) | repeated | List of group identifiers in which the parameter should appear | | units | [string](#string) | | Units that this parameter is measured in | | data_type | [DataType](#ma-streaming-open_data-v1-DataType) | | Native data type | | format_string | [string](#string) | | Format strings, applied to sample values for display | | min_value | [double](#double) | | Minimum value expected for samples of this parameter | | max_value | [double](#double) | | Maximum value expected for samples of this parameter | | warning_min_value | [double](#double) | | Threshold minimum value for warnings | | warning_max_value | [double](#double) | | Threshold maximum value for warnings | | frequencies | [double](#double) | repeated | Frequencies at which samples of this parameter may be sent | | includes_row_data | [bool](#bool) | | Whether row data samples may be sent for this parameter | | includes_synchro_data | [bool](#bool) | | Whether synchro data samples may be sent for this parameter | | conversion | [TextConversionDefinition](#ma-streaming-open_data-v1-TextConversionDefinition) | | Text conversion rule to be applied if applicable | | formula | [FormulaDefinition](#ma-streaming-open_data-v1-FormulaDefinition) | | Formula for calculating sample values if applicable | ### ParameterList List of parameter identifiers | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | parameter_identifiers | [string](#string) | repeated | | ### PeriodicDataPacket Telemetry samples in periodic format (equally spaced / fixed frequency samples) | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format | [SampleDataFormat](#ma-streaming-open_data-v1-SampleDataFormat) | | Data format | | start_time | [fixed64](#fixed64) | | Start time (timestamp of first sample) | | interval | [uint32](#uint32) | | Step / time between samples | | columns | [SampleColumn](#ma-streaming-open_data-v1-SampleColumn) | repeated | The rest of the Data Each column must have the same no of values | ### RawCANDataPacket Raw CAN message | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | timestamp | [fixed64](#fixed64) | | Timestamp of the packet | | bus | [uint32](#uint32) | | CAN Bus | | can_id | [uint32](#uint32) | | CAN ID | | payload | [bytes](#bytes) | | Message payload | | type | [CanType](#ma-streaming-open_data-v1-CanType) | | CAN RxTx | ### RowDataPacket Telemetry samples in time series format (timestamped rows) | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format | [SampleDataFormat](#ma-streaming-open_data-v1-SampleDataFormat) | | Data format | | timestamps | [fixed64](#fixed64) | repeated | Timestamps | | rows | [SampleRow](#ma-streaming-open_data-v1-SampleRow) | repeated | The rest of the Data Each row must have the same no of values = the same as the parameters | ### SampleColumn "Column" of telemetry sample values | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | double_samples | [DoubleSampleList](#ma-streaming-open_data-v1-DoubleSampleList) | | | | int32_samples | [Int32SampleList](#ma-streaming-open_data-v1-Int32SampleList) | | | | bool_samples | [BoolSampleList](#ma-streaming-open_data-v1-BoolSampleList) | | | | string_samples | [StringSampleList](#ma-streaming-open_data-v1-StringSampleList) | | | ### SampleDataFormat Desribes the parameters in this data packet | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format_identifier | [uint64](#uint64) | | | | parameter_identifiers | [ParameterList](#ma-streaming-open_data-v1-ParameterList) | | | ### SampleRow "Row" of telemetry sample values | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | double_samples | [DoubleSampleList](#ma-streaming-open_data-v1-DoubleSampleList) | | | | int32_samples | [Int32SampleList](#ma-streaming-open_data-v1-Int32SampleList) | | | | bool_samples | [BoolSampleList](#ma-streaming-open_data-v1-BoolSampleList) | | | | string_samples | [StringSampleList](#ma-streaming-open_data-v1-StringSampleList) | | | ### SessionInfoPacket Session info | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_source | [string](#string) | | Data Source name | | identifier | [string](#string) | | Identifier | | type | [string](#string) | | Type | | version | [uint32](#uint32) | | Version | | associate_session_keys | [string](#string) | repeated | Associate session keys | | details | [SessionInfoPacket.DetailsEntry](#ma-streaming-open_data-v1-SessionInfoPacket-DetailsEntry) | repeated | Session details (detail name, detail value) | ### SessionInfoPacket.DetailsEntry | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | key | [string](#string) | | | | value | [string](#string) | | | ### StringSample | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | value | [string](#string) | | | | status | [DataStatus](#ma-streaming-open_data-v1-DataStatus) | | | ### StringSampleList | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | samples | [StringSample](#ma-streaming-open_data-v1-StringSample) | repeated | | ### SynchroDataPacket Telemetry samples in synchro format (variable frequency samples) | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | data_format | [SampleDataFormat](#ma-streaming-open_data-v1-SampleDataFormat) | | Data format | | start_time | [fixed64](#fixed64) | | Start time (timestamp of first sample) | | intervals | [uint32](#uint32) | repeated | Step / time between each sample and the previous | | column | [SampleColumn](#ma-streaming-open_data-v1-SampleColumn) | repeated | The rest of the Data Each column must have number of values = number of intervals + 1 | ### SystemStatusMessage A system status message | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | service | [string](#string) | | Service to which this message relates | | data_source | [string](#string) | | Data source name that this service relates to | | type | [SystemStatusType](#ma-streaming-open_data-v1-SystemStatusType) | | Type of message Contents......... TBD | ### TextConversionDefinition A text conversion definition | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | conversion_identifier | [string](#string) | | The conversion rule identifier | | input_values | [double](#double) | repeated | The values to convert from | | string_values | [string](#string) | repeated | The strings to convert to | | default | [string](#string) | | The default string if the input does not match | ### ValueArray "Array" of values | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | values | [double](#double) | repeated | | ### CanType CAN type | Name | Number | Description | | ---- | ------ | ----------- | | CAN_TYPE_UNSPECIFIED | 0 | | | CAN_TYPE_TRANSMIT | 1 | | | CAN_TYPE_RECEIVE | 2 | | ### DataFormatType Data format type | Name | Number | Description | | ---- | ------ | ----------- | | DATA_FORMAT_TYPE_UNSPECIFIED | 0 | | | DATA_FORMAT_TYPE_PARAMETER | 1 | | | DATA_FORMAT_TYPE_EVENT | 2 | | ### DataStatus Data status showing if samples are invalid for any reason | Name | Number | Description | | ---- | ------ | ----------- | | DATA_STATUS_UNSPECIFIED | 0 | | | DATA_STATUS_VALID | 1 | | | DATA_STATUS_MISSING | 2 | | | DATA_STATUS_ERROR | 3 | | ### DataType Native data type | Name | Number | Description | | ---- | ------ | ----------- | | DATA_TYPE_UNSPECIFIED | 0 | Unspecified | | DATA_TYPE_FLOAT64 | 1 | 64 bit floating point (i.e. “double”) | | DATA_TYPE_FLOAT32 | 2 | 32 bit floating point | | DATA_TYPE_UINT32 | 3 | 32 bit unsigned integer | | DATA_TYPE_SINT32 | 4 | 32 bit signed integer | | DATA_TYPE_UINT16 | 5 | 16 bit unsigned integer | | DATA_TYPE_SINT16 | 6 | 16 bit signed integer | | DATA_TYPE_UINT8 | 7 | 8 bit unsigned integer | | DATA_TYPE_SINT8 | 8 | 8 bit signed integer | | DATA_TYPE_STRING | 9 | String | ### ErrorStatus Error Status | Name | Number | Description | | ---- | ------ | ----------- | | ERROR_STATUS_UNSPECIFIED | 0 | | | ERROR_STATUS_SET | 1 | | | ERROR_STATUS_CLEARED | 2 | | ### ErrorType Error type | Name | Number | Description | | ---- | ------ | ----------- | | ERROR_TYPE_UNSPECIFIED | 0 | | | ERROR_TYPE_CURRENT | 1 | | | ERROR_TYPE_LOGGED | 2 | | ### EventPriority Event priority type | Name | Number | Description | | ---- | ------ | ----------- | | EVENT_PRIORITY_UNSPECIFIED | 0 | | | EVENT_PRIORITY_CRITICAL | 1 | | | EVENT_PRIORITY_HIGH | 2 | | | EVENT_PRIORITY_MEDIUM | 3 | | | EVENT_PRIORITY_LOW | 4 | | | EVENT_PRIORITY_DEBUG | 5 | | ### FormulaType Types of formula. Can be extended to support other languages in the future | Name | Number | Description | | ---- | ------ | ----------- | | FORMULA_TYPE_UNSPECIFIED | 0 | Unspecified | | FORMULA_TYPE_FDL | 1 | Motion Applied proprietary FDL, used for virtual parameters with System Monitor | | FORMULA_TYPE_CSHARP | 2 | An open format translated from FDL to C# | ### SystemStatusType Types of status message | Name | Number | Description | | ---- | ------ | ----------- | | SYSTEM_STATUS_TYPE_UNSPECIFIED | 0 | Unspecified | | SYSTEM_STATUS_TYPE_HEARTBEAT | 1 | Service is still alive | | SYSTEM_STATUS_TYPE_STATUS | 2 | Service has a status update | | SYSTEM_STATUS_TYPE_ERROR | 3 | Service has generated an error | | SYSTEM_STATUS_TYPE_WARNING | 4 | Service has generated a warning | | SYSTEM_STATUS_TYPE_CONNECTION | 5 | A source has connected/disconnected |

Top

## ma/streaming/key_generator/v1/key_generator.proto ### GenerateUniqueKeyRequest Request for generating a unique key | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | type | [KeyType](#ma-streaming-key_generator-v1-KeyType) | | | ### GenerateUniqueKeyResponse Response to a request for generating a unique key | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | string_key | [string](#string) | | | | ulong_key | [uint64](#uint64) | | | ### KeyType The type of key that should be generated | Name | Number | Description | | ---- | ------ | ----------- | | KEY_TYPE_UNSPECIFIED | 0 | | | KEY_TYPE_ULONG | 1 | | | KEY_TYPE_STRING | 2 | | ### UniqueKeyGeneratorService Unique key generator service | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | GenerateUniqueKey | [GenerateUniqueKeyRequest](#ma-streaming-key_generator-v1-GenerateUniqueKeyRequest) | [GenerateUniqueKeyResponse](#ma-streaming-key_generator-v1-GenerateUniqueKeyResponse) | Generate a unique key which can be used to uniquely identify entities | ## Scalar Value Types | .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | | ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | | double | | double | double | float | float64 | double | float | Float | | float | | float | float | float | float32 | float | float | Float | | int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | | uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | | uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | | sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | | fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | | fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | | sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | | sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | | bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | | string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | | bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | --- # Overview ## Introduction The Bridge Service decodes a raw stream of quads from an ATLAS Data Server (ADS), converting the data to engineering values then into the Stream Protocol via the Stream API. It includes components to manage sessions, configuration, metadata etc. All RDA protected data will be filtered out at this point, based on the active team license, so any downstream components will only receive data available to the team. ## Choose your deployment Bridge Service is the same program in every deployment. What changes is how it is started, and therefore which configuration file it reads. | How Bridge Service runs | Configuration file it reads | |---|---| | **Launched by ADS** (installed with ADS) | `Documents\McLaren Electronic Systems\ATLAS 9\BridgeService\BridgeServiceConfig.json` | | **Standalone on Windows** (you run `MA.DataPlatforms.Bridge.Host.exe` yourself) | `Configs\AppConfig.json` next to the exe, or the file you pass with `-c` | | **Docker** (`atlasplatformdocker/bridge-service-host`) | `/app/Configs/AppConfig.json` in the container, or the file named by the `CONFIG_FILE_PATH` environment variable | `BridgeServiceConfig.json` and `AppConfig.json` have the same format and accept the same sections, described in the [Configuration Guide](configuration-guide.md). === "Launched by ADS" On first launch, ADS copies the default `AppConfig.json` from `\BridgeService\Configs\` to `Documents\McLaren Electronic Systems\ATLAS 9\BridgeService\BridgeServiceConfig.json`. From then on, ADS always starts Bridge Service with that copy. - **Edit `BridgeServiceConfig.json`**, not the `AppConfig.json` in the install folder. Once the copy exists, changes to the install-folder file have no effect. - To start again from the default file (for example after upgrading ADS), delete `BridgeServiceConfig.json`. ADS creates a new copy the next time it launches Bridge Service. - ADS passes its own feed port to Bridge Service, which overrides `BridgeConfig.FeedPort` in the file. - If `EssentialsConfig` and `RdaConfig` paths are left empty, Bridge Service uses the config, PGV and PUL folders set in ATLAS 9 for the Windows user running it. Bridge service uses the name of the ADS as the DataSource name. This can be configured within the ADS on Setup | General | General Network Settings | Name. === "Standalone on Windows" Run `MA.DataPlatforms.Bridge.Host.exe`. It accepts three optional command-line arguments: | Argument | Meaning | Example | |---|---|---| | `-c` | Configuration file path | `-c "C:\Bridge\MyConfig.json"` | | `-l` | Log file path (see [Logging](configuration-guide.md#7-serilog-configuration)) | `-l "C:\Logs\bridge.txt"` | | `-p` | Feed port that ADS connects to | `-p 9697` | !!! warning "Use an absolute `-c` path, or start from the install folder" A relative configuration path is resolved from the folder Bridge Service is started from. If you start it from anywhere else, pass an absolute path with `-c`. As with ADS, empty `EssentialsConfig` and `RdaConfig` paths fall back to the config, PGV and PUL folders set in ATLAS 9 for the Windows user running Bridge Service. === "Docker" The image runs from `/app` and reads `/app/Configs/AppConfig.json`. Either mount your own file over it, or mount it anywhere and set `CONFIG_FILE_PATH` to its absolute path inside the container. ```bash docker run -p 9697:9697 -p 10011:10011 -v /srv/bridge/AppConfig.json:/app/Configs/AppConfig.json:ro -v /srv/essentials:/essentials:ro atlasplatformdocker/bridge-service-host ``` !!! warning "Settings you must change for Docker" - **`EssentialsConfig.CfgDirectoryPaths` and `PgvDirectoryPaths` are required.** There is no ATLAS 9 fallback in a container: if either is empty, Bridge Service stops at startup with *"DirectoryPaths needs to be specified in EssentialsConfig"*. List the folders as they appear **inside** the container. If you use PUL files, set `RdaConfig.PulFilesSearchPaths` the same way. - **`StreamApiConfig.BrokerUrl`**: `localhost` refers to the container itself. Point it at your Kafka broker's host name. - **Ports**: publish the feed port (default `9697`) that ADS connects to, and the Prometheus metrics port (`10011` in the default configuration file). Any setting can also be overridden with an [environment variable](configuration-guide.md#overriding-configuration-with-environment-variables), for example `-e StreamApiConfig__BrokerUrl=kafka:9092`. ## Splitting data across Kafka topics or partitions The data can be split across multiple Kafka topics or partitions. For more information, refer to [Stream Creation Strategy](../stream_api/reference_docs/configuration/server-config.md/#stream-creation-strategies). The examples below show only the two relevant sections. !!! note "Add or update — don't replace the whole file" Change these sections in your existing configuration file and leave the other sections in place. === "Example with StreamCreationStrategy=2 (Topic based)" ```json title="BridgeServiceConfig.json" linenums="1" hl_lines="3" { "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "localhost:9094", "PartitionMappings": [ {} ], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "BatchingResponses": false }, "StreamSelectionConfig": { "Auto": true, "Mappings": [ {} ] } } ``` === "Example with StreamCreationStrategy=1 (Partition based)" ```json title="BridgeServiceConfig.json" linenums="1" hl_lines="3" { "StreamApiConfig": { "StreamCreationStrategy": 1, "BrokerUrl": "localhost:9094", "PartitionMappings": [ { "Stream": "Stream1", "Partition": 1 }, { "Stream": "Stream2", "Partition": 2 }, { "Stream": "Stream3", "Partition": 3 }, { "Stream": "Stream4", "Partition": 4 }, { "Stream": "Stream5", "Partition": 5 }, { "Stream": "Stream6", "Partition": 6 }, { "Stream": "Stream7", "Partition": 7 }, { "Stream": "Stream8", "Partition": 8 }, { "Stream": "Stream9", "Partition": 9 } ], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "BatchingResponses": false }, "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "ATLAS", "Stream": "Stream1" }, { "AppName": "FIA", "Stream": "Stream2" }, { "AppName": "Chassis", "Stream": "Stream3" }, { "AppName": "Arbitrator", "Stream": "Stream4" }, { "AppName": "Coordinator", "Stream": "Stream5" }, { "AppName": "Controller", "Stream": "Stream6" }, { "AppName": "Dash", "Stream": "Stream7" }, { "AppName": "BrakeControl", "Stream": "Stream8" }, { "AppName": "TAG320BIOS", "Stream": "Stream9" } ] } } ``` !!! info "Logs" From Bridge Service 2.1.1, the default configuration file logs to the console window only, so no log file is written unless you configure one. See [Logging](configuration-guide.md#7-serilog-configuration). ## Setup **Prerequisite:** Kafka broker - The Bridge service needs to be configured to write to a Kafka broker. As you can see in the above example, the config file points to a broker url, which is hosting kafka locally on the same machine as the ADS. To enable the Bridge Service, set "Enable remote data feed" setting to "True" in ADS | Tools | Options | Recording tab | General ![Enable remote data feed](../assets/enable_remote_data_feed.png) Restart ADS once the setting is applied. Once the data is published on Kafka, users can use the Stream API to consume the data. For more information, please refer to the [Stream API Sample code](https://github.com/mat-docs/MA.Streaming.Api.UsageSample) !!! note Currently, ADS will launch the Bridge Service in a separate command prompt for ease of testing/validation and diagnostics. ## UNIX Timestamps Timestamps from Motion Applied ECUs are relative to midnight local time. To convert the timestamps to UNIX epoch time in UTC, the date and the local timezone offset on the machine hosting the bridge service is used. !!! warning It also applies to File Telemetry Recorders and Multi-Session Offload Recorders in addition to live telemetry. Replaying a raw file on a machine that is incorrectly configured could publish timestamps at the incorrect date and timezone. --- # Configuration Guide The MA DataPlatforms Bridge service uses multiple configuration sections to control various aspects of data streaming, processing, and integration. This guide provides comprehensive documentation for all configuration sections. ## Configuration File Structure The configuration file is `AppConfig.json` when you run Bridge Service yourself (standalone or in Docker), and `BridgeServiceConfig.json` when ADS launches it. Both have the same format — see [Choose your deployment](index.md#choose-your-deployment) for which file applies to you. It contains the following top-level sections: ```json { "StreamApiConfig": { ... }, "StreamSelectionConfig": { ... }, "BridgeConfig": { ... }, "EssentialsConfig": { ... }, "RdaConfig": { ... }, "DiskBackedBufferConfig": { ... }, "Serilog": { ... } } ``` ## Overriding Configuration with Environment Variables Bridge Service loads `AppConfig.json` first, then applies any matching environment variables on top of it — environment variables take precedence over values in the JSON file. This applies in both deployment modes (standalone host and ADS-integrated). Address nested configuration keys with a double underscore (`__`) between each segment: ```powershell $env:StreamApiConfig__BrokerUrl = "kafka-broker.company.com:9092" $env:BridgeConfig__DataSource = "RaceTrack01" ``` !!! tip Use environment variables for values you don't want committed to `AppConfig.json` — for example broker credentials — instead of writing them into the file. !!! note Environment variables are read once at startup. Restart the Bridge Service after changing one. ## 1. StreamApiConfig Controls the Stream API integration and Kafka broker connectivity. ### Properties | Property | Type | Default | Description | | :--- | :--- | :--- | :--- | | `StreamCreationStrategy` | int/enum| 2 | Strategy for creating Kafka topics/streams
• `1` = PartitionBased
• `2` = TopicBased (recommended) | | `BrokerUrl` | string | "localhost:9092" | Kafka broker connection string | | `PartitionMappings` | array | [] | Partition mapping configurations (PartitionBased only) | | `IntegrateSessionManagement` | boolean | true | Enable session management integration | | `IntegrateDataFormatManagement`| boolean | true | Enable data format management integration | | `UseRemoteKeyGenerator` | boolean | false | Use remote key generator service | | `RemoteKeyGeneratorServiceAddress` | string | "" | URL of remote key generator service | | `InitialisationTimeoutSeconds` | int | 1 | Timeout in seconds for Stream API init | | `Domain` | string | "" | Optional domain specification | | `Security` | object | null | Kafka SASL/SSL credentials — see [Kafka Security (SASL / SSL)](#kafka-security-sasl-ssl) below | ### Example Configuration ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "kafka-broker.company.com:9092", "PartitionMappings": [], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "InitialisationTimeoutSeconds": 1, "Domain": "Telemetry" } } ``` ### Kafka Security (SASL / SSL) If your Kafka broker requires authentication or an encrypted connection, add a `Security` object inside `StreamApiConfig`. Security is only activated once you set `SaslUsername`, `SslCaLocation`, or `SslCertificateLocation` — leave `Security` out entirely to connect without any security, which is unchanged for existing unsecured deployments. | Property | Type | Description | | :--- | :--- | :--- | | `SaslUsername` | string? | SASL username. Setting this activates security. | | `SaslPassword` | string? | SASL password. | | `Protocol` | string? | `Plaintext`, `Ssl`, `SaslPlaintext`, or `SaslSsl`. Defaults to `SaslPlaintext` once security is activated. | | `Mechanism` | string? | `Plain`, `ScramSha256`, or `ScramSha512`. Defaults to `Plain` once security is activated. | | `SslCaLocation` | string? | Path to the CA certificate (PEM) used to verify the broker's certificate. | | `SslCertificateLocation` | string? | Path to the client certificate (PEM), for mutual TLS. Setting this activates security. | | `SslKeyLocation` | string? | Path to the client private key (PEM), for mutual TLS. | | `SslKeyPassword` | string? | Password for the client private key, if it's encrypted. | === "SASL/PLAIN" ```json title="AppConfig.json" linenums="1" hl_lines="4-7" { "StreamApiConfig": { "BrokerUrl": "kafka-broker.company.com:9094", "Security": { "Protocol": "SaslPlaintext", "Mechanism": "Plain", "SaslUsername": "myuser", "SaslPassword": "mypassword" } } } ``` === "SASL/SCRAM-SHA-512" ```json title="AppConfig.json" linenums="1" hl_lines="4-7" { "StreamApiConfig": { "BrokerUrl": "kafka-broker.company.com:9094", "Security": { "Protocol": "SaslPlaintext", "Mechanism": "ScramSha512", "SaslUsername": "myuser", "SaslPassword": "mypassword" } } } ``` === "SSL (broker certificate only)" ```json title="AppConfig.json" linenums="1" hl_lines="4-6" { "StreamApiConfig": { "BrokerUrl": "kafka-broker.company.com:9094", "Security": { "Protocol": "Ssl", "SslCaLocation": "/certs/ca.pem" } } } ``` === "SSL (mutual TLS)" ```json title="AppConfig.json" linenums="1" hl_lines="4-8" { "StreamApiConfig": { "BrokerUrl": "kafka-broker.company.com:9094", "Security": { "Protocol": "Ssl", "SslCaLocation": "/certs/ca.pem", "SslCertificateLocation": "/certs/client-cert.pem", "SslKeyLocation": "/certs/client-key.pem", "SslKeyPassword": "keypassword" } } } ``` === "SASL over TLS" ```json title="AppConfig.json" linenums="1" hl_lines="4-8" { "StreamApiConfig": { "BrokerUrl": "kafka-broker.company.com:9094", "Security": { "Protocol": "SaslSsl", "Mechanism": "ScramSha512", "SaslUsername": "myuser", "SaslPassword": "mypassword", "SslCaLocation": "/certs/ca.pem" } } } ``` !!! tip Bridge Service also reads [environment variables](#overriding-configuration-with-environment-variables), so credentials can be supplied without writing them into `AppConfig.json`: ``` StreamApiConfig__Security__SaslUsername=myuser StreamApiConfig__Security__SaslPassword=mypassword ``` ### Stream Creation Strategy Explained The Bridge service works with Stream API's two streaming strategies. Your choice affects how Bridge streams are mapped to Kafka topics. #### Topic-Based Strategy (`StreamCreationStrategy = 2`) Each stream from the Bridge service maps to its own Kafka topic. This is simpler and recommended for most use cases. **How Mapping Works:** * Bridge uses `StreamSelectionConfig` to determine stream names. * Each unique stream name creates a separate Kafka topic. * **Topic naming:** `{Domain}.Data.{StreamName}` !!! example "Example Scenario" **Bridge Configuration:** ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "Domain": "Production" }, "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "Engine*", "Stream": "EngineTelemetry" }, { "AppName": "Chassis*", "Stream": "ChassisTelemetry" }, { "AppName": "Aero*", "Stream": "AeroTelemetry" } ] } } ``` **Resulting Kafka Topics:** * `Production.Data.EngineTelemetry` (All Engine* app groups) * `Production.Data.ChassisTelemetry` (All Chassis* app groups) * `Production.Data.AeroTelemetry` (All Aero* app groups) **Use Topic-Based When:** * You have fewer than 100 unique streams. * You want simple debugging (each stream has its own topic). * Topic limits are not a concern. * Message time alignment across streams is not important, or will be handled at the consumer level (note: Kafka does not align messages between topics). #### Partition-Based Strategy (`StreamCreationStrategy = 1`) Multiple streams share a single Kafka topic but are distributed across different partitions. Requires explicit partition mappings in `StreamApiConfig`. **How Mapping Works:** * Bridge uses `StreamSelectionConfig` to determine stream names. * Stream API uses `PartitionMappings` to assign each stream to a partition. * All streams share the same topic: `{Domain}.Data.MainDataSource`. * **IMPORTANT:** Partition 0 is reserved by Stream API. !!! example "Example Scenario" **Bridge Configuration:** ```json { "StreamApiConfig": { "StreamCreationStrategy": 1, "Domain": "Production", "PartitionMappings": [ { "Stream": "EngineTelemetry", "Partition": 1 }, { "Stream": "ChassisTelemetry", "Partition": 2 }, { "Stream": "AeroTelemetry", "Partition": 3 }, { "Stream": "BrakeTelemetry", "Partition": 4 } ] }, "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "Engine*", "Stream": "EngineTelemetry" }, { "AppName": "Chassis*", "Stream": "ChassisTelemetry" }, { "AppName": "Aero*", "Stream": "AeroTelemetry" }, { "AppName": "Brake*", "Stream": "BrakeTelemetry" } ] } } ``` **Resulting Kafka Topic with Partitions:** Topic: `Production.Data.MainDataSource` * Partition 0: [Reserved by Stream API] * Partition 1: `EngineTelemetry` data * Partition 2: `ChassisTelemetry` data * Partition 3: `AeroTelemetry` data * Partition 4: `BrakeTelemetry` data !!! danger "Critical Rules for Partition-Based" 1. Stream names in `StreamSelectionConfig` **MUST** match `Stream` in `PartitionMappings`. 2. **Partition 0 is reserved** - all mappings must use partition 1 and above. 3. Each stream must have a unique partition number. 4. Kafka topic must be created with enough partitions (max partition number + 1). **Use Partition-Based When:** * You have hundreds or thousands of streams. * Kafka topic limits are a concern. * You need better resource utilization. * You need better resource utilization. * **You need time-aligned data across multiple streams** - Kafka guarantees message order only within a single partition, so partition-based allows all streams to share one topic with proper temporal alignment. !!! note "Time-Aligned Data Consideration" For both Topic-Based and Partition-Based strategies: If you need time-aligned data across multiple application groups, map all of them to a **single stream name** in `StreamSelectionConfig`. This ensures: * **Topic-Based (Strategy 2):** All data goes to one Kafka topic with aligned timestamps. * **Partition-Based (Strategy 1):** All data goes to one partition within the topic with aligned timestamps. **Example for time-aligned data:** ```json { "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "Engine*", "Stream": "VehicleTelemetry" }, { "AppName": "Chassis*", "Stream": "VehicleTelemetry" }, { "AppName": "Aero*", "Stream": "VehicleTelemetry" }, { "AppName": "Brake*", "Stream": "VehicleTelemetry" } ] } } ``` All application groups now write to the same stream, ensuring temporal alignment in Kafka. ### Stream Mapping Flow 1. **Data arrives** at Bridge with `AppName` (e.g., "EngineData") 2. **`StreamSelectionConfig`** maps `AppName` to **Stream** * *Example:* "EngineData" -> matches "Engine*" -> "EngineTelemetry" 3. **Stream API** receives "EngineTelemetry" stream 4. **`StreamCreationStrategy`** determines Kafka destination: * **Topic-Based (Strategy 2):** Writes to Kafka topic `Production.Data.EngineTelemetry`. * **Partition-Based (Strategy 1):** * Looks up "EngineTelemetry" in `PartitionMappings`. * Finds Partition 1. * Writes to `Production.Data.MainDataSource`, Partition 1. ## 2. StreamSelectionConfig Controls how data streams are mapped from application groups to Stream API streams, including wildcard pattern matching support. ### Properties | Property | Type | Default | Description | | :--- | :--- | :--- | :--- | | `Auto` | boolean | true | Automatically map app names to stream names without transformation | | `Mappings` | array | [] | Custom mappings from app names to streams with wildcard support (when `Auto=false`) | ### Mappings Structure Each mapping entry contains: | Field | Type | Description | | :--- | :--- | :--- | | `AppName` | string | Application group name or pattern (supports wildcards: `*`) | | `Stream` | string | Target stream name in Stream API | ### Wildcard Pattern Matching The `AppName` field supports wildcard patterns using the `*` character: * **Exact match:** `"EngineData"` - matches only "EngineData" * **Prefix match:** `"Engine*"` - matches "EngineData", "EngineTemp", "EnginePressure" * **Suffix match:** `"*Telemetry"` - matches "EngineTelemetry", "ChassisTelemetry" * **Contains:** `"*_Data_*"` - matches any app name containing "_Data_" **Pattern Evaluation Rules:** 1. Patterns are evaluated in order of definition. 2. First matching pattern wins. 3. Case-insensitive matching. 4. Results are cached for performance. ### Example Configurations #### Auto Mode (Simple) ```json { "StreamSelectionConfig": { "Auto": true } } ``` *In Auto mode, each app group streams to its own stream with the same name.* #### Manual Mapping (No Wildcards) ```json { "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "EngineData", "Stream": "engine-telemetry-stream" }, { "AppName": "ChassisData", "Stream": "chassis-telemetry-stream" } ] } } ``` #### Wildcard Mapping (Flexible) ```json { "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "Engine*", "Stream": "engine-telemetry-stream" }, { "AppName": "Chassis*", "Stream": "chassis-telemetry-stream" }, { "AppName": "*Telemetry", "Stream": "general-telemetry-stream" }, { "AppName": "Aero_*_Data", "Stream": "aerodynamics-stream" } ] } } ``` ### Best Practices 1. **Ordering Matters:** Place more specific patterns before general ones. 2. **Auto Mode:** Use when stream names match app group names exactly. 3. **Wildcards:** Use to reduce configuration complexity when dealing with many similar app groups. ## 3. BridgeConfig Controls core Bridge service behavior including processing units, data handling, and flow control strategies. ### Properties | Property | Type | Default | Description | | :--- | :--- | :--- | :--- | | `DataSource` | string | "Default" | Name identifier for the data source | | `UseStringIdentifier` | boolean | false | Use string-based identifiers instead of numeric IDs | | `AdsTimeoutInSeconds` | int | 720 | Timeout in seconds for detecting session stop when no data is received from ADS (ATLAS Data Server) | | `ProcessFlow` | enum | SequentialAll | Data processing flow strategy (see below for details) | | `FeedPort` | int? | null | Optional port override for the data feed (see [Feed Port and Metric Port Resolution](#feed-port-and-metric-port-resolution)) | | `MetricPort` | int? | null | Optional port override for the Prometheus metrics endpoint (see [Feed Port and Metric Port Resolution](#feed-port-and-metric-port-resolution)) | | `LiveConcurrencyFactor` | int | 4 | Number of parallel processing units for live telemetry | | `OffloadConcurrencyFactor` | int | 12 | Number of parallel processing units for offloaded (historical/replayed) telemetry | | `OffloadProcessing` | boolean | true | Whether offloaded data is processed at all; when `false`, offloaded data is discarded instead of being written out | | `OffloadStream` | string | "" (empty) | Optional stream name that offloaded data is routed to, instead of the default stream | | `StandalonePgvList` | string[] | [] (empty) | PGV app IDs, as decimal or `0x`-prefixed hex strings, that Bridge Service should accept as standalone and output data for (see below) | ### Feed Port and Metric Port Resolution The feed port (data ingress) and the Prometheus metrics port are each resolved from multiple sources, in priority order: 1. **Feed port:** the `-p` command-line argument or the `FEED_PORT` environment variable. 2. **Metric port:** the `METRIC_PORT` environment variable (there is no `-m` command-line argument). 3. `BridgeConfig.FeedPort` / `BridgeConfig.MetricPort` in `AppConfig.json` (or the equivalent `BridgeConfig__FeedPort` / `BridgeConfig__MetricPort` environment variables described in [Overriding Configuration with Environment Variables](#overriding-configuration-with-environment-variables)). 4. Built-in defaults: **9697** for the feed port, **10010** for the metric port. !!! note `FEED_PORT` and `METRIC_PORT` are read directly by the host process and take priority over any `BridgeConfig__FeedPort` / `BridgeConfig__MetricPort` value — if a stale `FEED_PORT` environment variable is set, it silently overrides a `FeedPort` you set in `AppConfig.json`. ### ProcessFlow Strategies The `ProcessFlow` property controls how the Bridge service handles data flow and backpressure: #### SequentialAll (Default) * **Value:** `"SequentialAll"` or `0` * **Behavior:** Processes all data sequentially without dropping any packets * **Use when:** Data integrity is critical and you need to ensure every packet is processed * **Impact:** May experience backpressure if downstream systems are slow #### DropOldest * **Value:** `"DropOldest"` or `1` * **Behavior:** Drops oldest unprocessed data when buffers are full, prioritizing recent data * **Use when:** Real-time data is more important than historical completeness * **Impact:** Better performance under load, but some data may be dropped during high traffic * **Note:** Applies per-session dropping strategy to maintain data freshness ### Example Configuration ```json { "BridgeConfig": { "DataSource": "RaceTrack01", "UseStringIdentifier": true, "LiveConcurrencyFactor": 16, "AdsTimeoutInSeconds": 600, "ProcessFlow": "SequentialAll" } } ``` ### Example with DropOldest Flow ```json { "BridgeConfig": { "DataSource": "LiveTelemetry", "UseStringIdentifier": true, "LiveConcurrencyFactor": 24, "AdsTimeoutInSeconds": 720, "ProcessFlow": "DropOldest" } } ``` ### Performance Tuning * **`LiveConcurrencyFactor`**: Controls the number of parallel processing units for live telemetry. Increase for higher throughput on multi-core systems. Higher values use more CPU. * **`OffloadConcurrencyFactor`**: Controls the number of parallel processing units for offloaded (historical/replayed) telemetry. Increase for higher throughput on multi-core systems. Higher values use more CPU. * **`OffloadProcessing`**: Set to `false` to discard offloaded data instead of processing it, if offload telemetry isn't needed downstream. * **`AdsTimeoutInSeconds`**: Controls how long the Bridge waits without receiving data before considering a session stopped. Default is 720 seconds (12 minutes). Reduce for faster session timeout detection, increase if you expect longer gaps in data transmission. * **`ProcessFlow`**: Choose `SequentialAll` when data completeness is critical, or `DropOldest` when real-time performance is more important than historical completeness. ### Standalone PGV List `StandalonePgvList` lists the PGV app IDs Bridge Service should accept as standalone. When data from the ECU requests an app ID that's in this list, Bridge Service loads that PGV and starts outputting data from that app. If the list is empty, or an app's ID isn't in it, Bridge Service does not output data from that app. Each entry is a string, parsed as decimal or (with a `0x`/`0X` prefix) hexadecimal, and must fit in the range 0-65535. Leading and trailing whitespace is trimmed. ```json title="AppConfig.json" linenums="1" { "BridgeConfig": { "StandalonePgvList": [ "100", "0x64", "0xFF" ] } } ``` ## 4. EssentialsConfig Configuration files (CFG) and Parameter Group Values (PGV) are essential for data decoding. !!! note "Windows: empty paths fall back to ATLAS 9" On Windows, if `CfgDirectoryPaths` or `PgvDirectoryPaths` is empty, Bridge Service uses the config and PGV folders set in ATLAS 9 for the Windows user running it. That is why an ADS-launched Bridge Service usually needs no `EssentialsConfig` section. !!! warning "Docker / Linux: both paths are required" There is no ATLAS 9 fallback outside Windows. If `CfgDirectoryPaths` or `PgvDirectoryPaths` is empty, Bridge Service stops at startup with *"DirectoryPaths needs to be specified in EssentialsConfig"* — even when `RemoteConfigIndexerUrl` is set. ### Properties | Property | Type | Description | | :--- | :--- | :--- | | `CfgDirectoryPaths` | string[] | Array of directory paths containing CFG configuration files | | `PgvDirectoryPaths` | string[] | Array of directory paths containing PGV parameter files | | `RemoteConfigIndexerUrl`| string? | Optional base URL of the Configs service of MA.DataPlatforms.Indexing (see [Using Remote Config Indexer](#using-remote-config-indexer)). Leave it out, or empty, to search the local directories only. | ### Example Configuration ```json { "EssentialsConfig": { "CfgDirectoryPaths": [ "C:\\Data\\Logging\\AutoConfig", "C:\\Data\\Logging\\ManualConfig" ], "PgvDirectoryPaths": [ "C:\\Data\\Parameters\\PGVs", "C:\\SharedConfigs\\PGVs" ], "RemoteConfigIndexerUrl": null } } ``` ### Using Remote Config Indexer Use a remote config indexer when many CFG and PGV files are stored centrally and you want Bridge Service to look up where a file is instead of searching its own folders. `RemoteConfigIndexerUrl` must point at the **Configs service** of MA.DataPlatforms.Indexing, which listens on port **5050** by default. It is not the Files service on port 5000. Give the base URL only, with no path, for example `http://config-server.company.com:5050`. Bridge Service calls these endpoints on it: | Endpoint | Used to | |---|---| | `GET /api/processing/files/cfg?configId=…` | Find the CFG files for a config ID | | `GET /api/processing/files/pgv?appId=…&appVersion=…` | Find the PGV files for an app ID and version | | `GET /api/processing/info?configId=…` | Read the processing info for a config ID | | `GET /api/heartbeat` | Check the service is available | When the indexer doesn't return a file, Bridge Service falls back to searching `CfgDirectoryPaths` and `PgvDirectoryPaths`. !!! warning "The indexer returns file paths, not file contents" Bridge Service opens the files itself at the paths the indexer returns, so those paths must be readable from where Bridge Service runs: - **Windows**: run the indexer on the same machine, or index network share paths that both machines can reach. - **Docker**: the paths the indexer reports must exist inside the Bridge Service container. Mount the same storage at the same path in both. This works the same whether Bridge Service is launched by ADS, run standalone on Windows, or run in Docker. ```json title="AppConfig.json" linenums="1" { "EssentialsConfig": { "CfgDirectoryPaths": ["C:\\Data\\Configs\\CFG"], "PgvDirectoryPaths": ["C:\\Data\\Configs\\PGV"], "RemoteConfigIndexerUrl": "http://config-server.company.com:5050" } } ``` ## 5. RdaConfig Reduced Data Access (RDA) configuration for reading telemetry data files. ### Properties | Property | Type | Description | | :--- | :--- | :--- | | `PulFilesSearchPaths` | string[] | Array of directory paths to search for PUL files | ### Example Configuration ```json { "RdaConfig": { "PulFilesSearchPaths": [ "C:\\RDA\\SessionData\\2026", "C:\\RDA\\Archive", "\\\\FileServer\\RDA\\Shared" ] } } ``` ## 6. DiskBackedBufferConfig Controls the per-session buffer that holds messages pending validation retry. Once a session's buffer exceeds `MaxQueueLength`, further items spill to disk instead of growing memory unbounded, and are read back from disk as the queue drains. ### Properties | Property | Type | Default | Description | | :--- | :--- | :--- | :--- | | `MaxQueueLength` | int | 1000 | Number of items kept in memory per session before overflow spills to disk | | `TempFolderPath` | string | `%TEMP%\EngineeringConversion\DiskBackedBuffer` | Directory used to store the disk-backed overflow files | ### Example Configuration ```json title="AppConfig.json" linenums="1" { "DiskBackedBufferConfig": { "MaxQueueLength": 1000 } } ``` ## 7. Serilog Configuration Bridge Service logging is configured with the `Serilog` section of the configuration file, in the standard [Serilog JSON settings](https://github.com/serilog/serilog-settings-configuration) format. Bridge Service includes two sinks: **Console** and **File**. ### Where logs go | Your configuration file | Where logs go | |---|---| | **Has a `Serilog` section** | Only where that section says. `-l` and `LOG_FILE_PATH` are **ignored**. | | **Has no `Serilog` section** | The console, plus a log file that rolls over daily, at the first of: the path given by `-l` or `LOG_FILE_PATH`; on Windows, `bridge-svc-log.txt` in the ADS log file folder; otherwise `Logs/bridge-svc-log.txt` relative to the folder Bridge Service was started from. | !!! warning "No log file by default" From Bridge Service 2.1.1, the default configuration file contains a `Serilog` section that writes to the **console only**. No log file is written in any deployment — including when ADS launches Bridge Service, because the log path ADS passes with `-l` is ignored while a `Serilog` section is present. To get a log file, either: - add a File sink to the `Serilog` section (see [Write logs to a file](#write-logs-to-a-file)), or - delete the `Serilog` section, so the fallback locations in the table above apply. When launched by ADS, this writes `BridgeLog.txt` to the ADS log file folder. ### Write logs to a file Add a File sink next to the Console sink. Use an absolute path: a relative path is resolved from the folder Bridge Service was started from, which for an ADS-launched Bridge Service is its install folder and may not be writable. ```json title="AppConfig.json" linenums="1" hl_lines="8-15" { "Serilog": { "MinimumLevel": { "Default": "Information" }, "WriteTo": [ { "Name": "Console" }, { "Name": "File", "Args": { "path": "C:\\Logs\\Bridge\\bridge-svc-log.txt", "rollingInterval": "Day", "outputTemplate": "[{Timestamp:HH:mm:ss.fff} {Level:u3}]: {Message:lj}{NewLine}{Exception}" } } ] } } ``` With `"rollingInterval": "Day"`, the date is added to the file name, for example `bridge-svc-log20261014.txt`. The File sink also accepts the standard `retainedFileCountLimit` and `fileSizeLimitBytes` arguments to limit disk use. !!! note "Docker" Console output appears in `docker logs`. If you add a File sink, write to a path on a mounted volume, for example `"path": "/logs/bridge-svc-log.txt"` with `-v /srv/bridge-logs:/logs`. Otherwise the log file is lost when the container is removed. ### Log Levels Set `MinimumLevel.Default` to one of: * **Verbose:** Very detailed, trace-level information. * **Debug:** Debugging information. * **Information:** General informational messages (recommended default). * **Warning:** Warnings that don't stop operation. * **Error:** Errors and exceptions. * **Fatal:** Critical failures causing shutdown. You can also change the level without editing the file, with an [environment variable](#overriding-configuration-with-environment-variables): `Serilog__MinimumLevel__Default=Debug`. ## Complete Configuration Example ```json { "StreamApiConfig": { "StreamCreationStrategy": 2, "BrokerUrl": "kafka-prod.company.com:9094", "PartitionMappings": [], "IntegrateSessionManagement": true, "IntegrateDataFormatManagement": true, "UseRemoteKeyGenerator": false, "RemoteKeyGeneratorServiceAddress": "", "InitialisationTimeoutSeconds": 1, "Domain": "motorsport" }, "StreamSelectionConfig": { "Auto": false, "Mappings": [ { "AppName": "Engine*", "Stream": "engine-telemetry" }, { "AppName": "Chassis*", "Stream": "chassis-telemetry" }, { "AppName": "Aero*", "Stream": "aerodynamics-telemetry" }, { "AppName": "*Telemetry", "Stream": "general-telemetry" } ] }, "BridgeConfig": { "DataSource": "TestTrack_Simulator", "UseStringIdentifier": true, "LiveConcurrencyFactor": 16, "OffloadConcurrencyFactor": 8, "AdsTimeoutInSeconds": 600, "ProcessFlow": "SequentialAll" }, "EssentialsConfig": { "CfgDirectoryPaths": [ "C:\\ProgramData\\ATLAS\\Configs", "\\\\FileServer\\SharedConfigs\\CFG" ], "PgvDirectoryPaths": [ "C:\\ProgramData\\ATLAS\\Parameters", "\\\\FileServer\\SharedConfigs\\PGV" ], "RemoteConfigIndexerUrl": null }, "RdaConfig": { "PulFilesSearchPaths": [ "C:\\RDA\\Sessions\\2026", "\\\\FileServer\\RDA\\Archive" ] }, "DiskBackedBufferConfig": { "MaxQueueLength": 1000 }, "Serilog": { "MinimumLevel": { "Default": "Information", "Override": { "Microsoft": "Warning", "System": "Warning" } }, "WriteTo": [ { "Name": "Console", "Args": { "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {SourceContext}: {Message:lj}{NewLine}{Exception}" } }, { "Name": "File", "Args": { "path": "C:\\Logs\\bridge-service-.txt", "rollingInterval": "Day", "retainedFileCountLimit": 30, "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss}] [{Level:u3}] {Message:lj}{NewLine}{Exception}" } } ] } } ``` ## Troubleshooting ### Common Configuration Issues * **Issue: Stream API fails to initialize** * Check `BrokerUrl` is correct and accessible. * Verify Kafka broker is running. * Increase `InitialisationTimeoutSeconds`. * **Issue: Data not decoding correctly** * Verify CFG files exist in `CfgDirectoryPaths`. * Check PGV files are present in `PgvDirectoryPaths`. * **Issue: Stream mapping not working** * Verify `Auto` is set to `false` when using custom mappings. * Check wildcard patterns/order. * **Issue: Performance problems** * Increase `LiveConcurrencyFactor` / `OffloadConcurrencyFactor`. * Consider using `ProcessFlow: "DropOldest"` if real-time data is more important than completeness. * **Issue: Data is being dropped** * Check if `ProcessFlow` is set to `"DropOldest"`. This is the expected behavior when buffers are full. * If data completeness is critical, change to `"SequentialAll"`. * Alternatively, increase `LiveConcurrencyFactor` and `OffloadConcurrencyFactor` to improve throughput. * **Issue: Session stops prematurely or doesn't stop** * Adjust `AdsTimeoutInSeconds` based on your data streaming pattern. * Default (720 seconds / 12 minutes) works for most scenarios. * Reduce for faster timeout detection in live scenarios. * Increase if you have legitimate gaps in data transmission. * **Issue: Remote Config Indexer not working** * Verify `RemoteConfigIndexerUrl` is the base URL of the indexer's **Configs** service (port 5050 by default), not the Files service, and has no path. * Check network connectivity to the config server, for example by opening `/api/heartbeat`. * Ensure the configuration indexer service is running. * Check that the file paths the indexer returns are readable from where Bridge Service runs — in Docker, inside the container. * Add local `CfgDirectoryPaths` and `PgvDirectoryPaths` as backup. * **Issue: Bridge Service stops at startup with "DirectoryPaths needs to be specified in EssentialsConfig"** * Outside Windows (for example in Docker) there is no ATLAS 9 fallback: set both `CfgDirectoryPaths` and `PgvDirectoryPaths`, using paths inside the container. * On Windows, set them in `EssentialsConfig`, or set the config and PGV folders in ATLAS 9 for the Windows user running Bridge Service. * **Issue: No Bridge Service log file is written** * If the configuration file has a `Serilog` section, `-l` and `LOG_FILE_PATH` are ignored. Add a File sink to the `Serilog` section, or delete the section — see [Where logs go](#where-logs-go). * **Issue: Changes to `AppConfig.json` have no effect when ADS launches Bridge Service** * ADS uses `Documents\McLaren Electronic Systems\ATLAS 9\BridgeService\BridgeServiceConfig.json` instead. Edit that file — see [Choose your deployment](index.md#choose-your-deployment). * **Issue: Stream API fails to connect with a SASL/SSL handshake error** * Verify `Protocol` and `Mechanism` match what the broker is actually configured for. * For SASL, double-check `SaslUsername`/`SaslPassword`. * For SSL, verify certificate paths (`SslCaLocation`, `SslCertificateLocation`, `SslKeyLocation`) are correct and readable by the account running Bridge Service. * Remember `Security` only activates once `SaslUsername`, `SslCaLocation`, or `SslCertificateLocation` is set — an incomplete `Security` block may silently connect unsecured instead of failing. * **Issue: Environment variable override isn't applied** * Confirm the key uses `__` (double underscore) between each nested segment, e.g. `BridgeConfig__DataSource`. * Environment variables are only read at startup — restart the Bridge Service after changing one. * **Issue: Bridge Service fails to start after adding `StandalonePgvList`** * Each entry must be a plain decimal number or a `0x`/`0X`-prefixed hex number, and fit in the range 0-65535. * Remove or fix any entry that isn't one of those two formats. !!! warning "Resource Requirements" Increasing these values requires more system resources (CPU and memory). Setting values too high on smaller machines may lead to deadlock or threadpool starvation. **Recommended settings for an 8-core machine with 16GB RAM:** - `LiveConcurrencyFactor`: 12 - `OffloadConcurrencyFactor`: 24 --- # Overview Broadcast Service gives your live telemetry feed highly-available replication to multiple downstream consumers, so a downstream restart, network blip, or slow consumer never costs you data — every downstream tracks its own place in the stream and catches up independently once it reconnects. It sits between an upstream source and one or more downstream consumers. Towards its upstream source, it exposes the same ADS messages, so from the source's point of view it looks like an ordinary downstream. Towards its own downstreams, it acts as the upstream — replaying that same ADS messages — so each downstream can consume, fall behind, and recover without affecting the others. !!! note "Not the same as Bridge Service" Broadcast Service is a separate product from [the Bridge Service](../bridge_service/index.md), which decodes ATLAS quads from an ADS into Stream API packets. Broadcast Service does not decode anything — it replicates an ADS messages (which may originate from an ADS, or another Broadcast Service) to more places, with durable replay if a downstream falls behind. If you're looking for quad decoding, see the Bridge Service docs instead. ## Before you start At minimum, every deployment needs: - **An inbound feed port** — the port Broadcast Service listens on for the upstream ADS source (`FeedPort`). - **At least one downstream Bridge target** — Broadcast Service refuses to start if no targets are configured. - **A write-ahead log (WAL) directory** — where incoming records are durably persisted so targets can replay them (`Wal.Directory`). - **Enough disk for the WAL** — durability comes from disk. Plan capacity around your WAL segment size and how far behind your slowest target is allowed to fall before it's caught up again. ## At a glance | | | |---|---| | **Default feed port** | `9697` | | **Default metrics port** | `10010` | | **Default WAL directory** | `./wal` | | **Config file** | `AppConfig.json` (override the path with the `CONFIG_FILE_PATH` environment variable) | ## Two ways to run it The same binary runs either way — there's no separate "standalone" build versus an "integrated" one: - **`dotnet run`** — run the host project (or published binaries) directly. It reads `Configs/AppConfig.json` next to the executable by default. - **Docker** — the shipped `Dockerfile` exposes the feed port (9697) and metrics port (10010). Supply configuration via a mounted config file plus `CONFIG_FILE_PATH`, or via `BroadcastConfig__...` environment variables. See [Getting Started](getting-started.md) for step-by-step instructions for both. ## Next steps - [Getting Started](getting-started.md) — run your first instance and confirm it's forwarding data. - [Configuration Guide](configuration-guide.md) — task-by-task guidance for adding targets, tuning the WAL, and overriding config with environment variables. - [Configuration Reference](configuration-reference.md) — the full field-by-field reference for `AppConfig.json`. - [Troubleshooting](troubleshooting.md) — diagnosing startup failures, unhealthy targets, and disk growth. --- # Getting Started This walks through running a first Broadcast Service instance with the smallest config that starts successfully, and confirming it's forwarding data. Pick the tab for how you want to run it — each one is self-contained. === "dotnet run" **Prerequisites** - .NET 8.0 runtime available on the machine. - At least one reachable downstream Bridge endpoint. - An upstream source such as ADS or another Broadcast Service to point at the feed port once Broadcast Service is running. **Minimal config** Save this next to the host executable as `Configs/AppConfig.json` (or point `CONFIG_FILE_PATH` at it): ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "FeedPort": 9697, "MetricPort": 10010, "Wal": { "Directory": "./wal" }, "Targets": { "BridgeConfigs": [ { "Name": "downstream-bridge", "Host": "downstream-host", "Port": 9697 } ] } } } ``` This configures the inbound feed port, the metrics port, the WAL directory, and a single Bridge target to forward to. `Targets.BridgeConfigs` needs at least one entry — Broadcast Service won't start with zero targets configured. **Start it** ```bash dotnet run MA.DataPlatforms.Bridge.Broadcast.Host.exe ``` **Confirm it's working** Open `http://localhost:10010/metrics` (using the `MetricPort` from your config) and look for the `broadcast_target_up` gauge with a `target` label matching your target's name — it reads `1` once that target has connected successfully. Point your upstream source at the feed port (`9697` by default) and confirm `broadcast_ingest_total` is increasing as records arrive. **First-run problems** - **Startup fails with `Invalid broadcast configuration: ...`** — configuration validation failed. The message lists every problem found, joined together. Common causes: `FeedPort` outside 1-65535, no targets configured, two targets with the same name, or a target missing `Host`. - **Feed port already in use** — if another process (including a second Broadcast Service instance) is already bound to `FeedPort`, startup fails. Pick a different `FeedPort` or stop the conflicting process. - **WAL directory not writable** — if the process account can't create or write files under `Wal.Directory`, the service can't persist incoming records. Check the path exists and is writable by whichever account is running the process. === "Docker" **Prerequisites** - Docker (or a compatible container runtime). - At least one reachable downstream Bridge endpoint. - An upstream source such as ADS or another Broadcast Service to point at the feed port once Broadcast Service is running. **Minimal config** No config file is baked into the image. Mount one in, or supply the same values as environment variables. As a file, mounted at (for example) `/config/AppConfig.json`: ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "FeedPort": 9697, "MetricPort": 10010, "Wal": { "Directory": "/data/wal" }, "Targets": { "BridgeConfigs": [ { "Name": "downstream-bridge", "Host": "downstream-host", "Port": 9697 } ] } } } ``` !!! warning "Mount a volume for the WAL directory" Mount a volume at whatever path you set `Wal.Directory` to — `/data/wal` in the example above. If that path isn't backed by a mounted volume, the WAL — and everything not yet replayed to every target — is lost when the container is recreated. **Start it** ```bash docker run \ -p 9697:9697 -p 10010:10010 \ -e CONFIG_FILE_PATH=/config/AppConfig.json \ -v /host/path/config:/config \ -v /host/path/wal:/data/wal \ atlasplatformdocker/broadcast-service-host:latest ``` **Confirm it's working** Open `http://localhost:10010/metrics` on the host (mapped from the container's `MetricPort`) and look for the `broadcast_target_up` gauge with a `target` label matching your target's name — it reads `1` once that target has connected successfully. **First-run problems** - **Startup fails with `Invalid broadcast configuration: ...`** — same validation as above: check `FeedPort` range, that at least one target is configured, target names are unique, and required target fields (`Host`/`Port`) are set. - **Feed port already in use** — make sure the host port you mapped to 9697 isn't already taken by another container or process. - **WAL directory not writable / disappears on restart** — if `/data/wal` (or wherever `Wal.Directory` points) isn't backed by a mounted volume, or the mount isn't writable by the container's process, the service can't persist records, or loses everything on container recreation. --- # Configuration Guide Task-by-task guidance for the changes you're most likely to make to `AppConfig.json`. For the full field-by-field list, defaults, and validation rules, see the [Configuration Reference](configuration-reference.md). ## I want to add a downstream Bridge target **Situation**: you want Broadcast Service to forward the feed on to another Bridge endpoint — replicating bridge-to-bridge. Add an entry to `Targets.BridgeConfigs`: ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "Targets": { "BridgeConfigs": [ { "Name": "site-b-bridge", "Host": "site-b-bridge.internal", "Port": 9697 } ] } } } ``` `Host` is required — the target won't validate without one. `Port` defaults to `9697` if omitted. Broadcast Service connects to this target at `http://{Host}:{Port}`. **Result**: on connect, this target performs a licence handshake with the downstream and replays any cached session-control history, then forwards live records over gRPC. If the target's queue fills (it's capped at `QueueCapacity`, default `100000`), Broadcast Service blocks and waits rather than dropping records for a Bridge target — the target must catch up on its own. **When to use**: any time the downstream is itself a Bridge consumer. ## I want to run more than one target **Situation**: fan-out replication — for example, two Bridge targets at different sites. Add multiple entries to `BridgeConfigs`. Every target needs a unique `Name` (case-insensitive); startup fails otherwise. `Host`/`Port` aren't part of the effective name, so give each target a distinct `Name` explicitly even if they point at different hosts. ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "Targets": { "BridgeConfigs": [ { "Name": "site-a-bridge", "Host": "site-a.internal", "Port": 9697 }, { "Name": "site-b-bridge", "Host": "site-b.internal", "Port": 9697 } ] } } } ``` **When to use**: whenever you need the same feed replicated to more than one place — each target maintains its own replay cursor, so a slow or disconnected target never affects the others. ## I want to control whether a target receives offload (bulk historical) data **Situation**: some downstreams should only see live telemetry, not the bulk historical data a source sends during an offload. Set `OffloadProcessing` on the target: ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "Targets": { "BridgeConfigs": [ { "Name": "live-only-bridge", "Host": "site-c.internal", "Port": 9697, "OffloadProcessing": false } ] } } } ``` `OffloadProcessing` defaults to `false` on every target. With it `false`, offload records are not forwarded to that target — by design, not as a failure — though the target's replay cursor still advances past them. Session-control records are always forwarded regardless of this setting. Set it to `true` if that target should also receive the bulk historical data. **When to use**: set to `true` for targets that need the full historical record (for example, an archive), and leave `false` (the default) for targets that only care about live data. ## I want to tune WAL segment size or compaction behavior **Situation**: you're seeing more disk usage on the WAL directory than expected, or want to change how often compaction runs. ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "Wal": { "Directory": "./wal", "CompactionEnabled": true, "CompactionIntervalMs": 5000, "SegmentRollBytes": 10485760 } } } ``` - `SegmentRollBytes` (default `10485760`, i.e. 10 MiB, minimum `1024`) controls how large each WAL segment file grows before a new one is started. - `CompactionEnabled` (default `true`) turns segment trimming on or off. If disabled, no segments are ever removed. - `CompactionIntervalMs` (default `5000`, minimum `100`, only checked when compaction is enabled) controls how often compaction runs. Compaction only removes a sealed segment once **every** target's replay cursor has passed it — see [Disk usage growing unexpectedly](troubleshooting.md#disk-usage-growing-unexpectedly-on-the-wal-directory) for what happens when one target lags behind. The full field list, including `FsyncOnFlush` and `OffsetFlushIntervalMs`, is in the [Configuration Reference](configuration-reference.md). **When to use**: increase `SegmentRollBytes` to reduce the number of segment files under high-throughput feeds; tighten `CompactionIntervalMs` if you need disk space reclaimed sooner after targets catch up. ## I want to override config with environment variables instead of editing AppConfig.json **Situation**: you don't want a value (or don't want to redeploy a file) to change `AppConfig.json` directly — for example, per-environment overrides in a deployment pipeline. Broadcast Service loads `AppConfig.json` first, then applies matching environment variables on top — environment variables win. Address nested keys with a double underscore (`__`) between each segment: ```bash BroadcastConfig__FeedPort=9697 BroadcastConfig__Wal__Directory=/data/wal ``` This is standard .NET configuration layering (`AddJsonFile` then `AddEnvironmentVariables`), so it applies to any nested object in the config — not just the examples above. !!! tip Environment variables are read once at startup. Restart the service after changing one. **When to use**: containerized or pipeline-driven deployments where you want to keep `AppConfig.json` generic and inject environment-specific values (like the WAL path or feed port) at deploy time. --- # Configuration Reference Field-by-field reference for the `BroadcastConfig` section of `AppConfig.json`. For task-based how-tos (adding a target, tuning WAL compaction, and so on), see the [Configuration Guide](configuration-guide.md). ## Structure ```json title="AppConfig.json" linenums="1" { "BroadcastConfig": { "FeedPort": 9697, "MetricPort": 10010, "Wal": { ... }, "Targets": { "BridgeConfigs": [ ... ] } } } ``` | Property | Type | Default | Notes | |---|---|---|---| | `FeedPort` | int | `9697` | Inbound `RemoteDataFeed` gRPC server port. Validated 1-65535. | | `MetricPort` | int | `10010` | Prometheus metrics port. Not validated; a bind failure is caught and logged as a warning at startup — the service continues without metrics. | | `Wal` | object | — | Write-ahead log settings. See [Wal](#wal). | | `Targets` | object | — | Downstream Bridge targets. See [Targets](#targets). | ## Wal `BroadcastConfig.Wal`. | Property | Type | Default | Notes | |---|---|---|---| | `Directory` | string | `"./wal"` | Data WAL root. The control WAL is written to `{Directory}/control`; per-target cursor offsets are persisted to `{Directory}/offsets.json`. | | `FsyncOnFlush` | bool | `true` | Applies to both the data and control WAL. | | `CompactionEnabled` | bool | `true` | If `false`, no WAL segment trimming occurs. | | `CompactionIntervalMs` | int | `5000` | Validated minimum `100`, only enforced when `CompactionEnabled` is `true`. | | `SegmentRollBytes` | long | `10485760` (10 MiB) | Validated minimum `1024`. | | `OffsetFlushIntervalMs` | int | `1000` | Validated minimum `50`. Controls how often per-target cursor offsets are flushed to disk. | !!! warning `FsyncOnFlush` is mission-critical for durability. Do not disable it in production — doing so trades data-loss protection for throughput. ## Targets `BroadcastConfig.Targets`. | Property | Type | Default | Notes | |---|---|---|---| | `BridgeConfigs` | array of bridge target objects | `[]` | Downstream targets reached over `RemoteDataFeed` gRPC. See [Bridge targets](#bridge-targets). | At least one entry must be configured in `BridgeConfigs`. ### Shared target fields Every entry in `BridgeConfigs` accepts these fields in addition to its own type-specific fields below: | Property | Type | Default | Notes | |---|---|---|---| | `Name` | string | `""` | Display/tracking name for the target. | | `QueueCapacity` | int | `100000` | Validated minimum `1`. | | `OffloadProcessing` | bool | `false` | When `false`, offload (bulk historical) telemetry is dropped for this target and its cursor still advances; session control records are always forwarded regardless of this setting. | | `Retry` | object | — | Reconnect/backoff behavior. See [Retry](#retry). | The effective name used for uniqueness checks (see [Validation](#validation)) is `Name`. ### Bridge targets Additional fields for entries in `Targets.BridgeConfigs`: | Property | Type | Default | Notes | |---|---|---|---| | `Host` | string | `""` | Required — an empty host fails validation. | | `Port` | int | `9697` | Validated 1-65535. | The target address is built as `http://{Host}:{Port}`. ```json title="AppConfig.json" linenums="1" { "Name": "DownstreamBridge01", "Host": "bridge01.internal", "Port": 9697, "QueueCapacity": 500000, "OffloadProcessing": false, "Retry": { "InitialDelayMs": 500, "MaxDelayMs": 30000, "JitterMs": 250 } } ``` ## Retry Each target's `Retry` object (`RetryConfig`). | Property | Type | Default | |---|---|---| | `InitialDelayMs` | int | `500` | | `MaxDelayMs` | int | `30000` | | `JitterMs` | int | `250` | Reconnect backoff is calculated as: `delay = min(InitialDelayMs * 2^attempt, MaxDelayMs) + random(0, JitterMs)` The attempt exponent is capped at 30. Retries are unbounded — a target can remain down indefinitely without affecting the other targets. `InitialDelayMs` and `JitterMs` must each be `>= 0`, and `MaxDelayMs` must be `>= InitialDelayMs`. ## Validation Broadcast Service validates `BroadcastConfig` at startup. If any of the following rules are violated, the service fails to start with an exception of the form `Invalid broadcast configuration: ; ; ...` (all violated rules are joined by `; ` into a single message): - `FeedPort` must be between 1 and 65535. - At least one target must be configured in `Targets.BridgeConfigs`. - If `Wal.CompactionEnabled` is `true`, `Wal.CompactionIntervalMs` must be `>= 100`. - `Wal.OffsetFlushIntervalMs` must be `>= 50` (checked regardless of `CompactionEnabled`). - `Wal.SegmentRollBytes` must be `>= 1024`. - Every target must have a non-empty `Name`, and names must be unique case-insensitively across all targets. - Every target's `QueueCapacity` must be `>= 1`. - Every target's `Host` must be non-empty, and `Port` must be between 1 and 65535. ## Environment variable overrides Broadcast Service loads `AppConfig.json` first, then applies environment variables on top — environment variables win over values in the JSON file. Address nested keys with a double underscore (`__`) between each segment, for example: ``` BroadcastConfig__FeedPort=9697 BroadcastConfig__Wal__Directory=/data/wal ``` --- # Troubleshooting ## Service fails to start with "Invalid broadcast configuration: ..." **Symptom**: the process exits at startup with an error beginning `Invalid broadcast configuration:`, followed by one or more problems joined together. **Cause**: configuration validation runs before anything else starts, and rejects the whole config if it finds a problem — an out-of-range `FeedPort`, zero targets configured, two targets with the same effective name, a target missing `Host`/`Port`, or a WAL setting below its minimum. See [Validation](configuration-reference.md#validation) in the Configuration Reference for the complete, exact list of rules and valid ranges. **Fix**: read every problem listed in the error message (they're all reported together, joined by `; `) and correct each one in `AppConfig.json` or your environment variable overrides against the rules linked above. ## Metrics endpoint not reachable **Symptom**: `http://localhost:/metrics` doesn't respond, but the service otherwise appears to be running and forwarding data. **Cause**: `MetricPort` isn't validated at startup, and a failure to bind the metrics endpoint (for example, the port is already in use) is caught and logged as a warning rather than crashing the process. The service keeps running — and keeps replicating — without a working metrics endpoint. **Fix**: check the service's logs for the bind warning to see why the metrics port failed to bind (commonly, something else is already listening on it), free up the port or change `MetricPort`, and restart. This is not a sign the replication path itself is broken — check `broadcast_target_up` and `broadcast_ingest_total` are what you actually care about once metrics are reachable again. ## A target never goes healthy / `broadcast_target_up` stays 0 **Symptom**: the `broadcast_target_up` gauge for a given `target` label stays at `0` and never flips to `1`. **Cause**: the target's connection attempts are failing (unreachable `Host`/`Port`). Retries for a target are unbounded — a target can be down indefinitely without affecting any other target — and back off exponentially: each retry waits `min(InitialDelayMs * 2^attempt, MaxDelayMs)` plus a small random jitter (up to `JitterMs`), so retries get progressively less frequent up to `MaxDelayMs` (default `30000` ms). **Fix**: check the service's logs for connection/reconnection errors for that target's name, and confirm the target's `Host`/`Port` is actually reachable from where Broadcast Service is running. Once the target does connect, `broadcast_target_up` flips to `1` and `broadcast_reconnect_total` records how many reconnect attempts it took. ## Offload (bulk historical) data isn't reaching a target **Symptom**: live telemetry arrives at a target fine, but the bulk historical data sent during an offload never shows up there. **Cause**: this is expected, not a bug, if that target has `OffloadProcessing: false` (the default). With `OffloadProcessing` set to `false`, offload records are deliberately not forwarded to that target — its replay cursor still advances past them as if they'd been sent, so it won't try to re-fetch them later. Session-control records are unaffected and are always forwarded. **Fix**: set `OffloadProcessing: true` on that target if it should also receive offload data. See [I want to control whether a target receives offload data](configuration-guide.md#i-want-to-control-whether-a-target-receives-offload-bulk-historical-data) in the Configuration Guide. ## Disk usage growing unexpectedly on the WAL directory **Symptom**: the WAL directory keeps growing and doesn't shrink, even with compaction enabled. **Cause**: compaction only removes a sealed WAL segment once **every** configured target's replay cursor has advanced past it. A single stalled, disconnected, or very slow target holds compaction back for every other target too — the WAL has to keep everything that target might still need to replay. **Fix**: check `broadcast_target_up` and `broadcast_cursor_gsn` per target to find which one is lagging or down, and fix or remove that target (see [A target never goes healthy](#a-target-never-goes-healthy-broadcast_target_up-stays-0) above). If a target is permanently gone and won't come back, removing it from `Targets.BridgeConfigs` lets compaction proceed for the ones that remain. Confirm `Wal.CompactionEnabled` is `true` and `Wal.CompactionIntervalMs` is set to a reasonable value — if `CompactionEnabled` is `false`, no segments are ever removed regardless of target state. --- # Overview ## Overview The MA DataPlatforms Streaming Support Library is a .NET library designed to facilitate real-time and historical data streaming for telemetry and data analysis applications. The library provides a modular architecture for session management, data format handling, packet reading/writing, data buffering, merging, and interpolation. ## Key Features - **Session Management**: Create, track, and manage live and historical sessions with metadata and associations - **Data Format Management**: Define and manage parameter and event data formats with automatic ID generation - **Packet Reading/Writing**: Stream packets to and from Kafka brokers with support for live and historical data - **Essential Packet Reading**: Efficiently read configuration and metadata packets without full data processing - **Buffering & Merging**: Buffer incoming data packets, merge them, and extract time-aligned samples - **Interpolation**: Subscribe to parameters at custom frequencies with automatic super-sampling or sub-sampling - **Event-Driven Architecture**: Rich event system for tracking session lifecycle, data availability, and processing status - **Kafka Integration**: Built-in support for Kafka-based streaming with configurable retry policies, [secured (SASL/SSL) brokers, and broker tuning](kafka-configuration.md) - **Prometheus Metrics**: Built-in monitoring and metrics support ## Architecture The library follows a modular, pipeline-based architecture where data flows through independent but interconnected modules: ![Architecture Diagram](images/architecture-diagram.svg) ### Module Dependencies **Independent Core Modules:** - **Session Manager**: No dependencies on other modules - **Data Format Manager**: No dependencies on other modules **Dependent Modules:** - **Writer Module**: Uses Session Manager and Data Format Manager APIs - **Reader Module**: Can use Session Manager and Data Format Manager for context - **Essentials Module**: Lightweight reader for essential/configuration packets only - **Sample Reader (Buffering)**: Pipelines with Reader Module - **Data Reader (Interpolation)**: Pipelines with Sample Reader Module ## API Response Format All APIs in the Support Library return responses in a **consistent, standardized format** using the `ApiResult` pattern. This ensures predictable error handling and makes the library easier to use. ### ApiResult Pattern Every API call returns an `ApiResult` object with the following structure: ```csharp public class ApiResult { public bool Success { get; } // Indicates if operation succeeded public T? Data { get; } // Contains the result data if successful public string Message { get; } // Contains success/error message details } ``` ### Best Practices for Handling API Responses #### 1. Always Check Success Status ```csharp var result = sessionService.CreateNewSession(sessionDto); if (result.Success && result.Data != null) { // Use result.Data safely var sessionInfo = result.Data; Console.WriteLine($"Session created: {sessionInfo.SessionKey}"); } else { // Handle error Console.WriteLine($"Failed to create session: {result.Message}"); } ``` #### 2. Null-Safe Data Access Even when `Success` is `true`, always check that `Data` is not null before accessing it: ```csharp var formatResult = dataFormatService.GetParameterDataFormatId("source", parameters); if (formatResult.Success && formatResult.Data != null) { var formatId = formatResult.Data.DataFormatId; // Use formatId safely } ``` #### 3. Centralized Error Handling For applications with multiple API calls, consider creating a helper method: ```csharp public static bool TryGetData(ApiResult result, out T data, Action? onError = null) { if (result.Success && result.Data != null) { data = result.Data; return true; } onError?.Invoke(result.Message ?? "Unknown error"); data = default!; return false; } // Usage if (TryGetData(sessionResult, out var sessionInfo, error => _logger.LogError(error))) { // Work with sessionInfo safely ProcessSession(sessionInfo); } ``` #### 4. Pattern Matching (C# 8.0+) Use pattern matching for cleaner code: ```csharp var result = dataFormatService.GetEventDataFormatId("source", "EventName"); var message = result switch { { Success: true, Data: not null } => $"Format ID: {result.Data.DataFormatId}", { Success: false } => $"Error: {result.Message}", _ => "Unexpected result state" }; ``` ### Common Response Types - **Service Creation**: `ApiResult`, `ApiResult`, `ApiResult`, `ApiResult`, `ApiResult` - **Session Operations**: `ApiResult`, `ApiResult>` - **Data Format Operations**: `ApiResult`, `ApiResult` - **Writer Operations**: `ApiResult` ### Note on Subscription Methods Some operations like `Subscribe()` and `Unsubscribe()` in the Buffering and Interpolation modules return `bool` directly rather than `ApiResult`, for simpler synchronous operations. This consistent pattern across all modules ensures: - **Predictable error handling** throughout your application - **Type safety** with compile-time checks - **Clear success/failure states** without exceptions for expected failures - **Detailed messages** for troubleshooting through the `Message` property ## Getting Started ### Installation ```bash # Install the NuGet package (when available) dotnet add package MA.DataPlatforms.Streaming.Support.Lib.Core ``` ### Prerequisites - .NET 8.0 or later - Kafka broker (configured and accessible) - Prometheus (optional, for metrics) ## Pipeline Initialization Best Practices When using the reading pipeline (Reader → Buffering → Interpolation), **always start services from the innermost module outward** to prevent data loss due to timing issues. ### ⚠️ Critical: Startup Order Initialization can be done in any order, but **starting services must follow this order** to ensure downstream modules are ready to receive data before upstream modules start producing it: ``` 1. Interpolation Module (Data Reader) ← START FIRST (innermost) 2. Buffering Module (Sample Reader) ← START SECOND 3. Reader Module (Packet Reader) ← START LAST (begins reading) ``` ### Why Startup Order Matters If you start in the wrong order (Reader → Buffering → Interpolation): - ❌ Reader starts receiving packets from Kafka - ❌ Buffering module is not started yet → **Packets are dropped** - ❌ Interpolation module is not subscribed → **Data is lost** If you start in the correct order (Interpolation → Buffering → Reader): - ✅ Interpolation module is started and subscribed - ✅ Buffering module is started and ready to receive - ✅ Reader starts receiving packets → **All data is captured** ### Complete Pipeline Initialization Example ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Abstractions; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts; // Assume supportLibApi is already initialized and started // Create all services (order doesn't matter for creation) var dataReaderApi = supportLibApi.GetDataReaderApi(); var dataReaderService = dataReaderApi.CreateService().Data; var sampleReaderApi = supportLibApi.GetSampleReaderApi(); var bufferingConfig = new BufferingConfiguration(["Param1", "Param2", "Param3"]); var sampleReaderService = sampleReaderApi.CreateService(bufferingConfig).Data; var readerApi = supportLibApi.GetReadingPacketApi(); var readerService = readerApi.CreateService("MyDataSource","Session_Key").Data; // Initialize all services (order doesn't matter for initialization) dataReaderService.Initialise(); sampleReaderService.Initialise(); readerService.Initialise(); // Subscribe to parameters at custom frequency dataReaderService.Subscribe( subscriptionKey: "MySubscription", parameterIdentifiers: new[] { "Param1", "Param2", "Param3" }, subscriptionFrequencyHz: 100, // 100 Hz subscription handler:new MyBatchResultHandler() ); // START in correct order: INNERMOST to OUTERMOST (CRITICAL!) // Step 1: Start Interpolation Module FIRST dataReaderService.Start(); // Step 2: Start Buffering Module SECOND sampleReaderService.Start(); // Step 3: Start Reader Module LAST readerService.Start(); ``` ### Quick Reference | Phase | Order | Modules | |-------|-------|---------| | **Creation** | Any order | Can create in any sequence | | **Initialization** | Any order | Can initialize in any sequence | | **Startup** | Inside → Out | Interpolation → Buffering → Reader ⚠️ | | **Shutdown** | Outside → In | Reader → Buffering → Interpolation | ### Common Mistake Example ```csharp // ❌ WRONG - Will lose data! var reader = readerApi.CreateService(config).Data; reader.Initialise(); reader.Start(); // Started reading! var buffering = sampleReaderApi.CreateService(config).Data; buffering.Initialise(); buffering.Start(); // Too late - already missing packets! var interpolation = dataReaderApi.CreateService().Data; interpolation.Initialise(); interpolation.Start(); // Too late - already lost data! ``` ```csharp // ✅ CORRECT - No data loss // Create and initialize all (any order) var interpolation = dataReaderApi.CreateService().Data; var buffering = sampleReaderApi.CreateService(config).Data; var reader = readerApi.CreateService(config).Data; interpolation.Initialise(); buffering.Initialise(); reader.Initialise(); // Start in correct order: inside-out interpolation.Start(); interpolation.Subscribe(...); // Ready to receive buffering.Start(); buffering.Subscribe(...); // Ready to receive reader.Start(); // Now start reading - pipeline is ready! ``` ### Basic Usage #### 1. Initialize the Support Library ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Abstractions; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts; using MA.DataPlatforms.Streaming.Support.Lib.Core.Shared; using MA.Streaming.Abstraction; using MA.Streaming.Core.Configs; var logger = new ConsoleLogger(); // Configure Kafka connection using Stream API Configuration var streamApiConfig = new StreamingApiConfiguration( StreamCreationStrategy.PartitionBased, "localhost:9094", []); // To connect to a secured (SASL/SSL) broker or tune the Kafka // producer/consumer, see the Kafka Configuration page. // Create the support library API with retry policy (the retry policy used for when the broker not available) var supportLibApi = new SupportLibApiFactory().Create( logger, streamApiConfig, new RetryPolicy(10, TimeSpan.FromSeconds(5), RetryMode.Finite)); // Subscribe to initialization event supportLibApi.Initialised += (sender, time) => { Console.WriteLine($"Library initialized at {time}"); }; // Initialize asynchronously var cts = new CancellationTokenSource(); await supportLibApi.InitialiseAsync(cts.Token); // Start the library if (supportLibApi.IsInitialised) { supportLibApi.Start(); } ``` #### 2. Create a Session ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule; // Get the session manager API var sessionManagerApi = supportLibApi.GetSessionManagerApi(); var sessionService = sessionManagerApi.CreateService().Data; sessionService.Initialise(); sessionService.Start(); // Create a new session var sessionResult = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "MyDataSource", identifier: "TestSession001", type: "session", version: 1, utcOffset: TimeSpan.Zero, details: new[] { new SessionDetailDto("Location", "LabA"), new SessionDetailDto("OperatorId", "OP_123") })); if (sessionResult.Success && sessionResult.Data != null) { var sessionKey = sessionResult.Data.SessionKey; Console.WriteLine($"Created session: {sessionKey}"); } ``` #### 3. Define Data Formats ```csharp // Get the data format manager API var dataFormatApi = supportLibApi.GetDataFormatManagerApi(); var dataFormatService = dataFormatApi.CreateService().Data; dataFormatService.Initialise(); dataFormatService.Start(); // Define parameter format var paramResult = dataFormatService.GetParameterDataFormatId( "MyDataSource", new[] { "Param1", "Param2", "Param3" }); if (paramResult.Success && paramResult.Data != null) { var formatId = paramResult.Data.DataFormatId; Console.WriteLine($"Parameter format ID: {formatId}"); } // Define event format var eventResult = dataFormatService.GetEventDataFormatId( "MyDataSource", "LapCompleted"); ``` #### 4. Write Data ```csharp // Get the writer API var writerApi = supportLibApi.GetWritingPacketApi(); var writerService = writerApi.CreateService().Data; writerService.Initialise(); writerService.Start(); // Write packet data writerService.WriteData( dataSource: "MyDataSource", stream: "MainStream", sessionKey: "Session_Key", packet: new Packet()); ``` #### 5. Read Data !!! note You will need to spin / wait on the main thread as the Reader Service reads packets or the application will exit. For example: while(true) { await Task.Delay(1000); } See [Reader Module](reader-module.md) for a more complete example including handling session start / completion events. ```csharp // Get the reader API var readerApi = supportLibApi.GetReadingPacketApi(); var readerService = readerApi.CreateService("MyDataSource", sessionKey).Data; // Create a handler for received packets public class MyPacketHandler : IHandler { public void Handle(IReceivedPacketDto packet) { Console.WriteLine($"Received packet: {packet.Packet.Type}"); } } readerService.AddHandler(new MyPacketHandler()); readerService.Initialise(); readerService.Start(); ``` #### 6. Buffer and Extract Samples ```csharp // Get the sample reader API var sampleReaderApi = supportLibApi.GetSampleReaderApi(); // Configure buffering var bufferingConfig = new BufferingConfiguration( subscribedParameters: new[] { "Param1", "Param2" }, bufferingWindowLength: 3000, // 3 seconds slidingWindowPercentage: 5); var sampleReaderService = sampleReaderApi.CreateService(bufferingConfig).Data; // Create a timestamp handler public class MyTimestampHandler : IHandler { public void Handle(ITimestampData data) { foreach (var column in data.TimeColumns) { Console.WriteLine($"Parameter: {column.Identifier}, Values: {column.SampleValues.Count}"); } } } sampleReaderService.SetReaderService(readerService); sampleReaderService.AddHandler(new MyTimestampHandler()); sampleReaderService.Initialise(); sampleReaderService.Start(); ``` #### 7. Subscribe to Interpolated Data ```csharp // Get the data reader API (interpolation) var dataReaderApi = supportLibApi.GetDataReaderApi(); var dataReaderService = dataReaderApi.CreateService().Data; // Create a batch result handler public class MyBatchResultHandler : IBatchResultHandler { public void Handle(BatchResult result) { Console.WriteLine($"Received {result.Results.Count} interpolated samples"); } } // Set the sample reader as the data source dataReaderService.SetSampleReaderService(sampleReaderService); // Subscribe to parameters at 10 Hz dataReaderService.Subscribe( subscriptionKey: "MySubscription", parameterIdentifiers: new[] { "Param1", "Param2", "Param3" }, subscriptionFrequencyHz: 10.0, handler: new MyBatchResultHandler(), deliveryFrequencyHz: 5.0); // Deliver results every 5 Hz dataReaderService.Initialise(); dataReaderService.Start(); ``` ## Quick Links - [Session Manager Module](session-manager.md) - [Data Format Manager Module](data-format-manager.md) - [Writer Module](writer-module.md) - [Reader Module](reader-module.md) - [Essentials Module](essentials-module.md) - [Buffering Module](buffering-module.md) - [Interpolation Module](interpolation-module.md) - [API Reference](api-reference.md) ## Use Cases The library supports various data streaming scenarios: - **Live Session Monitoring**: Stream data from live sessions and process in real-time - **Historical Playback**: Read and analyze complete historical sessions - **On-the-Fly Subscription**: Dynamically subscribe to parameters during a live session - **Data Analysis**: Buffer, merge, and interpolate data for analysis applications - **Multi-Session Association**: Link related sessions using session GUIDs ## Support For issues or questions, please send email to atlas10@motionapplied.com --- # Session Manager ## Overview The Session Manager Module is one of the two independent core modules in the Support Library. It provides session lifecycle management, including session creation, metadata management, session associations, and real-time notifications for session state changes. ## Key Features - **Session Creation**: Create new sessions with custom metadata and identifiers - **Session Lifecycle**: Track live sessions from creation to completion - **Metadata Management**: Attach and update custom key-value details to sessions - **Session Associations**: Link related sessions (e.g., linking multiple data collection sessions) - **Real-time Events**: Subscribe to session lifecycle events - **Historical Sessions**: Mark sessions as complete and maintain session history - **Session Querying**: Retrieve all sessions or specific session information ## API Access ### Get the Session Manager API ```csharp var sessionManagerApi = supportLibApi.GetSessionManagerApi(); ``` ### Create a Service Instance ```csharp var serviceResult = sessionManagerApi.CreateService(); if (serviceResult.Success && serviceResult.Data != null) { var sessionService = serviceResult.Data; // Initialize and start the service sessionService.Initialise(); sessionService.Start(); } ``` ## Core Concepts ### Session Information Each session contains: - **SessionKey**: Unique identifier for the session (auto-generated) - **DataSource**: The data source name (e.g., car identifier, system name) - **Identifier**: User-defined session identifier - **Type**: Session type (e.g., "session", "virtual", "vtag") - **Version**: Session version number - **Historical**: Boolean indicating if session is complete - **UtcOffset**: Time zone offset for the session - **Details**: Dictionary of custom metadata (key-value pairs) - **AssociateSessionKeys**: List of associated session keys ### Session States Sessions can be in one of two states: 1. **Live Session**: Currently active, `Historical = false` 2. **Historical Session**: Completed session, `Historical = true` ## Methods ### CreateNewSession Creates a new session with specified metadata. ```csharp public ApiResult CreateNewSession(SessionCreationDto createSessionDto) ``` **Parameters:** ```csharp var sessionDto = new SessionCreationDto( dataSource: "DataSource_A", // Required: Data source identifier identifier: "Session_001", // Optional: Session identifier type: "Session", // Optional: Session type (Session, VirtualSession, VTS) version: 1, // Optional: Version number (default: 1) utcOffset: TimeSpan.FromHours(1), // Optional: UTC offset (default: local) details: new[] { // Optional: Custom metadata new SessionDetailDto("Location", "BuildingB"), new SessionDetailDto("Equipment", "Sensor_v2"), new SessionDetailDto("Operator", "User_42") }, associatedSessionKeys: new[] { // Optional: Related session GUIDs "a1b2c3d4-e5f6-7890-abcd-ef1234567890" } ); var result = sessionService.CreateNewSession(sessionDto); ``` **Example:** ```csharp var result = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "DataSourceX", identifier: Guid.NewGuid().ToString(), type: "VirtualSession", version: 1, details: new[] { new SessionDetailDto("CollectionId", "C12345"), new SessionDetailDto("BatchNumber", "B789") } )); if (result.Success && result.Data != null) { var sessionKey = result.Data.SessionKey; Console.WriteLine($"Session created: {sessionKey}"); } ``` ### GetSessionInfo Retrieves detailed information about a specific session. ```csharp public ApiResult GetSessionInfo(string dataSource, string sessionKey) ``` **Example:** ```csharp var result = sessionService.GetSessionInfo("MyDataSource", sessionKey); if (result.Success && result.Data != null) { var session = result.Data; Console.WriteLine($"Session: {session.Identifier}"); Console.WriteLine($"Type: {session.Type}"); Console.WriteLine($"Historical: {session.Historical}"); Console.WriteLine($"Details: {session.Details.Count}"); } ``` ### GetAllSessions Retrieves all sessions from the broker. ```csharp public ApiResult> GetAllSessions() ``` **Example:** ```csharp var result = sessionService.GetAllSessions(); if (result.Success && result.Data != null) { Console.WriteLine($"Found {result.Data.Count} sessions"); foreach (var session in result.Data) { Console.WriteLine($"- {session.SessionKey}: {session.Identifier} ({session.Type})"); } } ``` ### UpdateIdentifier Updates the session identifier. ```csharp public ApiResult UpdateIdentifier( string dataSource, string sessionKey, string newIdentifier) ``` **Example:** ```csharp var result = sessionService.UpdateIdentifier( "MyDataSource", sessionKey, "UpdatedSessionIdentifier_001"); if (result.Success && result.Data != null) { Console.WriteLine($"Updated identifier: {result.Data.Identifier}"); } ``` ### UpdateSessionDetails Adds or updates session metadata details. ```csharp public ApiResult UpdateSessionDetails( string dataSource, string sessionKey, IReadOnlyList sessionDetails) ``` **Example:** ```csharp var newDetails = new[] { new SessionDetailDto("LapCount", "45"), new SessionDetailDto("BestLapTime", "1:32.456") }; var result = sessionService.UpdateSessionDetails( "MyDataSource", sessionKey, newDetails); if (result.Success && result.Data != null) { Console.WriteLine($"Session now has {result.Data.Details.Count} details"); } ``` **Note**: Details with existing keys are updated; new keys are added. ### AddAssociateSession Links another session as an associate. ```csharp public ApiResult AddAssociateSession( string dataSource, string sessionKey, string addingAssociateKey) ``` **Example:** ```csharp // Create an associate session var associateResult = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "MyDataSource", identifier: "AssociateSession", type: "Data" )); if (associateResult.Success && associateResult.Data != null) { var associateKey = associateResult.Data.SessionKey; // Link it to the main session var linkResult = sessionService.AddAssociateSession( "MyDataSource", mainSessionKey, associateKey); if (linkResult.Success && linkResult.Data != null) { Console.WriteLine($"Associated sessions: {linkResult.Data.AssociateSessionKeys.Count}"); } } ``` ### EndSession Marks a session as historical (completed). ```csharp public ApiResult EndSession(string dataSource, string sessionKey) ``` **Example:** ```csharp var result = sessionService.EndSession("MyDataSource", sessionKey); if (result.Success && result.Data != null) { Console.WriteLine($"Session ended. Historical: {result.Data.Historical}"); } ``` **Important**: Once a session is marked as historical, it should not be modified. ## Events The Session Manager provides four events for tracking session lifecycle. **Important - Forward-Only Event Notifications**: All events are **only triggered for actions that occur after the session manager has started**. Events will not fire for sessions or changes that existed before the session manager was initialized. This means: - No replay of existing sessions or state when you subscribe - Only new changes happening after startup will trigger events - To retrieve existing sessions, use the `GetAllSessions()` method **Event Scope**: - `LiveSessionStarted`, `LiveSessionStateChange`, `LiveSessionEnded`: Only triggered for live sessions (where `Historical = false`) - `NewAssociationDetected`: Can be triggered for both live and historical sessions ### LiveSessionStarted Fired when a new live session is detected or created after the session manager has started. ```csharp sessionService.LiveSessionStarted += (sender, sessionInfo) => { Console.WriteLine($"Live session started: {sessionInfo.SessionKey}"); Console.WriteLine($"Data Source: {sessionInfo.DataSource}"); Console.WriteLine($"Identifier: {sessionInfo.Identifier}"); }; ``` **Note**: Only triggered for live sessions created or detected after session manager startup. Pre-existing live sessions will not trigger this event. ### LiveSessionStateChange Fired when a live session's state is updated (e.g., details changed, identifier updated) after the session manager has started. ```csharp sessionService.LiveSessionStateChange += (sender, sessionInfo) => { Console.WriteLine($"Live session state changed: {sessionInfo.SessionKey}"); Console.WriteLine($"Details count: {sessionInfo.Details.Count}"); }; ``` **Note**: Only triggered for state changes occurring after session manager startup. Does not fire for historical sessions. ### LiveSessionEnded Fired when a live session is marked as historical (transitioned from live to completed) after the session manager has started. ```csharp sessionService.LiveSessionEnded += (sender, sessionInfo) => { Console.WriteLine($"Live session ended: {sessionInfo.SessionKey}"); Console.WriteLine($"Final identifier: {sessionInfo.Identifier}"); Console.WriteLine($"Now historical: {sessionInfo.Historical}"); // Will be true }; ``` **Note**: Triggered during the transition from live to historical that occurs after session manager startup. Once a session is historical, no further live session events are triggered for it. ### NewAssociationDetected Fired when a session association is created between sessions after the session manager has started. ```csharp sessionService.NewAssociationDetected += (sender, associationInfo) => { Console.WriteLine($"New association detected:"); Console.WriteLine($" Main: {associationInfo.MainSessionInfo.SessionKey}"); Console.WriteLine($" Associate: {associationInfo.AssociateSessionInfo.SessionKey}"); }; ``` **Note**: Can be triggered for both live and historical sessions, but **only for associations created after the session manager has started**. Pre-existing associations will not trigger this event. ## Complete Example ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule.Abstractions; public class SessionManagerExample { private readonly ISessionManagementService sessionService; public SessionManagerExample(ISupportLibApi supportLibApi) { var sessionApi = supportLibApi.GetSessionManagerApi(); this.sessionService = sessionApi.CreateService().Data; // Subscribe to events this.sessionService.LiveSessionStarted += OnLiveSessionStarted; this.sessionService.LiveSessionStateChange += OnSessionStateChange; this.sessionService.LiveSessionEnded += OnSessionEnded; this.sessionService.NewAssociationDetected += OnAssociationDetected; // Initialize and start this.sessionService.Initialise(); this.sessionService.Start(); } public string CreateAndManageSession() { // Create session var createResult = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "TestCar", identifier: "Test_Run_001", type: "Testing", version: 1, details: new[] { new SessionDetailDto("Driver", "TestDriver"), new SessionDetailDto("Track", "TestTrack") })); if (!createResult.Success || createResult.Data == null) { Console.WriteLine($"Failed to create session: {createResult.Message}"); return null; } var sessionKey = createResult.Data.SessionKey; // Update session details sessionService.UpdateSessionDetails( "TestCar", sessionKey, new[] { new SessionDetailDto("Status", "InProgress") }); // Create and link associate session var associateResult = sessionService.CreateNewSession( new SessionCreationDto( dataSource: "TestCar", identifier: "Associated_Data", type: "Telemetry")); if (associateResult.Success && associateResult.Data != null) { sessionService.AddAssociateSession( "TestCar", sessionKey, associateResult.Data.SessionKey); } // Later... end the session sessionService.EndSession("TestCar", sessionKey); return sessionKey; } private void OnLiveSessionStarted(object sender, ISessionInfo session) { Console.WriteLine($"[Event] Live session started: {session.SessionKey}"); } private void OnSessionStateChange(object sender, ISessionInfo session) { Console.WriteLine($"[Event] Session state changed: {session.SessionKey}"); } private void OnSessionEnded(object sender, ISessionInfo session) { Console.WriteLine($"[Event] Session ended: {session.SessionKey}"); } private void OnAssociationDetected(object sender, ISessionAssociationInfo info) { Console.WriteLine($"[Event] Association: {info.MainSessionInfo.SessionKey} <-> {info.AssociateSessionInfo.SessionKey}"); } } ``` ## Best Practices 1. **Always Check Results**: All methods return `ApiResult` - always check the `Success` property and handle errors appropriately. 2. **Subscribe to Events Early**: Subscribe to events before calling `Start()` to ensure you don't miss any notifications. 3. **Handle Session Lifecycle**: Use the event system to track when sessions start and end for proper resource management. 4. **Store Session Keys**: Keep track of session keys for later reference, as they're needed for most operations. 5. **End Sessions Properly**: Always call `EndSession()` when a session is complete to mark it as historical. 6. **Use Details for Metadata**: Store important session metadata in details for easy retrieval and filtering. ## Error Handling ```csharp var result = sessionService.CreateNewSession(sessionDto); if (!result.Success) { Console.WriteLine($"Error: {result.Message}"); // Handle the error appropriately return; } if (result.Data == null) { Console.WriteLine("Error: No data returned despite success"); return; } // Proceed with the session var sessionKey = result.Data.SessionKey; ``` ## See Also - [Data Format Manager Module](data-format-manager.md) - [Writer Module](writer-module.md) - [Reader Module](reader-module.md) - [API Reference](api-reference.md) --- # Data Format Manager ## Overview The Data Format Manager Module is the second independent core module in the Support Library. It manages data format definitions for parameters and events, automatically generating and caching unique format IDs. This module is essential for defining the structure of data before it can be streamed. ## Key Features - **Parameter Format Management**: Define parameter groups with automatic ID generation - **Event Format Management**: Define event types with unique IDs - **Automatic ID Generation**: Consistent, deterministic ID generation based on content - **Format Caching**: Efficient retrieval of previously defined formats - **Data Source Isolation**: Separate format spaces per data source - **Real-time Notifications**: Events when new formats are detected - **Format Querying**: Retrieve all formats for a data source or specific format details ## API Access ### Get the Data Format Manager API ```csharp var dataFormatApi = supportLibApi.GetDataFormatManagerApi(); ``` ### Create a Service Instance ```csharp var serviceResult = dataFormatApi.CreateService(); if (serviceResult.Success && serviceResult.Data != null) { var dataFormatService = serviceResult.Data; // Initialize and start the service dataFormatService.Initialise(); dataFormatService.Start(); } ``` ## Core Concepts ### Data Format Types The module manages two types of data formats: 1. **Parameters**: Groups of parameters (e.g., `["Param1", "Param2", "Param3"]`) 2. **Events**: Single event identifiers (e.g., `"EventA"`, `"EventB"`) ### Data Format Information Each format definition contains: - **DataFormatId**: Unique identifier (ulong) for the format - **DataSource**: The data source this format belongs to - **Key**: Derived identifier string - **Identifiers**: List of parameter names or event name - **DataFormatInfoType**: Type (Parameters or Event) ### Format ID Generation Format IDs are generated deterministically based on: - Data source name - Parameter names (for parameters) - Format type (parameter group vs. event) This ensures the same format always gets the same ID, enabling efficient caching and lookup. ## Methods ### GetParameterDataFormatId Gets or creates a parameter format definition for a group of parameters. ```csharp public ApiResult GetParameterDataFormatId( string dataSource, IReadOnlyList parameterList) ``` **Parameters:** - `dataSource`: The data source identifier - `parameterList`: List of parameter names in the group **Returns:** - `ParameterDataFormatInfo` containing: - `DataFormatId`: Unique ID for this parameter group - `ParameterList`: The list of parameters **Example:** ```csharp var result = dataFormatService.GetParameterDataFormatId( "DataSourceA", new[] { "Param1", "Param2", "Param3", "Param4" }); if (result.Success && result.Data != null) { var formatId = result.Data.DataFormatId; var parameters = result.Data.ParameterList; Console.WriteLine($"Parameter Format ID: {formatId}"); Console.WriteLine($"Parameters: {string.Join(", ", parameters)}"); } ``` ### GetParametersList Retrieves the parameter list for a given format ID. ```csharp public ApiResult GetParametersList( string dataSource, ulong dataFormatId) ``` **Example:** ```csharp var result = dataFormatService.GetParametersList("DataSourceA", formatId); if (result.Success && result.Data != null) { Console.WriteLine($"Parameters in format {formatId}:"); foreach (var param in result.Data.ParameterList) { Console.WriteLine($" - {param}"); } } ``` ### GetEventDataFormatId Gets or creates an event format definition. ```csharp public ApiResult GetEventDataFormatId( string dataSource, string eventName) ``` **Parameters:** - `dataSource`: The data source identifier - `eventName`: The event identifier/name **Returns:** - `EventDataFormatInfo` containing: - `DataFormatId`: Unique ID for this event - `EventName`: The event name **Example:** ```csharp var result = dataFormatService.GetEventDataFormatId("DataSourceA", "EventTypeA"); if (result.Success && result.Data != null) { var eventFormatId = result.Data.DataFormatId; var eventName = result.Data.EventName; Console.WriteLine($"Event '{eventName}' has ID: {eventFormatId}"); } ``` ### GetEvent Retrieves event information for a given format ID. ```csharp public ApiResult GetEvent( string dataSource, ulong dataFormatId) ``` **Example:** ```csharp var result = dataFormatService.GetEvent("DataSourceA", eventFormatId); if (result.Success && result.Data != null) { Console.WriteLine($"Event format {dataFormatId}: {result.Data.EventName}"); } ``` ### GetDataSourceDataFormatInfo Retrieves all data format definitions for a data source. ```csharp public ApiResult?> GetDataSourceDataFormatInfo( string dataSource) ``` **Example:** ```csharp var result = dataFormatService.GetDataSourceDataFormatInfo("DataSourceA"); if (result.Success && result.Data != null) { Console.WriteLine($"Found {result.Data.Count} data formats for DataSourceA:"); foreach (var format in result.Data) { Console.WriteLine($" Format IDs: {string.Join(",",format.DataFormats)}"); Console.WriteLine($" Type: {format.DataFormatInfoType}"); Console.WriteLine($" Identifiers: {string.Join(", ", format.Identifiers)}"); } } ``` ## Events ### DataFormatInfoUpdated Fired when a new data format is created or an existing format is detected. ```csharp dataFormatService.DataFormatInfoUpdated += (sender, dataFormatInfo) => { Console.WriteLine($"Data format updated:"); Console.WriteLine($" Data Source: {dataFormatInfo.DataSource}"); Console.WriteLine($" Format IDs: {string.Join(",", dataFormatInfo.DataFormats)}"); Console.WriteLine($" Type: {dataFormatInfo.DataFormatInfoType}"); Console.WriteLine($" Identifiers: {string.Join(", ", dataFormatInfo.Identifiers)}"); }; ``` **Event Data (`DataFormatInfo`):** - `DataSource`: The data source identifier - `DataFormatId`: The unique format ID - `Key`: The format key - `Identifiers`: List of parameter names or event name - `DataFormatInfoType`: Either `DataFormatInfoType.Parameters` or `DataFormatInfoType.Event` ## Complete Example ```csharp public class DataFormatManagerExample { private readonly IDataFormatManagementService dataFormatService; public DataFormatManagerExample(ISupportLibApi supportLibApi) { var dataFormatApi = supportLibApi.GetDataFormatManagerApi(); this.dataFormatService = dataFormatApi.CreateService().Data; // Subscribe to events this.dataFormatService.DataFormatInfoUpdated += OnDataFormatUpdated; // Initialize and start this.dataFormatService.Initialise(); this.dataFormatService.Start(); } public void DefineDataFormats() { const string DataSource = "DataSourceX"; // Define first parameter group var group1Params = dataFormatService.GetParameterDataFormatId( DataSource, new[] { "Param1", "Param2", "Param3", "Param4" }); if (group1Params.Success && group1Params.Data != null) { Console.WriteLine($"Group 1 parameters format ID: {group1Params.Data.DataFormatId}"); } // Define second parameter group var group2Params = dataFormatService.GetParameterDataFormatId( DataSource, new[] { "Param5", "Param6", "Param7" }); if (group2Params.Success && group2Params.Data != null) { Console.WriteLine($"Group 2 parameters format ID: {group2Params.Data.DataFormatId}"); } // Define individual parameter (single parameter is still a list) var singleParam = dataFormatService.GetParameterDataFormatId( DataSource, new[] { "Param8" }); // Define events var event1 = dataFormatService.GetEventDataFormatId(DataSource, "EventA"); var event2 = dataFormatService.GetEventDataFormatId(DataSource, "EventB"); var event3 = dataFormatService.GetEventDataFormatId(DataSource, "EventC"); // Retrieve all formats var allFormats = dataFormatService.GetDataSourceDataFormatInfo(DataSource); if (allFormats.Success && allFormats.Data != null) { Console.WriteLine($"\nTotal formats defined: {allFormats.Data.Count}"); var paramFormats = allFormats.Data.Where(f => f.DataFormatInfoType == DataFormatInfoType.Parameters); var eventFormats = allFormats.Data.Where(f => f.DataFormatInfoType == DataFormatInfoType.Event); Console.WriteLine($"Parameter formats: {paramFormats.Count()}"); Console.WriteLine($"Event formats: {eventFormats.Count()}"); } } public void LookupFormats() { const string DataSource = "DataSourceX"; // Create a format var createResult = dataFormatService.GetParameterDataFormatId( DataSource, new[] { "Param1", "Param2" }); if (!createResult.Success || createResult.Data == null) { return; } var formatId = createResult.Data.DataFormatId; // Later, look up what parameters this format contains var lookupResult = dataFormatService.GetParametersList(DataSource, formatId); if (lookupResult.Success && lookupResult.Data != null) { Console.WriteLine($"Format {formatId} contains:"); foreach (var param in lookupResult.Data.ParameterList) { Console.WriteLine($" - {param}"); } } } private void OnDataFormatUpdated(object sender, DataFormatInfo formatInfo) { Console.WriteLine($"[Event] Data formats updated: {string.Join(",",formatInfo.DataFormats)}"); Console.WriteLine($" Type: {formatInfo.DataFormatInfoType}"); Console.WriteLine($" Identifiers: {string.Join(", ", formatInfo.Identifiers)}"); } } ``` ## Best Practices 1. **Define Formats Early**: Define all data formats at the beginning of your session before streaming data. 2. **Group Related Parameters**: Group parameters that are logically related and will be sent together in packets. 3. **Consistent Naming**: Use consistent parameter and event naming conventions across your application. 4. **Cache Format IDs**: Store format IDs after creation to avoid repeated lookups. 5. **Check Return Values**: Always validate the `ApiResult` success and data before using format IDs. 6. **Per-Source Isolation**: Remember that formats are isolated per data source - the same parameter names in different data sources will have different format IDs. ## Data Source Organization ```csharp public class MultiSourceFormats { private readonly IDataFormatManagementService formatService; public void DefineFormatsForMultipleSources() { var sources = new[] { "DataSourceA", "DataSourceB", "DataSourceC" }; var parameters = new[] { "Param1", "Param2", "Param3" }; foreach (var source in sources) { // Each source gets its own format space var result = formatService.GetParameterDataFormatId(source, parameters); if (result.Success && result.Data != null) { Console.WriteLine($"{source} format ID: {result.Data.DataFormatId}"); // Different IDs for each source even with same parameters } } } } ``` ## Error Handling ```csharp var result = dataFormatService.GetParameterDataFormatId(dataSource, parameterList); if (!result.Success) { Console.WriteLine($"Error creating parameter format: {result.Message}"); // Handle the error return; } if (result.Data == null) { Console.WriteLine("Error: No data returned despite success"); return; } // Use the format ID var formatId = result.Data.DataFormatId; ``` ## Integration with Other Modules ### With Writer Module ```csharp // Define the format var formatResult = dataFormatService.GetParameterDataFormatId( dataSource, new[] { "Speed", "RPM" }); // Use the format ID when writing packets if (formatResult.Success && formatResult.Data != null) { var formatId = formatResult.Data.DataFormatId; // Create packet with this format ID and write using Writer Module } ``` ### With Session Manager ```csharp // Create a session var sessionResult = sessionService.CreateNewSession( new SessionCreationDto(dataSource: "CarA")); // Define formats for this session's data var formatResult = dataFormatService.GetParameterDataFormatId( "CarA", new[] { "Speed", "RPM", "Throttle" }); ``` ## See Also - [Session Manager Module](session-manager.md) - [Writer Module](writer-module.md) - [Reader Module](reader-module.md) - [API Reference](api-reference.md) --- # Writer Module ## Overview The Writer Module (Packet Writing Module) provides API for publishing data packets and session information to the streaming broker (Kafka). It handles the serialization and transmission of telemetry data, events, markers, and session metadata. ## Key Features - **Packet Writing**: Publish data packets to specific streams - **Session Info Publishing**: Write session-related information to the broker - **Multiple Stream Support**: Write to different streams within a session - **Info Type Support**: Publish different types of information (SessionInfo, DataFormatInfo, etc.) - **Automatic Serialization**: Handles packet serialization internally - **Error Handling**: Returns detailed results for each write operation ## API Access ### Get the Writer API ```csharp var writerApi = supportLibApi.GetWritingPacketApi(); ``` ### Create a Writer Service ```csharp var serviceResult = writerApi.CreateService(); if (serviceResult.Success && serviceResult.Data != null) { var writerService = serviceResult.Data; // Initialize and start the service writerService.Initialise(); writerService.Start(); } ``` **Note**: Unlike Reader services, Writer services are not tied to a specific session. A single writer service can write to multiple sessions and data sources. ## Methods ### WriteData Writes a data packet to a specific stream within a session. ```csharp public ApiResult WriteData( string dataSource, string stream, string sessionKey, Packet packet) ``` **Parameters:** - `dataSource`: The data source identifier - `stream`: The stream name within the session - `sessionKey`: The session key to write to - `packet`: The packet to write **Returns:** - `ApiResult` indicating success or failure **Example:** ```csharp using MA.Streaming.OpenData; // Create a packet (example with PeriodicDataPacket) var periodicPacket = new PeriodicDataPacket { DataFormat = new SampleDataFormat { DataFormatIdentifier = 1234 }, StartTime = 123456, Interval = 1000000, // 1ms in nanoseconds Columns = { new SampleColumn[] { new() { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample[] { new DoubleSample { Status = DataStatus.Valid, Value = 123456 } } } } } } } }; var packet = new Packet { SessionKey = sessionKey, Content = periodicPacket.ToByteString(), Id = packetCounter++, Type = "PeriodicData", IsEssential = false }; var result = writerService.WriteData( dataSource: "CarA", stream: "Telemetry", sessionKey: sessionKey, packet: packet); if (result.Success) { Console.WriteLine("Packet written successfully"); } else { Console.WriteLine($"Failed to write packet: {result.Message}"); } ``` ### WriteInfo Writes information packets (metadata) to the broker. ```csharp public ApiResult WriteInfo( Packet packet, InfoType infoType = InfoType.SessionInfo) ``` **Parameters:** - `packet`: The information packet to write - `infoType`: The type of information (default: SessionInfo) **InfoType Values:** - `InfoType.SessionInfo`: Session-related information - `InfoType.DataFormatInfo`: Data format definitions - `InfoType.Other`: Other metadata types **Example:** ```csharp var infoPacket = new Packet { SessionKey = sessionKey, Content = sessionInfoData.ToByteString(), Type = "SessionInfo", IsEssential = true }; var result = writerService.WriteInfo( packet: infoPacket, infoType: InfoType.SessionInfo); if (!result.Success) { Console.WriteLine($"Failed to write info: {result.Message}"); } ``` ## Packet Types The library supports various packet types from the MA.Streaming.OpenData namespace: ### Data Packets - **PeriodicDataPacket**: Regular interval data (e.g., 100Hz telemetry) - **RowDataPacket**: Timestamped data rows - **SynchroDataPacket**: Data logged in synchronous of another. - **EventPacket**: Event occurrences - **MarkerPacket**: Session markers - **ErrorPacket**: Error occurrences. - **RawCANPacket**: Raw CAN packets. - **ConfigurationPacket**: Packet containing list of parameters, events, and group definitions. This packet must be sent after creating the session and before writing data, so that consumers (e.g. ATLAS) know how to interpret the parameters. ### Control Packets - **NewSessionPacket**: Session start indicator - **EndOfSessionPacket**: Session end indicator - **DataFormatDefinitionPacket**: Format definitions - **CoverageCursorInfoPacket**: Data coverage information ## Complete Example ```csharp using Google.Protobuf; using MA.DataPlatforms.Streaming.Support.Lib.Core.Abstractions; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.DataFormatInfoModule; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule.Abstractions; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.WritingModule; using MA.Streaming.OpenData; public class DataPublisher { private readonly IPacketWriterService writerService; private readonly ISessionManagementService sessionService; private readonly IDataFormatManagementService formatService; private readonly string dataSource; private ulong packetId; public DataPublisher( ISupportLibApi supportLibApi, string dataSource) { this.dataSource = dataSource; // Get writer service var writerApi = supportLibApi.GetWritingPacketApi(); this.writerService = writerApi.CreateService().Data; this.writerService.Initialise(); this.writerService.Start(); // Get session and format services var sessionApi = supportLibApi.GetSessionManagerApi(); this.sessionService = sessionApi.CreateService().Data; this.sessionService.Initialise(); this.sessionService.Start(); var formatApi = supportLibApi.GetDataFormatManagerApi(); this.formatService = formatApi.CreateService().Data; this.formatService.Initialise(); this.formatService.Start(); } public void PublishSession() { // 1. Create session var sessionResult = this.sessionService.CreateNewSession( new SessionCreationDto( dataSource: this.dataSource, identifier: "TestRun_001", type: "Testing", details: [ new SessionDetailDto("Driver", "TestDriver"), new SessionDetailDto("Track", "TestTrack") ])); if (!sessionResult.Success || sessionResult.Data == null) { Console.WriteLine($"Failed to create session: {sessionResult.Message}"); return; } var sessionKey = sessionResult.Data.SessionKey; Console.WriteLine($"Created session: {sessionKey}"); // 2. Define data formats var engineFormatResult = this.formatService.GetParameterDataFormatId( this.dataSource, [ "RPM", "EngineTemp", "Throttle" ]); if (!engineFormatResult.Success || engineFormatResult.Data == null) { Console.WriteLine("Failed to create format"); return; } var engineFormatId = engineFormatResult.Data.DataFormatId; // 3. Send session start packet this.SendSessionStartPacket(sessionKey); // 4. Send periodic data for (int i = 0; i < 100; i++) { this.SendPeriodicData(sessionKey, engineFormatId, (ulong)i * 10000); // 10ms intervals Thread.Sleep(10); // Simulate real-time } // 5. Send session end packet this.SendSessionEndPacket(sessionKey); // 6. Mark session as complete this.sessionService.EndSession(this.dataSource, sessionKey); Console.WriteLine("Session publishing complete"); } private void SendSessionStartPacket(string sessionKey) { var startPacket = new NewSessionPacket { DataSource = this.dataSource, }; var packet = new Packet { SessionKey = sessionKey, Content = startPacket.ToByteString(), Id = this.packetId++, Type = nameof(NewSessionPacket).Replace("Packet", ""), IsEssential = true }; this.writerService.WriteData(this.dataSource, "Main", sessionKey, packet); } private void SendPeriodicData(string sessionKey, ulong formatId, ulong timestamp) { var periodicData = new PeriodicDataPacket { DataFormat = new SampleDataFormat { DataFormatIdentifier = formatId }, StartTime = timestamp, Interval = 10000000, // 10ms in nanoseconds // Add sample values (RPM, EngineTemp, Throttle) Columns = { new SampleColumn[] { new() { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample[] { new() { Value = 8500.0, Status = DataStatus.Valid }, new() { Value = 8501.0, Status = DataStatus.Valid }, new() { Value = 8502.0, Status = DataStatus.Valid } } } } }, new() { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample[] { new() { Value = 92.5, Status = DataStatus.Valid }, new() { Value = 92.5, Status = DataStatus.Valid }, new() { Value = 92.6, Status = DataStatus.Valid } } } } }, new() { DoubleSamples = new DoubleSampleList { Samples = { new DoubleSample[] { new() { Value = 0.85, Status = DataStatus.Valid }, new() { Value = 0.86, Status = DataStatus.Valid }, new() { Value = 0.87, Status = DataStatus.Valid } } } } } } } }; var packet = new Packet { SessionKey = sessionKey, Content = periodicData.ToByteString(), Id = this.packetId++, Type = nameof(PeriodicDataPacket).Replace("Packet", ""), IsEssential = false }; var result = this.writerService.WriteData(this.dataSource, "Telemetry", sessionKey, packet); if (!result.Success) { Console.WriteLine($"Failed to write data: {result.Message}"); } } private void SendSessionEndPacket(string sessionKey) { var endPacket = new EndOfSessionPacket { DataSource = this.dataSource }; var packet = new Packet { SessionKey = sessionKey, Content = endPacket.ToByteString(), Id = this.packetId++, Type = nameof(EndOfSessionPacket).Replace("Packet", ""), IsEssential = true }; this.writerService.WriteData(this.dataSource, "Main", sessionKey, packet); } } ``` ## Writing Different Packet Types ### Periodic Data (Regular Intervals) ```csharp public void WritePeriodicData(string sessionKey, ulong formatId, ulong timestamp) { var periodicData = new PeriodicDataPacket { DataFormat = new SampleDataFormat { DataFormatIdentifier = formatId }, StartTime = timestamp, Interval = 1000000, // 1ms in nanoseconds Columns = [ new SampleColumn { DoubleSamples = new DoubleSampleList { Samples = [ new DoubleSample { Value = 100.5, Status = DataStatus.Valid }, new DoubleSample { Value = 101.2, Status = DataStatus.Valid }, new DoubleSample { Value = 102.0, Status = DataStatus.Valid } ] } } ] }; var packet = new Packet { SessionKey = sessionKey, Content = periodicData.ToByteString(), Id = packetId++, Type = "PeriodicData", IsEssential = false }; writerService.WriteData(dataSource, "Telemetry", sessionKey, packet); } ``` ### Row Data (Timestamped Rows) ```csharp public void WriteRowData(string sessionKey, ulong formatId) { var rowData = new RowDataPacket { DataFormat = new SampleDataFormat { DataFormatIdentifier = formatId }, // The number of rows and timestamps must match. Timestamps = [ 1000000, 1001000, 1002000 ], Rows = [ new SampleRow { DoubleSamples = new DoubleSampleList { Samples = [ // Add values for each parameter new DoubleSample { Value = 100.0 Status = DataStatus.Valid }, new DoubleSample { Value = 101.0 Status = DataStatus.Valid }, new DoubleSample { Value = 102.0 Status = DataStatus.Valid }, ] } }, new SampleRow { DoubleSamples = new DoubleSampleList { Samples = [ // Add values for each parameter new DoubleSample { Value = 101.0 Status = DataStatus.Valid }, new DoubleSample { Value = 102.0 Status = DataStatus.Valid }, new DoubleSample { Value = 103.0 Status = DataStatus.Valid }, ] } }, new SampleRow { DoubleSamples = new DoubleSampleList { Samples = [ // Add values for each parameter new DoubleSample { Value = 102.0 Status = DataStatus.Valid }, new DoubleSample { Value = 103.0 Status = DataStatus.Valid }, new DoubleSample { Value = 104.0 Status = DataStatus.Valid }, ] } } ] }; var packet = new Packet { SessionKey = sessionKey, Content = rowData.ToByteString(), Id = packetId++, Type = "RowData", IsEssential = false }; writerService.WriteData(dataSource, "VariableRate", sessionKey, packet); } ``` ### Event Data ```csharp public void WriteEvent(string sessionKey, ulong eventFormatId, ulong timestamp) { var eventData = new EventPacket { DataFormat = new EventDataFormat { DataFormatIdentifier = eventFormatId }, Timestamp = timestamp RawValues = [/* add raw values for the event if needed */] }; var packet = new Packet { SessionKey = sessionKey, Content = eventData.ToByteString(), Id = packetId++, Type = "EventData", IsEssential = true }; writerService.WriteData(dataSource, "Events", sessionKey, packet); } ``` ### Marker Data ```csharp public void WriteMarker(string sessionKey, string markerLabel, ulong timestamp) { var markerData = new MarkerPacket { Timestamp = timestamp, Label = markerLabel, Description = "Important event marker" }; var packet = new Packet { SessionKey = sessionKey, Content = markerData.ToByteString(), Id = packetId++, Type = "MarkerData", IsEssential = false }; writerService.WriteData(dataSource, "Markers", sessionKey, packet); } ``` ## Best Practices 1. **Initialize Once**: Create a single writer service that can write to multiple sessions. 2. **Sequential Packet IDs**: Maintain a counter for packet IDs to ensure proper ordering. 3. **Mark Essential Packets**: Set `IsEssential = true` for critical packets (session start/end, format definitions). 4. **Use Appropriate Streams**: Organize data into logical streams (e.g., "Telemetry", "Events", "Markers"). 5. **Send Session Lifecycle Packets**: Always send NewSessionPacket at start and EndOfSessionPacket at end of each stream. 6. **Send ConfigurationPacket Early**: After creating and starting the session, send a `ConfigurationPacket` with your parameter definitions before writing data. This tells consumers how to interpret the data. 7. **Define Formats Before Data**: Create data formats (to get format IDs) before sending data packets that reference them. 8. **Handle Write Failures**: Always check the result and handle failures appropriately. 9. **Batch When Possible**: Group related data in single packets to reduce overhead. ## Error Handling ```csharp public bool SafeWriteData(string dataSource, string stream, string sessionKey, Packet packet) { var result = writerService.WriteData(dataSource, stream, sessionKey, packet); if (!result.Success) { logger.Error($"Failed to write packet {packet.Id}: {result.Message}"); // Implement retry logic if needed for (int retry = 0; retry < 3; retry++) { Thread.Sleep(100 * (retry + 1)); result = writerService.WriteData(dataSource, stream, sessionKey, packet); if (result.Success) { logger.Info($"Packet {packet.Id} written on retry {retry + 1}"); return true; } } return false; } return true; } ``` ## Integration with Other Modules ### With Session Manager ```csharp // Create session first var sessionResult = sessionService.CreateNewSession(sessionDto); var sessionKey = sessionResult.Data.SessionKey; // Then write data to that session writerService.WriteData(dataSource, stream, sessionKey, packet); // End session when done sessionService.EndSession(dataSource, sessionKey); ``` ### With Data Format Manager ```csharp // Define format first var formatResult = formatService.GetParameterDataFormatId(dataSource, parameters); var formatId = formatResult.Data.DataFormatId; // Create packet with format ID var periodicData = new PeriodicDataPacket { DataFormat = new SampleDataFormat{DataFormatIdentifier = formatId}, // ... data }; // Write the packet writerService.WriteData(dataSource, stream, sessionKey, packet); ``` ## Performance Considerations 1. **Minimize Serialization Overhead**: Reuse packet objects when possible 2. **Batch Data**: Combine multiple values in single packets 3. **Use Appropriate Intervals**: Don't send data more frequently than needed 4. **Stream Selection**: Use different streams to parallelize writes 5. **Essential Flag**: Use sparingly to avoid unnecessary broker overhead ## Cleanup ```csharp // When done with the writer service writerService.Stop(); ``` ## See Also - [Session Manager Module](session-manager.md) - [Data Format Manager Module](data-format-manager.md) - [Reader Module](reader-module.md) - [API Reference](api-reference.md) --- # Reader Module ## Overview The Reader Module (Packet Reading Module) provides API for consuming data packets from the streaming broker. It supports both live session monitoring and historical session playback, with configurable options for stream filtering, auto-start behavior, and inactivity timeout. ## Key Features - **Live Reading**: Monitor live sessions as data arrives - **Historical Playback**: Read complete historical sessions - **Stream Filtering**: Read specific streams or all streams - **Auto-Start or Manual Start**: Control when reading begins - **Coverage Cursor Support**: Track data availability and gaps - **Event System**: Rich events for tracking reading progress - **Handler Pattern**: Attach custom handlers to process packets - **Session Pattern Matching**: Read sessions matching specific patterns - **Inactivity Timeout**: Automatically stop reading after inactivity period ## API Access ### Get the Reader API ```csharp var readerApi = supportLibApi.GetReadingPacketApi(); ``` ### Create a Reader Service There are two ways to create a reader service: #### Option 1: Simple Creation (Data Source + Session Key) ```csharp var serviceResult = readerApi.CreateService( dataSource: "CarA", sessionKey: "specific-session-key"); if (serviceResult.Success && serviceResult.Data != null) { var readerService = serviceResult.Data; } ``` #### Option 2: Advanced Creation (Using Configuration) ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.ReadingModule; var config = new PacketReadingConfiguration( dataSource: "CarA", sessionIdentifierPattern: "*", // Match all sessions readingType: ReadingType.Live, streams: new[] { "Telemetry", "Events" }, excludeMainStream: false, inactivityTimeoutSeconds: 30, sessionKey: null, liveReadingType: LiveReadingType.FromBeginning, bufferLength: 1000, groupId: "my-consumer-group"); var serviceResult = readerApi.CreateService(config); ``` ## Configuration Options ### PacketReadingConfiguration ```csharp public PacketReadingConfiguration( string? dataSource = "Default", string? sessionIdentifierPattern = "*", ReadingType? readingType = ReadingType.Live, IReadOnlyList? streams = null, bool? excludeMainStream = false, uint? inactivityTimeoutSeconds = 30, string? sessionKey = null, LiveReadingType? liveReadingType = null, int? bufferLength = null, string? groupId = null, IReadOnlyList? excludedStreams = null) ``` **Parameters:** - **dataSource**: The data source to read from (default: "Default") - **sessionIdentifierPattern**: Pattern for matching session identifiers - Use `"*"` to match all session identifires - Use specific identifier to match exact session - **readingType**: Type of reading mode - `ReadingType.Live`: Monitor live sessions - `ReadingType.Historical`: Read completed sessions - **streams**: List of specific streams to read (default: all streams) - `null` or empty list reads all streams - Specify streams like `["Telemetry", "Events"]` to filter - **excludeMainStream**: Whether to exclude the main stream (default: false) - **inactivityTimeoutSeconds**: Timeout in seconds after which reading stops if no data arrives (default: 30) - **sessionKey**: Specific session key to read (optional) - If provided, only this session is read - If null, pattern matching is used - **liveReadingType**: Type of live reading mode (default: `LiveReadingType.FromBeginning`) - `LiveReadingType.FromBeginning`: Read from the start of the session - `LiveReadingType.LeadingEdge`: Read only new data from the current point forward (skip historical data) - **bufferLength**: Size of the internal buffer for packets (default: 0 = use system default) - Larger buffers can handle higher data rates but use more memory - Set to appropriate size based on expected packet rate and processing speed - Value of 0 uses the default unbounded buffer - **groupId**: Kafka consumer group identifier (default: empty string = auto-generated) - Use a specific group ID to control consumer group membership - Multiple readers with same group ID share the load (Kafka consumer group behavior) - Use different group IDs for independent readers of the same topic - Empty string creates a unique consumer group per reader - **excludedStreams**: List of streams to skip when reading a session (default: empty = nothing excluded) - Applied after the `streams` selection: a stream that appears in both lists is not read - Useful to read everything except a few known-noisy streams, without having to list every stream you do want in `streams` ### ReadingType Enum ```csharp public enum ReadingType { Live, // Read live sessions Historical // Read historical (completed) sessions } ``` ### LiveReadingType Enum ```csharp public enum LiveReadingType { FromBeginning = 0, // Read live session from the beginning LeadingEdge = 1 // Read live session from the current point (skip past data) } ``` **LiveReadingType Details:** - **FromBeginning** (default): When reading a live session, start from the beginning and process all available data, then continue with live data as it arrives. This ensures you get the complete session history. - **LeadingEdge**: When reading a live session, skip any historical data and start reading from the current point forward. This is useful for real-time monitoring where you only care about the latest data and want to minimize latency. Also known as "tail mode" or "live-only mode". **Use Cases:** - Use `FromBeginning` when you need the complete session data (e.g., analysis, visualization) - Use `LeadingEdge` when you only need real-time updates (e.g., live dashboards, alerts) **Note**: `LiveReadingType` only applies when `ReadingType` is set to `Live`. It has no effect for historical session reading. ## Methods ### AddHandler Adds a packet handler to process received packets. ```csharp public void AddHandler(IHandler handler) ``` **Example:** ```csharp public class MyPacketHandler : IHandler { public void Handle(IReceivedPacketDto receivedPacket) { var packet = receivedPacket.Packet; Console.WriteLine($"Received: {packet.Type} (ID: {packet.Id})"); // Process based on packet type switch (packet.Type) { case "PeriodicData": HandlePeriodicData(packet); break; case "Event": HandleEvent(packet); break; } } private void HandlePeriodicData(Packet packet) { var periodicData = PeriodicDataPacket.Parser.ParseFrom(packet.Content); // Process periodic data... } private void HandleEvent(Packet packet) { var eventData = EventPacket.Parser.ParseFrom(packet.Content); // Process event... } } // Usage var handler = new MyPacketHandler(); readerService.AddHandler(handler); ``` ### RemoveHandler Removes a previously added handler. ```csharp public void RemoveHandler(IHandler handler) ``` **Example:** ```csharp readerService.RemoveHandler(handler); ``` ### Initialise and Start Initialize and start the reader service. ```csharp readerService.Initialise(); readerService.Start(); ``` **Note**: - `Initialise()` prepares the service - `Start()` begins reading packets - For auto-start readers, packets begin arriving immediately after `Start()` - For manual-start readers, `Start()` waits for the session to begin ### Stop Stops the reader service. ```csharp readerService.Stop(); ``` ## Events ### SessionReadingStarted Fired when reading of a session begins. ```csharp readerService.SessionReadingStarted += (sender, sessionInfo) => { Console.WriteLine($"Started reading session: {sessionInfo.SessionKey}"); Console.WriteLine($"Data Source: {sessionInfo.DataSource}"); Console.WriteLine($"Session Type: {sessionInfo.Type}"); }; ``` ### SessionReadingCompleted Fired when reading of a session is complete. ```csharp readerService.SessionReadingCompleted += (sender, sessionInfo) => { Console.WriteLine($"Completed reading session: {sessionInfo.SessionKey}"); Console.WriteLine($"Historical: {sessionInfo.Historical}"); }; ``` ### StreamReadingStarted Fired when reading of a stream begins. ```csharp readerService.StreamReadingStarted += (sender, streamInfo) => { Console.WriteLine($"Started reading stream: {streamInfo.Stream}"); Console.WriteLine($"Session: {streamInfo.SessionKey}"); }; ``` ### StreamReadingCompleted Fired when reading of a stream is complete. ```csharp readerService.StreamReadingCompleted += (sender, streamInfo) => { Console.WriteLine($"Completed reading stream: {streamInfo.Stream}"); }; ``` ### SessionInfoUpdated Fired when session information changes during reading. ```csharp readerService.SessionInfoUpdated += (sender, sessionInfo) => { Console.WriteLine($"Session info updated: {sessionInfo.SessionKey}"); Console.WriteLine($"Details: {sessionInfo.Details.Count}"); }; ``` ### SessionAssociationInfoUpdated Fired when session association information is detected. ```csharp readerService.SessionAssociationInfoUpdated += (sender, associationInfo) => { Console.WriteLine($"Association detected:"); Console.WriteLine($" Main: {associationInfo.MainSessionInfo.SessionKey}"); Console.WriteLine($" Associate: {associationInfo.AssociateSessionInfo.SessionKey}"); }; ``` ### CoverageCursorReceived Fired when a coverage cursor packet is received. ```csharp readerService.CoverageCursorReceived += (sender, coverageCursor) => { Console.WriteLine($"Coverage cursor received:"); Console.WriteLine($" Data Source: {coverageCursor.DataSource}"); Console.WriteLine($" Time: {coverageCursor.CoverageCursorTime}"); }; ``` **Coverage Cursor**: Indicates the latest timestamp where all of the data has been received. Useful for post processing data in real time. ## Complete Examples ### Example 1: Read Historical Session ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.ReadingModule.Abstractions; using MA.Streaming.OpenData; public class HistoricalSessionReader { private readonly IPacketReaderService readerService; private readonly AutoResetEvent completionEvent = new(false); private int packetCount = 0; public HistoricalSessionReader(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { var readerApi = supportLibApi.GetReadingPacketApi(); // Create reader for specific session var result = readerApi.CreateService(dataSource, sessionKey); this.readerService = result.Data; // Subscribe to events this.readerService.SessionReadingStarted += OnReadingStarted; this.readerService.SessionReadingCompleted += OnReadingCompleted; // Add packet handler this.readerService.AddHandler(new PacketCounter(this)); // Initialize this.readerService.Initialise(); } public void ReadSession() { Console.WriteLine("Starting to read historical session..."); readerService.Start(); // Wait for completion completionEvent.WaitOne(TimeSpan.FromMinutes(5)); Console.WriteLine($"Total packets read: {packetCount}"); } private void OnReadingStarted(object sender, ISessionInfo session) { Console.WriteLine($"Reading started: {session.Identifier}"); } private void OnReadingCompleted(object sender, ISessionInfo session) { Console.WriteLine($"Reading completed: {session.Identifier}"); completionEvent.Set(); } private class PacketCounter : IHandler { private readonly HistoricalSessionReader parent; public PacketCounter(HistoricalSessionReader parent) { this.parent = parent; } public void Handle(IReceivedPacketDto receivedPacket) { parent.packetCount++; if (parent.packetCount % 1000 == 0) { Console.WriteLine($"Processed {parent.packetCount} packets..."); } } } } ``` ### Example 2: Monitor Live Session ```csharp public class LiveSessionMonitor { private readonly IPacketReaderService readerService; public LiveSessionMonitor( ISupportLibApi supportLibApi, ISessionManagementService sessionService, string dataSource) { // Subscribe to session started event sessionService.LiveSessionStarted += (sender, session) => { if (session.DataSource == dataSource) { StartReadingLiveSession(supportLibApi, session); } }; } private void StartReadingLiveSession(ISupportLibApi supportLibApi, ISessionInfo session) { Console.WriteLine($"New live session detected: {session.SessionKey}"); var readerApi = supportLibApi.GetReadingPacketApi(); // Create reader for the live session var config = new PacketReadingConfiguration( dataSource: session.DataSource, sessionKey: session.SessionKey, readingType: ReadingType.Live, streams: new[] { "Telemetry", "Events" }, // Only read specific streams inactivityTimeoutSeconds: 60); var result = readerApi.CreateService(config); var readerService = result.Data; // Add handlers readerService.AddHandler(new LiveDataHandler()); // Subscribe to events readerService.SessionReadingCompleted += (sender, sessionInfo) => { Console.WriteLine($"Live session ended: {sessionInfo.SessionKey}"); readerService.Stop(); }; // Start reading readerService.Initialise(); readerService.Start(); } private class LiveDataHandler : IHandler { public void Handle(IReceivedPacketDto packet) { // Process live data in real-time Console.WriteLine($"[{DateTime.UtcNow:HH:mm:ss.fff}] {packet.Packet.Type}"); } } } ``` ### Example 3: Read All Sessions with Pattern ```csharp public class SessionPatternReader { public void ReadAllQualifyingSessions(ISupportLibApi supportLibApi) { var readerApi = supportLibApi.GetReadingPacketApi(); // Read all historical qualifying sessions var config = new PacketReadingConfiguration( dataSource: "RaceCar", sessionIdentifierPattern: "Qualifying_*", // Pattern matching readingType: ReadingType.Both, inactivityTimeoutSeconds: 30); var result = readerApi.CreateService(config); var readerService = result.Data; var sessionPackets = new Dictionary>(); // Track packets by session readerService.SessionReadingStarted += (sender, session) => { sessionPackets[session.SessionKey] = new List(); Console.WriteLine($"Reading session: {session.Identifier}"); }; // Collect packets readerService.AddHandler(new SessionPacketCollector(sessionPackets)); readerService.SessionReadingCompleted += (sender, session) => { var count = sessionPackets[session.SessionKey].Count; Console.WriteLine($"Session {session.Identifier} had {count} packets"); }; readerService.Initialise(); readerService.Start(); // Wait for completion... } private class SessionPacketCollector : IHandler { private readonly Dictionary> packets; public SessionPacketCollector(Dictionary> packets) { this.packets = packets; } public void Handle(IReceivedPacketDto packet) { var sessionKey = packet.Packet.SessionKey; if (packets.ContainsKey(sessionKey)) { packets[sessionKey].Add(packet); } } } } ``` ### Example 4: Leading Edge Live Reading ```csharp public class RealTimeDashboard { public void MonitorLiveSession(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { var readerApi = supportLibApi.GetReadingPacketApi(); // Configure for leading edge reading - only new data var config = new PacketReadingConfiguration( dataSource: dataSource, sessionKey: sessionKey, readingType: ReadingType.Live, liveReadingType: LiveReadingType.LeadingEdge, // Skip historical data inactivityTimeoutSeconds: 60); var result = readerApi.CreateService(config); var readerService = result.Data; readerService.SessionReadingStarted += (sender, session) => { Console.WriteLine($"Started monitoring live session from current point"); }; // Process only new live data readerService.AddHandler(new RealtimeDataHandler()); readerService.Initialise(); readerService.Start(); } private class RealtimeDataHandler : IHandler { private DateTime lastUpdate = DateTime.UtcNow; public void Handle(IReceivedPacketDto packet) { var latency = DateTime.UtcNow - packet.SubmitTime; Console.WriteLine($"[{DateTime.UtcNow:HH:mm:ss.fff}] Packet: {packet.Packet.Type}, Latency: {latency.TotalMilliseconds}ms"); lastUpdate = DateTime.UtcNow; } } } ``` ### Example 5: Bounded Buffer for High-Volume Sessions ```csharp public class HighVolumeSessionReader { public void ReadHighVolumeSession(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { var readerApi = supportLibApi.GetReadingPacketApi(); // Configure with bounded buffer to control memory usage var config = new PacketReadingConfiguration( dataSource: dataSource, sessionKey: sessionKey, readingType: ReadingType.Historical, bufferLength: 10000, // Limit buffer to 10,000 packets inactivityTimeoutSeconds: 30); var result = readerApi.CreateService(config); var readerService = result.Data; Console.WriteLine("Reading high-volume session with bounded buffer..."); readerService.AddHandler(new BatchProcessor()); readerService.Initialise(); readerService.Start(); } private class BatchProcessor : IHandler { private readonly List batch = new(); private const int BatchSize = 1000; public void Handle(IReceivedPacketDto packet) { batch.Add(packet); if (batch.Count >= BatchSize) { ProcessBatch(batch); batch.Clear(); } } private void ProcessBatch(List packets) { Console.WriteLine($"Processing batch of {packets.Count} packets"); // Batch processing logic here } } } ``` ### Example 6: Using Consumer Groups ```csharp public class DistributedProcessing { public void SetupDistributedReaders(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { // Create multiple readers in the same consumer group for load balancing var groupId = "processing-cluster-1"; var readers = new List(); for (int i = 0; i < 3; i++) { var config = new PacketReadingConfiguration( dataSource: dataSource, sessionKey: sessionKey, readingType: ReadingType.Historical, groupId: groupId, // Same group ID for load balancing inactivityTimeoutSeconds: 30); var readerApi = supportLibApi.GetReadingPacketApi(); var result = readerApi.CreateService(config); var reader = result.Data; reader.AddHandler(new DistributedHandler(i)); reader.Initialise(); readers.Add(reader); Console.WriteLine($"Reader {i} created in consumer group '{groupId}'"); } // Start all readers - they will share the workload foreach (var reader in readers) { reader.Start(); } Console.WriteLine("Distributed processing started - readers sharing workload via consumer group"); } private class DistributedHandler : IHandler { private readonly int readerId; private int packetCount = 0; public DistributedHandler(int readerId) { this.readerId = readerId; } public void Handle(IReceivedPacketDto packet) { packetCount++; if (packetCount % 100 == 0) { Console.WriteLine($"Reader {readerId}: Processed {packetCount} packets"); } } } } ``` ### Example 7: Resuming from Last Position ```csharp public class ResumableSessionReader { private const string CustomGroupId = "my-data-processor"; public void StartOrResumeSession(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { var readerApi = supportLibApi.GetReadingPacketApi(); // Create a stable group ID combining custom group identifier and session key // This allows resuming from the last committed position var groupId = $"{CustomGroupId}_{sessionKey}"; var config = new PacketReadingConfiguration( dataSource: dataSource, sessionKey: sessionKey, readingType: ReadingType.Live, // Use Live mode to enable resume behavior liveReadingType = LiveReadingType.LeadingEdge, groupId: groupId, // Consistent group ID for resumption inactivityTimeoutSeconds: 60); var result = readerApi.CreateService(config); var readerService = result.Data; Console.WriteLine($"Starting reader with group ID: {groupId}"); Console.WriteLine("If this is a restart, reading will resume from last committed position"); readerService.AddHandler(new CheckpointHandler()); readerService.SessionReadingStarted += (sender, session) => { Console.WriteLine($"Session reading started/resumed: {session.SessionKey}"); }; readerService.Initialise(); readerService.Start(); // When this reader restarts with the same groupId, it will automatically // resume from where it left off, thanks to Kafka consumer group offsets } private class CheckpointHandler : IHandler { private int processedCount = 0; public void Handle(IReceivedPacketDto packet) { processedCount++; // Process the packet if (processedCount % 1000 == 0) { Console.WriteLine($"Processed {processedCount} packets (position automatically tracked by consumer group)"); } } } } ``` ## Best Practices 1. **Subscribe to Events First**: Attach event handlers before calling `Start()` to capture all events. 2. **Handle All Packet Types**: Implement handlers that can process different packet types gracefully. 3. **Use Configuration for Flexibility**: Use `PacketReadingConfiguration` for complex scenarios with multiple options. 4. **Stream Filtering**: Filter streams when you only need specific data to reduce processing overhead. 5. **Inactivity Timeout**: Set appropriate timeout values based on expected data frequency. 6. **Resource Cleanup**: Always call `Stop()` when done to release resources. 7. **Thread Safety**: Handlers may be called from different threads; ensure thread-safe processing. 8. **Error Handling in Handlers**: Wrap handler logic in try-catch to prevent crashes from processing errors. 9. **Choose Appropriate LiveReadingType**: - Use `FromBeginning` for complete session analysis and when historical context is needed - Use `LeadingEdge` for real-time dashboards and alerts where only current data matters - `LeadingEdge` reduces latency and memory usage for live monitoring scenarios 10. **Buffer Size Configuration**: - Use bounded buffers (`bufferLength > 0`) for high-volume sessions to prevent memory pressure - Use unbounded buffers (default) for normal sessions where memory is not a concern - Monitor memory usage and adjust buffer size based on packet rate and processing speed - Ensure packet handlers are efficient to keep up with data flow; slow handlers with bounded buffers will cause backpressure 11. **Consumer Groups**: - Use unique `groupId` for each independent processing pipeline - Share `groupId` across readers only when you want load balancing/distributed processing - Leave `groupId` empty (default) for single-reader scenarios - Be aware that readers in the same group will split data - they won't all receive the same packets - Group ID naming: avoid spaces and follow Kafka naming principles (use alphanumeric characters, hyphens, underscores, and periods) - **Resuming from last position**: To rejoin a session and continue from where you left off, use `ReadingType.Live` with a consistent `groupId` pattern like `{customGroupId}_{sessionKey}`. This leverages Kafka consumer group offsets to resume from the last committed position. - **Warning**: Do not reuse the same static `groupId` across different sessions. Using the same consumer group ID for multiple sessions can lead to reading from incorrect offsets, missing data, or late reading as Kafka will resume from the last committed position which may be from a different session. Always include the session key or unique identifier in your group ID to avoid these issues. 12. **Resumable Live Reading**: - To implement fault-tolerant live reading that can resume after interruption, use `ReadingType.Live` with `LiveReadingType.LeadingEdge` and a session-specific `groupId` like `{customGroupId}_{sessionKey}` - This pattern is ideal for applications that need to process all data even after crashes or restarts - The reader will automatically resume from the last committed offset, avoiding data loss or duplicate processing - Ensure your application logic is idempotent if there's a possibility of receiving duplicate messages during recovery ## Common Patterns ### Buffered Handler ```csharp public class BufferedPacketHandler : IHandler { private readonly BlockingCollection buffer = new(); private readonly CancellationTokenSource cts = new(); public BufferedPacketHandler() { // Start processing thread Task.Run(() => ProcessBuffer()); } public void Handle(IReceivedPacketDto packet) { buffer.Add(packet); } private void ProcessBuffer() { foreach (var packet in buffer.GetConsumingEnumerable(cts.Token)) { // Process packet asynchronously ProcessPacket(packet); } } private void ProcessPacket(IReceivedPacketDto packet) { // Your processing logic } public void Stop() { buffer.CompleteAdding(); cts.Cancel(); } } ``` ### Type-Specific Handlers ```csharp public class TypeDispatchingHandler : IHandler { private readonly Dictionary> handlers = new(); public TypeDispatchingHandler() { handlers["PeriodicData"] = HandlePeriodicData; handlers["Event"] = HandleEventData; handlers["Marker"] = HandleMarkerData; } public void Handle(IReceivedPacketDto receivedPacket) { var packet = receivedPacket.Packet; if (handlers.TryGetValue(packet.Type, out var handler)) { handler(packet); } else { Console.WriteLine($"Unknown packet type: {packet.Type}"); } } private void HandlePeriodicData(Packet packet) { /* ... */ } private void HandleEventData(Packet packet) { /* ... */ } private void HandleMarkerData(Packet packet) { /* ... */ } } ``` ## See Also - [Session Manager Module](session-manager.md) - [Essentials Module](essentials-module.md) - Read essential/configuration packets only - [Writer Module](writer-module.md) - [Buffering Module](buffering-module.md) - [API Reference](api-reference.md) --- # Essential Module ## Overview The Essentials Module provides a streamlined API for reading essential session packets from the streaming broker. Essential packets contain critical session information such as configuration data, data format definitions, and other metadata required to properly interpret session data. This module is optimized for efficiently retrieving these foundational packets without the overhead of processing all session data. The lifecycle of the Essentials service is tied to the session lifecycle—it starts when the session begins and completes when all essential packets have been read or the session ends. ## Key Features - **Essential Packet Reading**: Read only essential/configuration packets from sessions - **Lightweight Processing**: Optimized for reading metadata without full data streams - **Event System**: Track when essential data reading starts and completes - **Handler Pattern**: Attach custom handlers to process essential packets - **Live and Historical Support**: Works with both live and historical sessions - **Stream Management**: Automatically manages connection to the Essential stream ## Use Cases The Essentials Module is ideal for scenarios where you need to: - Retrieve session configuration data before processing full telemetry - Read session configuration and settings - Access essential session metadata and setup information - Validate session structure before initiating full data processing - Build session indexes or catalogs ## API Access ### Get the Essentials API ```csharp var essentialsApi = supportLibApi.GetEssentialsApi(); ``` ### Create an Essentials Service ```csharp var serviceResult = essentialsApi.CreateService( dataSource: "CarA", sessionKey: "specific-session-key"); if (serviceResult.Success && serviceResult.Data != null) { var essentialsService = serviceResult.Data; } ``` **Parameters:** - **dataSource**: The data source to read from (e.g., "CarA", "CarB") - **sessionKey**: The specific session key to read essential packets from ## Methods ### AddHandler Adds a packet handler to process received essential packets. ```csharp public void AddHandler(IHandler handler) ``` **Example:** ```csharp public class EssentialPacketHandler : IHandler { public void Handle(IReceivedPacketDto receivedPacket) { var packet = receivedPacket.Packet; Console.WriteLine($"Essential Packet: {packet.Type} (ID: {packet.Id})"); Console.WriteLine($" Session: {receivedPacket.SessionKey}"); Console.WriteLine($" Data Source: {receivedPacket.DataSource}"); Console.WriteLine($" Submit Time: {receivedPacket.SubmitTime}"); Console.WriteLine($" Stream: {receivedPacket.Stream}"); // Process based on packet type switch (packet.Type) { case "Configuration": HandleConfiguration(packet); break; case "Marker": HandleMarker(packet); break; } } private void HandleConfiguration(Packet packet) { // Parse and process configuration packet var configData = ConfigurationPacket.Parser.ParseFrom(packet.Content); Console.WriteLine($"Configuration"); } private void HandleMarker(Packet packet) { // Parse and process marker packet var markerData = MarkerPacket.Parser.ParseFrom(packet.Content); Console.WriteLine($"Marker"); } } // Usage var handler = new EssentialPacketHandler(); essentialsService.AddHandler(handler); ``` ### RemoveHandler Removes a previously added handler. ```csharp public void RemoveHandler(IHandler handler) ``` **Example:** ```csharp essentialsService.RemoveHandler(handler); ``` ### Initialise and Start Initialize and start the essentials service. ```csharp essentialsService.Initialise(); essentialsService.Start(); ``` **Note**: - `Initialise()` prepares the service and establishes connections - `Start()` begins reading essential packets - Essential packets are buffered and delivered to handlers asynchronously ### Stop Stops the essentials service. ```csharp essentialsService.Stop(); ``` ## Events ### ReadingEssentialsStarted Fired when reading of essential packets begins for a session. ```csharp essentialsService.ReadingEssentialsStarted += (sender, sessionInfo) => { Console.WriteLine($"Started reading essentials for session: {sessionInfo.SessionKey}"); Console.WriteLine($"Data Source: {sessionInfo.DataSource}"); Console.WriteLine($"Session Type: {sessionInfo.Type}"); Console.WriteLine($"Historical: {sessionInfo.Historical}"); }; ``` ### ReadingEssentialsCompleted Fired when reading of essential packets is complete. ```csharp essentialsService.ReadingEssentialsCompleted += (sender, sessionInfo) => { Console.WriteLine($"Completed reading essentials for session: {sessionInfo.SessionKey}"); Console.WriteLine($"Data Source: {sessionInfo.DataSource}"); }; ``` ## Complete Example ### Reading Essential Packets from a Session ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core; using MA.DataPlatforms.Streaming.Support.Lib.Core.Shared.Abstractions; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.ReadingModule.Abstractions; using MA.Streaming.OpenData; // Create the Support Library API var config = new StreamApiConfiguration( kafkaConnection: "localhost:9092"); var supportLibApi = SupportLibApiFactory.CreateSupportLibApi(config); // Get the Essentials API var essentialsApi = supportLibApi.GetEssentialsApi(); // Create handler for essential packets var essentialHandler = new EssentialPacketHandler(); // Create the essentials service var serviceResult = essentialsApi.CreateService( dataSource: "CarA", sessionKey: "session-2026-02-26-001"); if (serviceResult.Success && serviceResult.Data != null) { var essentialsService = serviceResult.Data; // Subscribe to events essentialsService.ReadingEssentialsStarted += (s, sessionInfo) => { Console.WriteLine($"Started reading essentials: {sessionInfo.SessionKey}"); }; essentialsService.ReadingEssentialsCompleted += (s, sessionInfo) => { Console.WriteLine($"Completed reading essentials: {sessionInfo.SessionKey}"); }; // Add handler essentialsService.AddHandler(essentialHandler); // Initialize and start essentialsService.Initialise(); essentialsService.Start(); // Wait for completion or stop manually Console.WriteLine("Reading essential packets... Press any key to stop."); Console.ReadKey(); // Stop the service essentialsService.Stop(); } else { Console.WriteLine($"Failed to create essentials service: {serviceResult.Message}"); } ``` ## Architecture and Design ### Essential Stream The Essentials Module reads from a dedicated "Essential" stream that contains: - **Configuration Packets**: Session configuration and settings - **Marker Packets**: Stream control and session markers ### Buffering Strategy The module uses an unbounded auto-start buffer to handle essential packets efficiently: - **Auto-Start Buffer**: Automatically begins processing when packets arrive - **Unbounded**: No memory pressure constraints since essential packets are typically small - **Asynchronous Processing**: Packets are buffered and processed asynchronously ### Lifecycle 1. **Service Creation**: Creates connection to the Essential stream 2. **Initialization**: Establishes connections and prepares handlers 3. **Start**: Begins reading from the Essential stream 4. **Packet Processing**: Essential packets are received and dispatched to handlers 5. **Completion**: Automatically detects when all essential packets are read 6. **Stop**: Cleans up resources and disconnections ## Best Practices ### 1. Use Before Full Data Processing (Historic Sessions Only) Read essential packets before processing full session data to understand the session structure. **Note: This pattern applies to historic sessions only**, where all essentials are available upfront. For live sessions, essentials may still be arriving as the session progresses. ```csharp // Step 1: Read essentials to get data formats var essentialsService = essentialsApi.CreateService(dataSource, sessionKey).Data; var formatCollector = new DataFormatCollector(); essentialsService.AddHandler(formatCollector); essentialsService.Initialise(); essentialsService.Start(); // Wait for essentials to complete... await formatCollector.WaitForCompletion(); // Step 2: Now process full data with format knowledge var readerService = readerApi.CreateService(dataSource, sessionKey).Data; // ... process full session data ``` ### 2. Handle Multiple Session Essentials You can create multiple essentials services for different sessions: ```csharp var sessions = new[] { "session-001", "session-002", "session-003" }; var essentialsServices = sessions.Select(sessionKey => { var service = essentialsApi.CreateService(dataSource, sessionKey).Data; service.AddHandler(sharedHandler); service.Initialise(); return service; }).ToList(); // Start all services essentialsServices.ForEach(s => s.Start()); // Wait for all to complete... ``` ### 3. Error Handling Always check API results and handle errors appropriately: ```csharp var result = essentialsApi.CreateService(dataSource, sessionKey); if (!result.Success || result.Data == null) { logger.Error($"Failed to create essentials service: {result.Message}"); // Handle error - session might not exist or be accessible return; } var service = result.Data; // Proceed with service usage... ``` ### 4. Combine with Session Manager Use with the Session Manager module to discover and process sessions: ```csharp var sessionManager = supportLibApi.GetSessionManagerApi(); var sessions = sessionManager.GetSessions(dataSource).Data; foreach (var session in sessions) { var essentialsService = essentialsApi.CreateService( session.DataSource, session.SessionKey).Data; // Process essential packets for each session essentialsService.AddHandler(essentialHandler); essentialsService.Initialise(); essentialsService.Start(); } ``` ## Differences from Reader Module | Feature | Essentials Module | Reader Module | |---------|-------------------|---------------| | **Purpose** | Read essential/configuration packets only | Read all data packets | | **Stream** | Reads from "Essential" stream | Reads from all session streams | | **Data Volume** | Low (configuration data only) | High (all telemetry data) | | **Use Case** | Metadata and configuration retrieval | Full session data processing | | **Configuration** | Simple (data source + session key) | Complex (streams, filters, timeouts) | | **Performance** | Lightweight and fast | Heavier, processes all data | ## Python FFI Support The Essentials Module is also available through the Python FFI library: ```python from contracts.essentials.i_essentials_service_module_api import IEssentialsServiceModuleApi from core.essentials.essentials_service_module_api import EssentialsServiceModuleApi # Get essentials API essentials_api = support_lib_api.get_essentials_api() # Create service essentials_service = essentials_api.create_service(data_source, session_key) # Add handler essentials_service.add_handler(packet_handler) # Initialize and start essentials_service.initialise() essentials_service.start() ``` ## Related Modules - **[Session Manager](session-manager.md)**: Discover sessions before reading essentials - **[Reader Module](reader-module.md)**: Read full session data after essentials - **[Data Format Manager](data-format-manager.md)**: Define formats that appear in essentials --- # Buffering Module ## Overview The Buffering Module, also known as the Sample Reader Module, is a component that pipelines with the Packet Reader Module. It buffers incoming packets, merges data segments, and extracts time-aligned samples from the raw packet stream. This module transforms packet-level data into sample-level data suitable for analysis and visualization. ## Key Features - **Packet Buffering**: Buffer incoming packets in configurable time windows - **Data Merging**: Merge data from multiple packets into coherent samples - **Sample Extraction**: Extract time-aligned parameter samples - **Sliding Window**: Process data with configurable sliding windows - **Parameter Subscription**: Subscribe to specific parameters or all parameters - **Dynamic Subscription**: Add/remove parameter subscriptions on-the-fly - **Multiple Merge Strategies**: Support different merging approaches - **Event Filtering**: Include/exclude markers, events, errors, and CAN data - **Pipeline Architecture**: Seamlessly connects to Packet Reader output ## API Access ### Get the Sample Reader API ```csharp var sampleReaderApi = supportLibApi.GetSampleReaderApi(); ``` ### Create a Sample Reader Service ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.BufferingModule; var config = new BufferingConfiguration( subscribedParameters: new[] { "Speed", "RPM", "Throttle" }, bufferingWindowLength: 3000, // 3 seconds slidingWindowPercentage: 5, // 5% slide mergeStrategy: MergeStrategy.ReadingMerge, // better to use reading merge for performance includeMarkerData: false, includeEventData: false, includeErrorData: false, includeCanData: false); var result = sampleReaderApi.CreateService(config); if (result.Success && result.Data != null) { var sampleReaderService = result.Data; } ``` ## Configuration Options ### BufferingConfiguration ```csharp public BufferingConfiguration( IReadOnlyList? subscribedParameters = null, uint? bufferingWindowLength = 3000, uint? slidingWindowPercentage = 5, MergeStrategy? mergeStrategy = null, bool? includeMarkerData = null, bool? includeEventData = null, bool? includeErrorData = null, bool? includeCanData = null, bool? allowWildcardMatch = null) ``` **Parameters:** - **subscribedParameters**: List of parameter identifiers to subscribe to - `null` or empty list = no parameters initially subscribed - Can subscribe/unsubscribe dynamically later - Parameter names must match those defined in Data Format Manager - Entries may contain the `*` wildcard, or regex syntax, when `allowWildcardMatch` is enabled — see [Wildcard and regex parameter matching](#wildcard-and-regex-parameter-matching) - **bufferingWindowLength**: Length of buffering window in milliseconds (default: 3000ms = 3 seconds) - Larger windows provide more context but increase latency - Smaller windows reduce latency but may miss data correlations - **slidingWindowPercentage**: Percentage of window to advance on each slide (default: 5%) - Range: 1-100 - 5% = window slides by 5% of its length each iteration - 100% = window jumps by full length (no overlap) - Lower percentage = more overlap, smoother results, higher processing - **mergeStrategy**: Strategy for merging data segments - `MergeStrategy.ReadingMerge`: Default strategy optimized for reading - Other strategies may be available based on implementation - **includeMarkerData**: Include marker packets in buffering (default: false) - **includeEventData**: Include event packets in buffering (default: false) - **includeErrorData**: Include error packets in buffering (default: false) - **includeCanData**: Include CAN bus data in buffering (default: false) - **allowWildcardMatch**: Allow partial parameter matching using the `*` wildcard, or full regex syntax, in subscribed parameter names (default: true) — see [Wildcard and regex parameter matching](#wildcard-and-regex-parameter-matching) ### Wildcard and regex parameter matching By default (`allowWildcardMatch: true`), entries in `subscribedParameters` — and identifiers passed to `Subscribe` at runtime — are matched using more than plain equality. Which matching strategy applies depends on what the entry looks like: - **Plain identifiers** (no `*` and none of the regex characters below) are compared to the incoming identifier case-insensitively. - **`*` wildcard patterns** match any sequence of characters, and are compared case-sensitively. - **Patterns containing regex syntax** — any of `. $ ^ { [ ( | ) + ? \` — are compiled and matched as a real, case-sensitive regular expression, anchored to the whole identifier. This is the slowest of the three, so only reach for it when a `*` wildcard can't express what you need. ```csharp var config = new BufferingConfiguration( subscribedParameters: new[] { "Speed", // exact match, case-insensitive "Brake*", // matches every parameter identifier starting with "Brake" "Wheel[0-9]+", // regex fallback: matches Wheel0, Wheel12, ... }); ``` Matching rules: - A pattern must match the **whole** incoming parameter identifier — `Brake*` matches `BrakeTemp` but `rake*` does not match `BrakeTemp`. - Entries without `*` or regex syntax behave as an exact match, but that exact match is case-insensitive. - Entries with `*` or regex syntax are matched case-sensitively. !!! warning "Regex characters in identifiers are not escaped" If a subscribed identifier contains a character from `. $ ^ { [ ( | ) + ? \`, it is compiled as regex syntax rather than treated as a literal character. For example `Speed.Front` is not matched literally — the `.` matches any single character. If your naming convention uses these characters as literal text, avoid combining them with `*`/regex-style patterns, or escape them yourself before passing them in. Set `allowWildcardMatch: false` to treat every subscription as a literal identifier compared case-sensitively, including any that contain `*` or regex syntax. ```csharp public enum MergeStrategy { WritingMerge, ReadingMerge // Optimized for sequential reading and merging } ``` ## Methods ### SetReaderService Connects the sample reader to a packet reader service. ```csharp public void SetReaderService(IPacketReaderService readingService) ``` **Example:** ```csharp // Create packet reader var packetReaderService = readerApi.CreateService(dataSource, sessionKey).Data; // Create sample reader var sampleReaderService = sampleReaderApi.CreateService(bufferingConfig).Data; // Pipeline them together sampleReaderService.SetReaderService(packetReaderService); // Initialize both packetReaderService.Initialise(); sampleReaderService.Initialise(); // Start both (order matters: start packet reader last) sampleReaderService.Start(); packetReaderService.Start(); ``` ### AddHandler (Timestamp Data) Adds a handler for timestamp data (merged samples). ```csharp public void AddHandler(IHandler timestampHandler) ``` **Example:** ```csharp public class TimestampDataHandler : IHandler { public void Handle(ITimestampData data) { Console.WriteLine($"Start Timestamp: {data.TimeColumns[0].Timestamp}"); Console.WriteLine($"Data Source: {data.DataSource}"); Console.WriteLine($"Session: {data.SessionKey}"); foreach (var column in data.TimeColumns) { Console.WriteLine($" Timestamp: {column.Timestamp}"); Console.WriteLine($" Samples: {column.SampleValues.Count}"); foreach (var sample in column.SampleValues) { Console.WriteLine($" Parameter Identifier:{sample.Identifier} Value: {sample.Value} at {column.Timestamp}"); } } // Check if this is the end marker if (data is EndTimestampData) { Console.WriteLine("End of session data"); } } } var handler = new TimestampDataHandler(); sampleReaderService.AddHandler(handler); ``` ### AddHandler (Sample Data) Adds a handler for sample data (alternative data format). It contains all values for each parameter in that reading range, with values ordered per parameter but not time-aligned across parameters. ```csharp public void AddHandler(IHandler sampleHandler) ``` ### RemoveHandler Removes previously added handlers. ```csharp public void RemoveHandler(IHandler timestampHandler); public void RemoveHandler(IHandler sampleHandler); ``` ### Subscribe Subscribes to specific parameters. ```csharp public void Subscribe(IEnumerable parameterIdentifiers) ``` **Example:** ```csharp // Subscribe to additional parameters sampleReaderService.Subscribe(new[] { "BrakePressure", "SteeringAngle" }); ``` **Note**: Can be called before or during data processing for dynamic subscription. There is a limitation on the number of parameters that can be merged simultaneously. To maintain optimal live processing performance, avoid subscribing to an excessive number of parameters, as merging operations may be constrained by system resources and processing capabilities. ### Unsubscribe Unsubscribes from specific parameters. ```csharp public void Unsubscribe(IEnumerable parameterIdentifiers) ``` **Example:** ```csharp sampleReaderService.Unsubscribe(new[] { "Throttle" }); ``` ### UnsubscribeAll Unsubscribes from all parameters. ```csharp public void UnsubscribeAll() ``` ### GetLastTimeInfo Gets the last timestamp information for each stream. ```csharp public IReadOnlyList<(string Stream, ulong LastTimestamp)> GetLastTimeInfo( string dataSource, string sessionKey) ``` **Example:** ```csharp var timeInfo = sampleReaderService.GetLastTimeInfo(dataSource, sessionKey); foreach (var (stream, lastTimestamp) in timeInfo) { Console.WriteLine($"Stream: {stream}, Last Timestamp: {lastTimestamp}µs"); } ``` ## Events ### SessionDataMergingStarted Fired when sample merging starts for a session. ```csharp sampleReaderService.SessionDataMergingStarted += (sender, sessionInfo) => { Console.WriteLine($"Started merging data for session: {sessionInfo.SessionKey}"); Console.WriteLine($"Session Type: {sessionInfo.Type}"); }; ``` ### SessionDataMergingCompleted Fired when sample merging completes for a session. ```csharp sampleReaderService.SessionDataMergingCompleted += (sender, sessionInfo) => { Console.WriteLine($"Completed merging data for session: {sessionInfo.SessionKey}"); Console.WriteLine($"Historical: {sessionInfo.Historical}"); }; ``` ## Data Structures ### ITimestampData Represents merged sample data at a specific timestamp. ```csharp public interface ITimestampData { string DataSource { get; } string SessionKey { get; } ulong StartTime { get; } ulong EndTime { get; } IReadOnlyList TimeColumns { get; } } ``` ### ITimeColumn Represents sample values for a single parameter. ```csharp public interface ITimestampColumn { ulong Timestamp { get; } IReadOnlyList SampleValues { get; } } ``` ### ISampleValue Represents a single sample value. ```csharp public interface ISampleValue { string Identifier { get; } IGenericValue Value { get; } ValueStatus Status { get; } Type ValueType { get; } T? Cast(); } ``` ### EndTimestampData Special marker indicating end of session data and merging. ```csharp if (data is EndTimestampData) { // Session data processing complete } ``` ## Complete Examples ### Example 1: Basic Buffering Pipeline ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.BufferingModule; using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.BufferingModule.Abstractions; public class BasicBufferingExample { public void SetupBufferingPipeline( ISupportLibApi supportLibApi, string dataSource, string sessionKey) { // 1. Create packet reader var readerApi = supportLibApi.GetReadingPacketApi(); var packetReaderResult = readerApi.CreateService(dataSource, sessionKey); var packetReaderService = packetReaderResult.Data; // 2. Create sample reader with configuration var sampleReaderApi = supportLibApi.GetSampleReaderApi(); var bufferingConfig = new BufferingConfiguration( subscribedParameters: new[] { "Speed", "RPM", "Throttle", "BrakePressure" }, bufferingWindowLength: 5000, // 5 second window slidingWindowPercentage: 10); // 10% slide var sampleReaderResult = sampleReaderApi.CreateService(bufferingConfig); var sampleReaderService = sampleReaderResult.Data; // 3. Connect them sampleReaderService.SetReaderService(packetReaderService); // 4. Add handler sampleReaderService.AddHandler(new SampleProcessor()); // 5. Subscribe to events sampleReaderService.SessionDataMergingStarted += (s, session) => { Console.WriteLine($"Merging started for {session.SessionKey}"); }; sampleReaderService.SessionDataMergingCompleted += (s, session) => { Console.WriteLine($"Merging completed for {session.SessionKey}"); }; // 6. Initialize and start packetReaderService.Initialise(); sampleReaderService.Initialise(); sampleReaderService.Start(); packetReaderService.Start(); // Start packet reader last // Wait for completion... } private class SampleProcessor : IHandler { private int sampleCount = 0; public void Handle(ITimestampData data) { if (data is EndTimestampData) { Console.WriteLine($"Total samples processed: {this.sampleCount}"); return; } foreach (var column in data.TimeColumns) { this.sampleCount += column.SampleValues.Count; Console.WriteLine($"#Sample {column.SampleValues.Count} at {column.Timestamp}"); } } } } ``` ### Example 2: Dynamic On-the-Fly Subscription ```csharp public class DynamicSubscriptionExample { private ISampleReaderService sampleReaderService; private HashSet currentSubscriptions = new(); public void SetupDynamicSubscription(ISupportLibApi supportLibApi) { // Create with no initial subscriptions var sampleReaderApi = supportLibApi.GetSampleReaderApi(); var config = new BufferingConfiguration(); // Empty subscription list var result = sampleReaderApi.CreateService(config); sampleReaderService = result.Data; // Add handler sampleReaderService.AddHandler(new DynamicSampleHandler()); // Start with no subscriptions, add them dynamically Task.Run(() => ManageSubscriptionsAsync()); } private async Task ManageSubscriptionsAsync() { await Task.Delay(1000); // Subscribe to initial set Console.WriteLine("Subscribing to Speed and RPM"); SubscribeTo(new[] { "Speed", "RPM" }); await Task.Delay(5000); // Add more parameters Console.WriteLine("Adding Throttle and BrakePressure"); SubscribeTo(new[] { "Throttle", "BrakePressure" }); await Task.Delay(5000); // Remove a parameter Console.WriteLine("Removing RPM"); UnsubscribeFrom(new[] { "RPM" }); } private void SubscribeTo(string[] parameters) { sampleReaderService.Subscribe(parameters); foreach (var param in parameters) { currentSubscriptions.Add(param); } } private void UnsubscribeFrom(string[] parameters) { sampleReaderService.Unsubscribe(parameters); foreach (var param in parameters) { currentSubscriptions.Remove(param); } } private class DynamicSampleHandler : IHandler { public void Handle(ITimestampData data) { if (data is EndTimestampData) return; var receivedParams = data.TimeColumns.SelectMany(c => c.SampleValues.Select(sv=>sv.Identifier)).ToList(); Console.WriteLine($"Received parameters: {string.Join(", ", receivedParams)}"); } } } ``` ### Example 3: Advanced Sample Analysis ```csharp public class SampleAnalyzer : IHandler { private readonly Dictionary> parameterValues = new(); private readonly Dictionary ranges = new(); public void Handle(ITimestampData data) { if (data is EndTimestampData) { PrintSummary(); return; } foreach (var column in data.TimeColumns) { foreach (var sample in column.SampleValues) { if (!parameterValues.ContainsKey(sample.Identifier)) { parameterValues[sample.Identifier] = []; ranges[sample.Identifier] = (double.MaxValue, double.MinValue); } if (sample.Value.ValueType==typeof(double)) { var value = sample.Value.Cast(); parameterValues[sample.Identifier].Add(value); var (min, max) = ranges[sample.Identifier]; ranges[sample.Identifier] = ( Math.Min(min, value), Math.Max(max, value) ); } } } } private void PrintSummary() { Console.WriteLine("\n=== Sample Analysis Summary ==="); foreach (var param in parameterValues.Keys) { var values = parameterValues[param]; var (min, max) = ranges[param]; var avg = values.Average(); Console.WriteLine($"\nParameter: {param}"); Console.WriteLine($" Samples: {values.Count}"); Console.WriteLine($" Min: {min:F2}"); Console.WriteLine($" Max: {max:F2}"); Console.WriteLine($" Avg: {avg:F2}"); } } } ``` ### Example 4: Buffering with Events and Markers ```csharp public void SetupWithEventsAndMarkers(ISupportLibApi supportLibApi) { var sampleReaderApi = supportLibApi.GetSampleReaderApi(); // Configure to include events and markers var config = new BufferingConfiguration( subscribedParameters: new[] { "Speed", "RPM" }, bufferingWindowLength: 3000, slidingWindowPercentage: 5, includeMarkerData: true, // Include markers includeEventData: true); // Include events var result = sampleReaderApi.CreateService(config); var sampleReaderService = result.Data; sampleReaderService.AddHandler(new SampleHandler()); // Setup pipeline and start... } ``` ## Working with Generic Values ### Extracting Values from IGenericValue Use the `ValueType` property to check the type, then use `Cast()` to extract the value: ```csharp public object? ExtractValue(IGenericValue genericValue) { if (genericValue.ValueType == typeof(double)) return genericValue.Cast(); if (genericValue.ValueType == typeof(MarkerData)) return genericValue.Cast(); if (genericValue.ValueType == typeof(EventData)) return genericValue.Cast(); if (genericValue.ValueType == typeof(ErrorData)) return genericValue.Cast(); if (genericValue.ValueType == typeof(CanData)) return genericValue.Cast(); return null; } ``` ## Best Practices 1. **Configure Window Size Appropriately**: - Larger windows for analysis requiring more context - Smaller windows for low-latency applications 2. **Optimize Sliding Percentage**: - Lower percentage (10-20%) for smooth, continuous analysis - Higher percentage (50-100%) for distinct chunks with less overlap 3. **Subscribe Selectively**: - Only subscribe to parameters you need to reduce memory and processing - Use dynamic subscription to adapt to changing requirements 4. **Pipeline Order Matters**: - Always initialize services before starting - Start sample reader before packet reader - Stop in reverse order 5. **Handle End Markers**: Always check for `EndTimestampData` to detect completion 6. **Thread Safety**: Handlers may be called from different threads. Create separate handler instances for each sample reader service rather than sharing a single handler instance across multiple services to avoid concurrency issues and ensure proper isolation of processing logic. 7. **Memory Management**: Unsubscribe from parameters you no longer need 8. **Event Filtering**: Only include events/markers/CAN data if needed ## Performance Tuning ### For Real-Time Processing ```csharp var realtimeConfig = new BufferingConfiguration( subscribedParameters: criticalParameters, bufferingWindowLength: 3000, // 3 second slidingWindowPercentage: 10, // 20% slide for lower latency includeMarkerData: false, includeEventData: false); ``` ### For Analysis/Playback ```csharp var analysisConfig = new BufferingConfiguration( subscribedParameters: null, // Subscribe dynamically bufferingWindowLength: 20000, // 20 seconds slidingWindowPercentage: 5, // 5% for maximum overlap includeMarkerData: true, includeEventData: true); ``` ## Common Issues and Solutions ### Issue: Missing Samples **Solution**: Increase `bufferingWindowLength` to capture more data in each window. ### Issue: High Latency **Solution**: Decrease `bufferingWindowLength` or increase `slidingWindowPercentage`. ### Issue: High Memory Usage **Solution**: Reduce subscribed parameters or decrease window length. ## See Also - [Reader Module](reader-module.md) - [Essentials Module](essentials-module.md) - [Interpolation Module](interpolation-module.md) - [API Reference](api-reference.md) --- # Interpolation Module ## Overview The Interpolation Module, also known as the Data Reader or Data API Module, provides data by subscribing to parameters at custom frequencies and performing automatic interpolation or decimation. This module pipelines with the Sample Reader output and enables applications to receive data at their desired sampling rates regardless of the original data frequency. ## Key Features - **Custom Frequency Subscriptions**: Subscribe to parameters at any desired frequency - **Automatic Interpolation**: Super-sample lower frequency data to higher frequencies - **Automatic Decimation**: Sub-sample higher frequency data to lower frequencies - **Batch Result Delivery**: Receive results in configurable batches - **Multiple Subscriptions**: Manage multiple subscriptions with different frequencies - **Subscription Processors**: Apply custom processing logic to subscriptions - **Dynamic Management**: Add and remove subscriptions on-the-fly - **Pipeline Integration**: Seamlessly connects to Sample Reader output ## API Access ### Get the Data Reader API ```csharp var dataReaderApi = supportLibApi.GetDataReaderApi(); ``` ### Create a Data Reader Service ```csharp var result = dataReaderApi.CreateService(); if (result.Success && result.Data != null) { var dataReaderService = result.Data; } ``` **Note**: Unlike other services, the Data Reader has no configuration at creation time. All configuration is done through subscriptions. ## Core Concepts ### Subscription Model Each subscription specifies: - **Subscription Key**: Unique identifier for the subscription - **Parameters**: List of parameter identifiers to include - **Subscription Frequency**: Target frequency in Hz (samples per second) - **Delivery Frequency**: How often to deliver batched results (optional) - **Handler**: Callback to receive interpolated results - **Processor**: Optional custom processing logic ### Interpolation vs. Decimation - **Super-sampling (Interpolation)**: When subscription frequency > original data frequency - Data is interpolated to fill gaps - Creates smooth transitions between known values - **Sub-sampling (Decimation)**: When subscription frequency < original data frequency - Data is decimated to reduce sample count - Selects or averages samples to match target frequency ## Methods ### SetSampleReaderService Connects the data reader to a sample reader service. ```csharp public void SetSampleReaderService(ISampleReaderService sampleReaderService) ``` **Example:** ```csharp // Create the pipeline: PacketReader -> SampleReader -> DataReader // 1. Create packet reader var packetReaderService = readerApi.CreateService(dataSource, sessionKey).Data; // 2. Create sample reader var sampleReaderService = sampleReaderApi.CreateService(bufferingConfig).Data; // 3. Create data reader var dataReaderService = dataReaderApi.CreateService().Data; // 4. Connect the pipeline sampleReaderService.SetReaderService(packetReaderService); dataReaderService.SetSampleReaderService(sampleReaderService); // 5. Initialize and start (in order) packetReaderService.Initialise(); sampleReaderService.Initialise(); dataReaderService.Initialise(); dataReaderService.Start(); sampleReaderService.Start(); packetReaderService.Start(); ``` ### Subscribe Creates a new subscription for interpolated data. ```csharp public void Subscribe( string subscriptionKey, IReadOnlyList parameterIdentifiers, double subscriptionFrequencyHz, IBatchResultHandler handler, double? deliveryFrequencyHz = null, ISubscriptionProcessor? processor = null) ``` **Parameters:** - **subscriptionKey**: Unique identifier for this subscription (used to unsubscribe later) - **parameterIdentifiers**: List of parameter names to include in results - **subscriptionFrequencyHz**: Target sampling frequency in Hz (e.g., 10.0 = 10 samples/second) - **deliveryFrequencyHz**: (Optional) How often to deliver batched results in Hz (default: 10.0) - If not specified, defaults to 10 Hz (10 deliveries per second) - If specified, results are batched and delivered at this frequency - Should not be set to 0 - **processor**: (Optional) Custom processor for additional data transformation **Example:** ```csharp // Subscribe to parameters at 100 Hz with results delivered every 10 Hz dataReaderService.Subscribe( subscriptionKey: "TelemetryData_100Hz", parameterIdentifiers: new[] { "Speed", "RPM", "Throttle", "BrakePressure" }, subscriptionFrequencyHz: 100.0, // Want 100 samples per second handler: new MyBatchResultHandler(), deliveryFrequencyHz: 10.0); // Deliver batches 10 times per second ``` ### Unsubscribe Removes a subscription. ```csharp public void Unsubscribe(string subscriptionKey) ``` **Example:** ```csharp dataReaderService.Unsubscribe("TelemetryData_100Hz"); ``` ## Handlers ### IBatchResultHandler Interface Implement this interface to receive interpolation results. ```csharp public interface IBatchResultHandler { void Handle(BatchResult result); } ``` ### BatchResult Contains the interpolated results for a batch. ```csharp public class BatchResult { public readonly string DataSource; public readonly string SubscriptionKey; public readonly string SessionKey; public readonly ulong DeliveryIntervalStartTimeNano; public readonly ulong DeliveryIntervalEndTimeNano; public readonly IReadOnlyList Results; } ``` **Properties:** - **DataSource**: The data source identifier - **SubscriptionKey**: Unique identifier for the subscription - **SessionKey**: Session identifier for the data - **DeliveryIntervalStartTimeNano**: Start timestamp of the delivery interval in nanoseconds - **DeliveryIntervalEndTimeNano**: End timestamp of the delivery interval in nanoseconds - **Results**: Collection of interpolated results in this batch ### IProcessResult Individual interpolation result. ```csharp public interface IProcessResult { string Identifier { get; } ulong IntervalStartTimeNano { get; } ulong IntervalEndTimeNano { get; } } ``` **Properties:** - **Identifier**: parameter identifier - **IntervalStartTimeNano**: Start timestamp of the interval in nanoseconds - **IntervalEndTimeNano**: End timestamp of the interval in nanoseconds ## Complete Examples ### Example 1: Basic Interpolation ```csharp using MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.InterpolationModule.Abstractions; public class BasicInterpolationExample { public void SetupInterpolation(ISupportLibApi supportLibApi, string dataSource, string sessionKey) { // 1. Create the full pipeline var readerApi = supportLibApi.GetReadingPacketApi(); var packetReaderService = readerApi.CreateService(dataSource, sessionKey).Data; var sampleReaderApi = supportLibApi.GetSampleReaderApi(); var bufferingConfig = new BufferingConfiguration( subscribedParameters: new[] { "Speed", "RPM", "Throttle" }); var sampleReaderService = sampleReaderApi.CreateService(bufferingConfig).Data; var dataReaderApi = supportLibApi.GetDataReaderApi(); var dataReaderService = dataReaderApi.CreateService().Data; // 2. Connect pipeline sampleReaderService.SetReaderService(packetReaderService); dataReaderService.SetSampleReaderService(sampleReaderService); // 3. Create subscription dataReaderService.Subscribe( subscriptionKey: "MainTelemetry", parameterIdentifiers: new[] { "Speed", "RPM", "Throttle" }, subscriptionFrequencyHz: 50.0, // 50 Hz output handler: new InterpolationHandler(), deliveryFrequencyHz: 10.0); // Deliver 10 times per second // 4. Initialize and start packetReaderService.Initialise(); sampleReaderService.Initialise(); dataReaderService.Initialise(); dataReaderService.Start(); sampleReaderService.Start(); packetReaderService.Start(); } private class InterpolationHandler : IBatchResultHandler { private int batchCount = 0; public void Handle(BatchResult result) { batchCount++; Console.WriteLine($"\n=== Batch {batchCount} ==="); Console.WriteLine($"Subscription: {result.SubscriptionKey}"); Console.WriteLine($"Session: {result.SessionKey}"); Console.WriteLine($"Data Source: {result.DataSource}"); Console.WriteLine($"Interval: {result.DeliveryIntervalStartTimeNano}ns - {result.DeliveryIntervalEndTimeNano}ns"); Console.WriteLine($"Results: {result.Results.Count}"); foreach (var processResult in result.Results) { Console.WriteLine($" Parameter: {processResult.Identifier}"); Console.WriteLine($" Interval: {processResult.IntervalStartTimeNano}ns - {processResult.IntervalEndTimeNano}ns"); } } } } ``` ### Example 2: Multiple Subscriptions at Different Frequencies ```csharp public class MultiFrequencyExample { private IDataReaderService dataReaderService; public void SetupMultipleSubscriptions(IDataReaderService dataReaderService) { this.dataReaderService = dataReaderService; // High frequency subscription for critical parameters dataReaderService.Subscribe( subscriptionKey: "HighFreq_Critical", parameterIdentifiers: new[] { "Speed", "BrakePressure", "ThrottlePosition" }, subscriptionFrequencyHz: 200.0, // 200 Hz handler: new HighFrequencyHandler(), deliveryFrequencyHz: 20.0); // Medium frequency for engine data dataReaderService.Subscribe( subscriptionKey: "MediumFreq_Engine", parameterIdentifiers: new[] { "RPM", "EngineTemp", "OilPressure" }, subscriptionFrequencyHz: 50.0, // 50 Hz handler: new MediumFrequencyHandler(), deliveryFrequencyHz: 10.0); // Low frequency for slowly changing parameters dataReaderService.Subscribe( subscriptionKey: "LowFreq_Ambient", parameterIdentifiers: new[] { "AmbientTemp", "TrackTemp", "Humidity" }, subscriptionFrequencyHz: 1.0, // 1 Hz handler: new LowFrequencyHandler(), deliveryFrequencyHz: 1.0); } private class HighFrequencyHandler : IBatchResultHandler { public void Handle(BatchResult result) { // Process high-frequency critical data Console.WriteLine($"[200Hz] Received {result.Results.Count} samples"); } } private class MediumFrequencyHandler : IBatchResultHandler { public void Handle(BatchResult result) { // Process medium-frequency engine data Console.WriteLine($"[50Hz] Received {result.Results.Count} samples"); } } private class LowFrequencyHandler : IBatchResultHandler { public void Handle(BatchResult result) { // Process low-frequency ambient data Console.WriteLine($"[1Hz] Received {result.Results.Count} samples"); } } } ``` ## Best Practices 1. **Choose Appropriate Frequencies**: - Higher frequencies = more data, higher processing cost - Match frequency to your actual needs (display update rate, analysis requirements) 2. **Use Delivery Frequency Wisely**: - Batching results reduces handler call overhead - Balance between latency and efficiency 3. **Subscription Keys**: - Use descriptive keys for easier management - Follow a naming convention (e.g., "Feature_Frequency_Purpose") 4. **Pipeline Order**: - Always set up the full pipeline before starting - Initialize in order: Packet -> Sample -> Data - Start in reverse order: Data -> Sample -> Packet 5. **Resource Management**: - Unsubscribe from subscriptions you no longer need - Close file handles and dispose resources in handlers 6. **Error Handling**: - Wrap handler logic in try-catch to prevent crashes - Log errors for debugging ## Performance Considerations ### Interpolation Overhead - Higher subscription frequencies require more interpolation calculations - Consider the original data frequency vs. desired output frequency - Large frequency mismatches increase processing load ### Delivery Frequency Optimization ```csharp // High overhead - handler called very frequently dataReaderService.Subscribe("Sub1", params, 100.0, handler, deliveryFrequencyHz: 100.0); // Optimized - results batched, handler called less frequently dataReaderService.Subscribe("Sub2", params, 100.0, handler, deliveryFrequencyHz: 10.0); ``` ### Memory Management - Batched results accumulate in memory until delivered - Very low delivery frequencies with high subscription frequencies may use significant memory - Balance batch size with available memory ## See Also - [Buffering Module](buffering-module.md) - [Reader Module](reader-module.md) - [API Reference](api-reference.md) --- # API Reference ## Overview This document provides a comprehensive reference for all public APIs, interfaces, and types in the MA DataPlatforms Streaming Support Library. ## Namespace Organization - `MA.DataPlatforms.Streaming.Support.Lib.Core.Abstractions` - Core abstractions and factory - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.SessionInfoModule` - Session management - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.DataFormatInfoModule` - Data format management - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.WritingModule` - Packet writing - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.ReadingModule` - Packet reading - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.Essentials` - Essential packet reading - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.BufferingModule` - Sample buffering - `MA.DataPlatforms.Streaming.Support.Lib.Core.Contracts.InterpolationModule` - Data interpolation --- ## Core API ### ISupportLibApi Main entry point for the support library. ```csharp public interface ISupportLibApi : IServiceModel { ISessionManagerModuleApi? GetSessionManagerApi(); IDataFormatManagerModuleApi? GetDataFormatManagerApi(); IPacketWritingModuleApi? GetWritingPacketApi(); IPacketReadingModuleApi? GetReadingPacketApi(); IEssentialsModuleApi? GetEssentialsApi(); ISampleReadingModuleApi? GetSampleReaderApi(); IDataReadingModuleApi? GetDataReaderApi(); event EventHandler? Initialised; bool IsInitialised { get; } Task InitialiseAsync(CancellationToken cancellationToken); void Initialise(); void Start(); void Stop(); } ``` ### SupportLibApiFactory Factory for creating support library instances. ```csharp public class SupportLibApiFactory : ISupportLibApiFactory { public ISupportLibApi Create( ILogger logger, StreamingApiConfiguration streamingApiConfiguration, RetryPolicy retryPolicy); } ``` ### ApiResult Standard result wrapper for all API calls. ```csharp public class ApiResult { public bool Success { get; } public string Message { get; } public T? Data { get; } } ``` ### IServiceModel Base interface for all services. ```csharp public interface IServiceModel : IDisposable { event EventHandler? Initialised; event EventHandler? Started; event EventHandler? Stopped; bool IsInitialised { get; } bool IsStarted { get; } void Initialise(); Task InitialiseAsync(CancellationToken cancellationToken); void Start(); void Stop(); } ``` --- ## Session Manager Module ### ISessionManagerModuleApi ```csharp public interface ISessionManagerModuleApi : IModuleApi { ApiResult CreateService(); } ``` ### ISessionManagementService ```csharp public interface ISessionManagementService : IModuleApiService { // Events event EventHandler? LiveSessionStateChange; event EventHandler? LiveSessionStarted; event EventHandler? LiveSessionEnded; event EventHandler? NewAssociationDetected; // Properties Guid Id { get; } // Methods ApiResult> GetAllSessions(); ApiResult CreateNewSession(SessionCreationDto createSessionDto); ApiResult UpdateIdentifier(string dataSource, string sessionKey, string newIdentifier); ApiResult AddAssociateSession(string dataSource, string sessionKey, string addingAssociateKey); ApiResult UpdateSessionDetails(string dataSource, string sessionKey, IReadOnlyList sessionDetails); ApiResult GetSessionInfo(string dataSource, string sessionKey); ApiResult EndSession(string dataSource, string sessionKey); } ``` ### ISessionInfo ```csharp public interface ISessionInfo { string SessionKey { get; } string DataSource { get; } string Identifier { get; } string Type { get; } uint Version { get; } bool Historical { get; } TimeSpan UtcOffset { get; } IReadOnlyDictionary Details { get; } IReadOnlyList AssociateSessionKeys { get; } } ``` ### SessionCreationDto ```csharp public class SessionCreationDto { public readonly string DataSource; public readonly string Identifier; public readonly string Type; public readonly uint Version; public readonly TimeSpan UtcOffset; public readonly IReadOnlyList Details; public readonly IReadOnlyList AssociatedSessionKeys; public SessionCreationDto( string dataSource = "Default", string identifier = "", string type = "Session", uint version = 1, TimeSpan? utcOffset = null, IReadOnlyList? details = null, IReadOnlyList? associatedSessionKeys = null); } ``` ### SessionDetailDto ```csharp public class SessionDetailDto { public readonly string Key; public readonly string Value; public SessionDetailDto(string key, string value); } ``` ### ISessionAssociationInfo ```csharp public interface ISessionAssociationInfo { ISessionInfo MainSessionInfo { get; } ISessionInfo AssociateSessionInfo { get; } string Group { get; } } ``` --- ## Data Format Manager Module ### IDataFormatManagerModuleApi ```csharp public interface IDataFormatManagerModuleApi : IModuleApi { ApiResult CreateService(); } ``` ### IDataFormatManagementService ```csharp public interface IDataFormatManagementService : IModuleApiService { // Events event EventHandler DataFormatInfoUpdated; // Methods ApiResult GetEventDataFormatId(string dataSource, string eventName); ApiResult GetEvent(string dataSource, ulong dataFormatId); ApiResult GetParameterDataFormatId(string dataSource, IReadOnlyList parameterList); ApiResult GetParametersList(string dataSource, ulong dataFormatId); ApiResult?> GetDataSourceDataFormatInfo(string dataSource); } ``` ### DataFormatInfo ```csharp public class DataFormatInfo { public string DataSource { get; } public string Key { get; } public IReadOnlyList DataFormats { get; } public IReadOnlyList Identifiers { get; } public DataFormatInfoType DataFormatInfoType { get; } } ``` ### ParameterDataFormatInfo ```csharp public class ParameterDataFormatInfo { public readonly string DataSource; public IReadOnlyList ParameterList; public readonly ulong DataFormatId; } ``` ### EventDataFormatInfo ```csharp public class EventDataFormatInfo { public readonly string DataSource; public ulong DataFormatId; public string EventName; } ``` ### DataFormatInfoType ```csharp public enum DataFormatInfoType { Parameters, Event } ``` --- ## Writer Module ### IPacketWritingModuleApi ```csharp public interface IPacketWritingModuleApi : IModuleApi { ApiResult CreateService(); } ``` ### IPacketWriterService ```csharp public interface IPacketWriterService : IModuleApiService { ApiResult WriteData(string dataSource, string stream, string sessionKey, Packet packet); ApiResult WriteInfo(Packet packet, InfoType infoType = InfoType.SessionInfo); } ``` ### InfoType ```csharp public enum InfoType { Unspecified, SessionInfo, SystemStatus } ``` --- ## Essentials Module ### IEssentialsModuleApi ```csharp public interface IEssentialsModuleApi : IModuleApi { ApiResult CreateService(string dataSource, string sessionKey); } ``` ### IEssentialsService ```csharp public interface IEssentialsService : IModuleApiService { // Events event EventHandler ReadingEssentialsStarted; event EventHandler ReadingEssentialsCompleted; // Properties string DataSource { get; } string SessionKey { get; } // Methods void AddHandler(IHandler handler); void RemoveHandler(IHandler handler); } ``` **Description**: The Essentials Module provides access to essential session packets such as configuration data, data format definitions, and session metadata. It reads from the dedicated "Essential" stream which contains critical session information without the overhead of processing full telemetry data. The lifecycle of the Essentials service is tied to the session lifecycle—it starts when the session begins and completes when all essential packets have been read or the session ends. --- ## Reader Module ### IPacketReadingModuleApi ```csharp public interface IPacketReadingModuleApi : IModuleApi { ApiResult CreateService(string dataSource, string sessionKey); ApiResult CreateService(IPacketReadingConfiguration config); } ``` ### IPacketReaderService ```csharp public interface IPacketReaderService : IModuleApiService { // Events event EventHandler SessionReadingStarted; event EventHandler SessionReadingCompleted; event EventHandler StreamReadingStarted; event EventHandler StreamReadingCompleted; event EventHandler SessionInfoUpdated; event EventHandler SessionAssociationInfoUpdated; event EventHandler CoverageCursorReceived; // Properties Guid Id { get; } // Methods void AddHandler(IHandler handler); void RemoveHandler(IHandler handler); } ``` ### PacketReadingConfiguration ```csharp public class PacketReadingConfiguration : IPacketReadingConfiguration { public string SessionKey { get; } public uint InactivityTimeoutSeconds { get; } public bool ExcludeMainStream { get; } public string DataSource { get; } public string SessionIdentifierPattern { get; } public ReadingType ReadingType { get; } public IReadOnlyList Streams { get; } public IReadOnlyList ExcludedStreams { get; } public LiveReadingType LiveReadingType { get; } public int BufferLength { get; } public string GroupId { get; } public PacketReadingConfiguration( string? dataSource = "Default", string? sessionIdentifierPattern = "*", ReadingType? readingType = ReadingType.Live, IReadOnlyList? streams = null, bool? excludeMainStream = false, uint? inactivityTimeoutSeconds = 30, string? sessionKey = null, LiveReadingType? liveReadingType = null, int? bufferLength = null, string? groupId = null, IReadOnlyList? excludedStreams = null); } ``` ### ReadingType ```csharp public enum ReadingType { Live = 0, Historic = 1, Both = 2 } ``` ### LiveReadingType ```csharp public enum LiveReadingType { FromBeginning = 0, // Read live session from the beginning LeadingEdge = 1 // Read live session from current point (skip past data) } ``` ### IReceivedPacketDto ```csharp public interface IReceivedPacketDto { string DataSource { get; } string SessionKey { get; } Packet Packet { get; } DateTime SubmitTime { get; } string Stream { get; } } ``` ### IStreamInfo ```csharp public interface IStreamInfo { string Stream { get; } string SessionKey { get; } string DataSource { get; } } ``` ### ICoverageCursorInfo ```csharp public interface ICoverageCursorInfo { string DataSource { get; } string SessionKey { get; } ulong CoverageCursorTime { get; } } ``` --- ## Buffering Module ### ISampleReadingModuleApi ```csharp public interface ISampleReadingModuleApi : IModuleApi { ApiResult CreateService(IBufferingConfiguration bufferingConfiguration); } ``` ### ISampleReaderService ```csharp public interface ISampleReaderService : IModuleApiService { // Events event EventHandler SessionDataMergingStarted; event EventHandler SessionDataMergingCompleted; // Properties Guid Id { get; } // Methods void SetReaderService(IPacketReaderService readingService); void AddHandler(IHandler timestampHandler); void AddHandler(IHandler sampleHandler); void RemoveHandler(IHandler timestampHandler); void RemoveHandler(IHandler sampleHandler); IReadOnlyList<(string Stream, ulong LastTimestamp)> GetLastTimeInfo(string dataSource, string sessionKey); void Subscribe(IEnumerable parameterIdentifiers); void SubscribeAll(); // Not recommended due to performance implications void Unsubscribe(IEnumerable parameterIdentifiers); void UnsubscribeAll(); } ``` ### BufferingConfiguration ```csharp public class BufferingConfiguration : IBufferingConfiguration { public bool IncludeMarkerData { get; } public bool IncludeEventData { get; } public bool IncludeErrorData { get; } public bool IncludeCanData { get; } public MergeStrategy MergeStrategy { get; } public uint BufferingWindowLength { get; } public uint SlidingWindowPercentage { get; } public IReadOnlyList SubscribedParameters { get; } public bool AllowWildcardMatch { get; } public BufferingConfiguration( IReadOnlyList? subscribedParameters = null, uint? bufferingWindowLength = 3000, uint? slidingWindowPercentage = 5, MergeStrategy? mergeStrategy = null, bool? includeMarkerData = null, bool? includeEventData = null, bool? includeErrorData = null, bool? includeCanData = null, bool? allowWildcardMatch = null); } ``` ### MergeStrategy Defines the strategy for merging data packets. ```csharp public enum MergeStrategy { WritingMerge = 0, ReadingMerge = 1 // Default strategy; recommended for better performance } ``` ### ITimestampData ```csharp public interface ITimestampData { string DataSource { get; } string SessionKey { get; } ulong Timestamp { get; } IReadOnlyList TimeColumns { get; } } ``` ### ITimeColumn ```csharp public interface ITimeColumn { ulong Timestamp { get; } IReadOnlyList SampleValues { get; } } ``` ### ISampleValue ```csharp public interface ISampleValue { string Identifier { get; } IGenericValue Value { get; } ValueStatus Status { get; } Type ValueType { get; } T? Cast(); } ``` ### ISampleData ```csharp public interface ISampleData { string DataSource { get; } string SessionKey { get; } ulong StartTime { get; } ulong EndTime { get; } string Identifier { get; } IReadOnlyList Values { get; } } ``` ### ITimestampValue ```csharp public interface ITimestampValue { ulong Timestamp { get; } IGenericValue Value { get; } ValueStatus Status { get; } Type ValueType { get; } T? Cast(); } ``` --- ## Interpolation Module ### IDataReadingModuleApi ```csharp public interface IDataReadingModuleApi : IModuleApi { ApiResult CreateService(); } ``` ### IDataReaderService ```csharp public interface IDataReaderService : IModuleApiService { // Properties Guid Id { get; } // Methods void SetSampleReaderService(ISampleReaderService sampleReaderService); void Subscribe( string subscriptionKey, IReadOnlyList parameterIdentifiers, double subscriptionFrequencyHz, IBatchResultHandler handler, double? deliveryFrequencyHz = null, ISubscriptionProcessor? processor = null); void Unsubscribe(string subscriptionKey); } ``` ### IBatchResultHandler ```csharp public interface IBatchResultHandler { void Handle(BatchResult result); } ``` ### BatchResult ```csharp public class BatchResult { public readonly string DataSource; public readonly string SubscriptionKey; public readonly string SessionKey; public readonly ulong DeliveryIntervalStartTimeNano; public readonly ulong DeliveryIntervalEndTimeNano; public readonly IReadOnlyList Results; public BatchResult( string dataSource, string sessionKey, string subscriptionKey, ulong start, ulong end, IReadOnlyList results); } ``` ### IProcessResult ```csharp public interface IProcessResult { string Identifier { get; } ulong IntervalStartTimeNano { get; } ulong IntervalEndTimeNano { get; } } ``` ### ProcessResult ```csharp public abstract class ProcessResult : IProcessResult { protected ProcessResult(string identifier, ulong start, ulong end); public string Identifier { get; } public ulong IntervalStartTimeNano { get; } public ulong IntervalEndTimeNano { get; } } ``` ### DefaultResult Default implementation of `IProcessResult` that provides standard aggregation results. This is the default result created by the default processor within the interpolation module. ```csharp public class DefaultResult : ProcessResult { // Constructor for Subsampling public DefaultResult(string paramId, ulong start, ulong end, double first, double last, double min, double max, double mean, int count) : base(paramId, start, end); // Constructor for Super-sampling public DefaultResult(string paramId, ulong start, ulong end, double lastValueBefore) : base(paramId, start, end); // Empty Result Constructor public DefaultResult(string paramId, ulong start, ulong end) : base(paramId, start, end); public bool IsSuperSampled { get; } public double? First { get; } public double? Last { get; } public double? Min { get; } public double? Max { get; } public double? Mean { get; } public int Count { get; } } ``` ### ISubscriptionProcessor ```csharp public interface ISubscriptionProcessor { // Custom processor interface for advanced subscription processing IProcessResult Process(ProcessContext context); } ``` --- ## Common Interfaces ### IHandler ```csharp public interface IHandler { void Handle(T obj); } ``` ### IModuleApi ```csharp public interface IModuleApi { internal void Initialise(); internal Task InitialiseAsync(CancellationToken cancellationToken); internal void Start(); internal void Stop(); } ``` ### IModuleApiService ```csharp public interface IModuleApiService : IServiceModel { // Base interface for all module services } ``` --- ## Configuration Classes ### StreamingApiConfiguration ```csharp public class StreamingApiConfiguration : IStreamingApiConfiguration { public StreamCreationStrategy StreamCreationStrategy { get; } public string BrokerUrl { get; } public IReadOnlyList? PartitionMappings { get; } public int StreamApiPort { get; } public bool IntegrateSessionManagement { get; } public bool IntegrateDataFormatManagement { get; } public bool UseRemoteKeyGenerator { get; } public string RemoteKeyGeneratorServiceAddress { get; } public bool BatchingResponses { get; } public uint InitialisationTimeoutSeconds { get; } public uint TerminationTimeoutSeconds { get; } public string Domain { get; } public int BatchingSizesKb { get; } public int BatchingTimeMs { get; } public bool EnableGrpcReflection { get; } public string KafkaBrokerPublishingConfigFilePath { get; } public string KafkaBrokerConsumingConfigFilePath { get; } public string KafkaBrokerConfigWhitelistFilePath { get; } public KafkaSecurityConfiguration? Security { get; set; } public bool EnableRouterPublishMetrics { get; } public bool EnableRouterConsumeMetrics { get; } public StreamingApiConfiguration( StreamCreationStrategy? streamCreationStrategy = null, string? brokerUrl = null, IReadOnlyList? partitionMappings = null, int? streamApiPort = 13579, bool? integrateSessionManagement = true, bool? integrateDataFormatManagement = true, bool? useRemoteKeyGenerator = false, string? remoteKeyGeneratorServiceAddress = "", bool? batchingResponses = false, uint? initialisationTimeoutSeconds = 3, uint? terminationTimeoutSeconds = 1, string? domain = null, int? batchingSizesKb = null, int? batchingTimeMs = null, bool? enableGrpcReflection = false, string? kafkaBrokerPublishingConfigFilePath = null, string? kafkaBrokerConsumingConfigFilePath = null, string? kafkaBrokerConfigWhitelistFilePath = null, KafkaSecurityConfiguration? security = null, bool? enableRouterPublishMetrics = false, bool? enableRouterConsumeMetrics = false); } ``` The three `kafkaBroker...FilePath` parameters point to optional JSON files that tune the underlying Kafka producer and consumer, and `security` carries the credentials for connecting to a secured broker. Both are covered in [Kafka Configuration](kafka-configuration.md). ### KafkaSecurityConfiguration Credentials for connecting to a secured Kafka broker. When `Security` is `null` (or no credential field is set) the library connects without security. ```csharp namespace MA.Streaming.Abstraction; public class KafkaSecurityConfiguration { public string? SaslUsername { get; set; } public string? SaslPassword { get; set; } public string? Protocol { get; set; } // "Plaintext" | "Ssl" | "SaslPlaintext" | "SaslSsl" public string? Mechanism { get; set; } // "Plain" | "ScramSha256" | "ScramSha512" public string? SslCaLocation { get; set; } public string? SslCertificateLocation { get; set; } public string? SslKeyLocation { get; set; } public string? SslKeyPassword { get; set; } } ``` See [Kafka Configuration](kafka-configuration.md) for usage scenarios, defaults, and the Python equivalents. ### StreamCreationStrategy ```csharp public enum StreamCreationStrategy { PartitionBased = 1, TopicBased = 2, } ``` ### RetryPolicy ```csharp public class RetryPolicy : IRetryPolicy { public int MaxRetryCount { get; } public TimeSpan DelayBetweenRetries { get; } public RetryMode RetryMode { get; } public RetryPolicy( int maxRetryCount, TimeSpan delayBetweenRetries, RetryMode retryMode); } ``` ### RetryMode ```csharp public enum RetryMode { Finite, Infinite } ``` --- ## Packet Types (MA.Streaming.OpenData) ### Packet ```csharp public class Packet { public string SessionKey { get; set; } public ByteString Content { get; set; } public ulong Id { get; set; } public string Type { get; set; } public bool IsEssential { get; set; } } ``` ### Common Packet Types - **NewSessionPacket**: Session start marker - **EndOfSessionPacket**: Session end marker - **PeriodicDataPacket**: Regular interval data - **RowDataPacket**: Timestamped row data - **EventPacket**: Event occurrences - **MarkerPacket**: Session markers - **DataFormatDefinitionPacket**: Format definitions - **CoverageCursorInfoPacket**: Coverage information --- ## Usage Patterns ### Standard Initialization Pattern ```csharp // Create and initialize var supportLibApi = new SupportLibApiFactory().Create(logger, config, retryPolicy); await supportLibApi.InitialiseAsync(cancellationToken); if (supportLibApi.IsInitialised) { supportLibApi.Start(); } ``` ### Service Creation Pattern ```csharp // Get API var moduleApi = supportLibApi.GetXxxApi(); // Create service var result = moduleApi.CreateService(/* config */); if (result.Success && result.Data != null) { var service = result.Data; service.Initialise(); service.Start(); } ``` ### Pipeline Pattern ```csharp // Create pipeline: Reader -> Sample Reader -> Data Reader var packetReader = readerApi.CreateService(config).Data; var sampleReader = sampleReaderApi.CreateService(bufferingConfig).Data; var dataReader = dataReaderApi.CreateService().Data; // Connect pipeline sampleReader.SetReaderService(packetReader); dataReader.SetSampleReaderService(sampleReader); // Initialize packetReader.Initialise(); sampleReader.Initialise(); dataReader.Initialise(); // Start (reverse order) dataReader.Start(); sampleReader.Start(); packetReader.Start(); ``` ### Cleanup Pattern ```csharp // Stop services (reverse of start order) packetReader.Stop(); sampleReader.Stop(); dataReader.Stop(); // Stop main API supportLibApi.Stop(); ``` --- ## See Also - [Index](index.md) - [Session Manager Module](session-manager.md) - [Data Format Manager Module](data-format-manager.md) - [Writer Module](writer-module.md) - [Reader Module](reader-module.md) - [Buffering Module](buffering-module.md) - [Interpolation Module](interpolation-module.md) --- # Kafka Configuration The Support Library connects to Kafka through the embedded Stream API. From release 2.1.4.22 (July 2026), `StreamingApiConfiguration` lets you do two things that previously required changes on the broker side or custom builds: - **Connect to a secured broker** — supply SASL and/or SSL credentials through the `Security` property. - **Tune the Kafka producer and consumer** — point the library at JSON override files that adjust librdkafka settings without any code change. Both are optional: with no security and no override files, the library behaves exactly as in earlier releases. ## Connecting to a secured broker Credentials are carried by the `KafkaSecurityConfiguration` object on the `Security` property of `StreamingApiConfiguration`: ```csharp namespace MA.Streaming.Abstraction; public class KafkaSecurityConfiguration { public string? SaslUsername { get; set; } public string? SaslPassword { get; set; } public string? Protocol { get; set; } // "Plaintext" | "Ssl" | "SaslPlaintext" | "SaslSsl" public string? Mechanism { get; set; } // "Plain" | "ScramSha256" | "ScramSha512" public string? SslCaLocation { get; set; } // Path to CA certificate (PEM) public string? SslCertificateLocation { get; set; } // Path to client certificate (PEM) public string? SslKeyLocation { get; set; } // Path to client private key (PEM) public string? SslKeyPassword { get; set; } // Password for the private key } ``` All fields are plain strings, so the object binds directly from `IConfiguration` (JSON files and environment variables) without custom converters. Protocol and mechanism values are parsed case-insensitively. !!! info "When is security applied?" Security is applied only when at least one of `SaslUsername`, `SslCaLocation`, or `SslCertificateLocation` is set. Otherwise — including when `Security` is `null` — the library connects without security, so existing unsecured deployments are unaffected. When security is applied and `Protocol` or `Mechanism` is empty or unrecognised, they default to `SaslPlaintext` and `Plain` respectively. ### SASL username/password === "C#" ```csharp var config = new StreamingApiConfiguration( streamCreationStrategy: StreamCreationStrategy.TopicBased, brokerUrl: "kafka-broker:9094", partitionMappings: [], security: new KafkaSecurityConfiguration { Protocol = "SaslPlaintext", Mechanism = "Plain", // or "ScramSha256" / "ScramSha512" SaslUsername = "myuser", SaslPassword = "mypassword", }); ``` === "JSON (IConfiguration)" ```json title="appsettings.json (Security section)" linenums="1" { "StreamApiConfig": { "BrokerUrl": "kafka-broker:9094", "Security": { "Protocol": "SaslPlaintext", "Mechanism": "Plain", "SaslUsername": "myuser", "SaslPassword": "mypassword" } } } ``` === "Environment variables" ``` StreamApiConfig__Security__Protocol=SaslPlaintext StreamApiConfig__Security__Mechanism=Plain StreamApiConfig__Security__SaslUsername=myuser StreamApiConfig__Security__SaslPassword=mypassword ``` ### SSL (one-way TLS) The broker presents a certificate that the client verifies against your CA certificate: ```csharp security: new KafkaSecurityConfiguration { Protocol = "Ssl", SslCaLocation = "/certs/ca.pem", } ``` ### Mutual TLS (client certificate) ```csharp security: new KafkaSecurityConfiguration { Protocol = "Ssl", SslCaLocation = "/certs/ca.pem", SslCertificateLocation = "/certs/client-cert.pem", SslKeyLocation = "/certs/client-key.pem", SslKeyPassword = "keypassword", } ``` Combine SASL credentials with `Protocol = "SaslSsl"` and `SslCaLocation` for SASL over an encrypted connection. ### Python The Python package mirrors the same shape with snake_case names: ```python from ma_dataplatforms_streaming_support_library.contracts.shared.kafka_security_configuration import KafkaSecurityConfiguration from ma_dataplatforms_streaming_support_library.contracts.shared.stream_api_configuration import StreamingApiConfiguration config = StreamingApiConfiguration( broker_url="kafka-broker:9094", security=KafkaSecurityConfiguration( protocol="SaslPlaintext", mechanism="Plain", sasl_username="myuser", sasl_password="mypassword", ), ) ``` ## Tuning the Kafka producer and consumer Three optional parameters on `StreamingApiConfiguration` point at JSON files that override the built-in librdkafka settings: | Constructor parameter | Property | Resolved default | |---|---|---| | `kafkaBrokerPublishingConfigFilePath` | `KafkaBrokerPublishingConfigFilePath` | `kafka-publisher-broker.json` | | `kafkaBrokerConsumingConfigFilePath` | `KafkaBrokerConsumingConfigFilePath` | `kafka-consumer-broker.json` | | `kafkaBrokerConfigWhitelistFilePath` | `KafkaBrokerConfigWhitelistFilePath` | `kafka-broker-whitelist.json` | Paths may be absolute or relative to the host application's working directory. A value without a `json` extension gets `.json` appended automatically. If a file does not exist, the built-in Stream API defaults are used — so leaving everything at its default with no files present changes nothing. ### Override file format Override files map **librdkafka dot-separated property names** to string values: ```json title="kafka-publisher-broker.json" linenums="1" { "compression.type": "lz4", "linger.ms": "50", "batch.size": "8388608" } ``` !!! warning "Use librdkafka key names" Keys must use the librdkafka dot format (`linger.ms`, `compression.type`) — the Confluent C# PascalCase names (`LingerMs`) are not recognised. See the [librdkafka configuration reference](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) for the full list of valid keys. Some keys are managed by the library and ignored if present in an override file: `bootstrap.servers` is always taken from `BrokerUrl`, and the consumer `group.id` is assigned internally. ### Whitelist file The whitelist file is an optional filter deciding which override-file keys actually take effect — useful when the override files are edited by people who should only touch an approved set of settings. It contains a rule list per direction; rules are evaluated first-match-wins and `*` is a wildcard: ```json title="kafka-broker-whitelist.json" linenums="1" { "Publish": [ { "Pattern": "compression.type", "Allow": true }, { "Pattern": "linger.ms", "Allow": true }, { "Pattern": "*", "Allow": false } ], "Consume": [ { "Pattern": "session.timeout.ms", "Allow": true }, { "Pattern": "*", "Allow": false } ] } ``` When no whitelist file exists, every key from the override files is applied unfiltered. ### Python and FFI The same three settings are available from the Python package (and the underlying FFI contract) as `kafka_broker_publishing_config_file_path`, `kafka_broker_consuming_config_file_path`, and `kafka_broker_config_whitelist_file_path`: ```python config = StreamingApiConfiguration( broker_url="kafka-broker:9092", kafka_broker_publishing_config_file_path="kafka-publisher-broker.json", kafka_broker_consuming_config_file_path="kafka-consumer-broker.json", kafka_broker_config_whitelist_file_path="kafka-broker-whitelist.json", ) ``` An empty string (the Python default) means "use the default file path". ## See Also - [API Reference](api-reference.md#streamingapiconfiguration) — full `StreamingApiConfiguration` signature - [Overview](index.md) - [librdkafka configuration reference](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) --- # Overview A microservice solution for file and configuration indexing, designed specifically for Motion Applied's data platform infrastructure. ## Overview The solution provides robust file system indexing and specialized configuration file processing capabilities through two complementary microservices: - **Files Service** (Port 5000): General-purpose file indexing and management - **Configs Service** (Port 5050): Specialized CFG and PGV file processing This solution is particularly suited for configuration management, and large-scale file system auditing scenarios. ## Architecture ![Architecture Diagram](images/architecture.svg) ## Features ### Files Service Features - **Recursive Directory Scanning** with configurable file extensions - **Metadata Extraction** (size, modification dates, checksums) - **Status Tracking** (Exist, Processed, NotFound, Error) - **Advanced Search** with pagination and filtering - **Scheduled Scanning** using Quartz cron expressions - **Manual Operations** via REST APIs - **Health Monitoring** with heartbeat endpoints ### Configs Service Features - **CFG File Processing** with configuration ID extraction - **PGV File Processing** with application version tracking - **Relationship Mapping** between CFG and PGV files - **Specialized Queries** for configuration management - **Dual Scanning** processes for different file types - **Processing Coordination** and aggregation ## Technology Stack - **Framework**: ASP.NET Core 8.0+ with C# 11+ - **Databases**: MongoDB (production) / LiteDB (development) - **Scheduling**: Quartz.NET with cron expressions - **Logging**: Serilog with structured logging - **Configuration**: ASP.NET Core Configuration with environment variables - **API**: RESTful APIs with comprehensive error handling ## Documentation Comprehensive documentation is available: | Document | Description | |----------|-------------| | **[Service Overview](service-overview.md)** | High-level architecture and service descriptions | | **[API Documentation](api-documentation.md)** | Complete REST API reference with examples | | **[Configuration Guide](appsettings-configuration.md)** | Detailed appsettings and environment variables | ## Getting Started ### Prerequisites - .NET 8.0 SDK or later - MongoDB (for production) or LiteDB (automatic for development) - Visual Studio 2022 or VS Code ### Configuration #### Basic Configuration (appsettings.json) **Files Service:** ```json { "Directories": ["C:\\YourDataPath"], "Extensions": [".pgv", ".cfg", ".pul"], "StorageType": 0, "LiteDb": { "FileName": "FileIndexingDB" } } ``` **Configs Service:** ```json { "FileIndexerEndpoints": { "CfgEndpoint": "http://your-files-service:5000/", "PgvEndpoint": "http://your-files-service:5000/" }, "StorageType": 0, "LiteDb": { "FileName": "ConfigIndexingDB" } } ``` #### Environment Variables ```bash # Files Service Directories__0=C:\Data\Files Extensions__0=.pgv StorageType=1 MongoDb__ConnectionString=mongodb://your-mongo-server:27017 # Configs Service FileIndexerEndpoints__CfgEndpoint=http://your-files-service:5000/ StorageType=1 ``` ## Production Deployment ### Docker Deployment ```yaml # docker-compose.yml example version: '3.8' services: file-indexer-service: image: atlasplatformdocker/file-indexing-service:latest ports: - "5000:5000" environment: - StorageType=1 - MongoDb__ConnectionString=mongodb://mongo:27017 - ASPNETCORE_URLS='http://*:5000' - Directories__0='/app/data' - Extensions__0='.pgv' - Extensions__1='.cfg' volumes: - /data:/app/data depends_on: - mongodb config-indexer-service: image: atlasplatformdocker/config-indexing-service:latest ports: - "5050:5050" environment: - StorageType=1 - MongoDb__ConnectionString=mongodb://mongo:27017 - ASPNETCORE_URLS='http://*:5050' - FileIndexerEndpoints__CfgEndpoint='http://file-indexer-service:5000' - FileIndexerEndpoints__PgvEndpoint='http://file-indexer-service:5000' depends_on: - mongodb - file-indexer-service volumes: - /data:/app/data mongodb: image: mongodb/mongodb-community-server:latest ports: - "27017:27017" ``` ## Configuration Reference | Service | Default Port | Database | Primary File Types | |---------|-------------|----------|------------------| | Files | 5000 | MongoDB/LiteDB | .pgv, .cfg, .pul | | Configs | 5050 | MongoDB/LiteDB | .cfg, .pgv | ### Storage Types - **0**: LiteDB (embedded, file-based) - **1**: MongoDB (production, scalable) **Built by Motion Applied Data Platform Team** --- # Service Overview This document provides a brief description of each service in the MA.DataPlatforms.Indexing solution. ## Architecture Overview The solution consists of two main microservices that work together to provide comprehensive file and configuration indexing capabilities for Motion Applied's data platform. ![Architecture Diagram](images/architecture.svg) ## Services ### 1. MA.DataPlatforms.Indexing.Files Service **Purpose**: General-purpose file indexing and management service **Port**: 5000 **Description**: This service provides comprehensive file system indexing capabilities. It scans specified directories for files with configured extensions and maintains a searchable index of file metadata including paths, sizes, modification dates, and optional checksums. **Key Responsibilities**: - **File Discovery**: Scans directories recursively for files matching specified extensions - **Metadata Extraction**: Captures file size, modification dates, and optional checksums - **Status Tracking**: Monitors file processing states (Exist, Processed, NotFound, Error) - **Search & Query**: Provides advanced search capabilities with pagination - **Scheduled Scanning**: Automated scanning using Quartz scheduler - **Manual Operations**: Add/remove file metadata manually via API - **Health Monitoring**: Provides heartbeat and status endpoints **Supported File Types**: All file types are configurable via appsettings. Common examples include: - `.pgv` - `.cfg` - `.pul` **Use Cases**: - File system auditing and monitoring - Data lake file discovery - File integrity checking with checksums - Integration with data processing pipelines - File availability validation --- ### 2. MA.DataPlatforms.Indexing.Configs Service **Purpose**: Specialized configuration and parameter file indexing service **Port**: 5050 **Description**: This service focuses specifically on indexing and processing .cfg and .pgv files. It extracts meaningful metadata from these specialized file formats and provides APIs for retrieving configuration relationships and processing information. **Key Responsibilities**: - **CFG File Processing**: Extracts configuration IDs and metadata from .cfg files - **PGV File Processing**: Extracts application IDs, versions, and header information from .pgv files - **Relationship Mapping**: Links CFG and PGV files through processing information - **Specialized Queries**: Provides APIs for config-specific and PGV-specific searches - **Dual Scanning**: Separate scanning processes for CFG and PGV files - **Processing Coordination**: Manages the relationship between different file types - **Integration**: Communicates with the Files service for file discovery **Specialized Data Extraction**: - **CFG Files**: Configuration IDs, file relationships - **PGV Files**: Application IDs, versions, header hash codes - **Processing Info**: Aggregates related files **Use Cases**: - Configuration management and tracking - Parameter version control - Application configuration discovery - Racing data configuration analysis - Telemetry parameter management --- ## Service Interaction ### Data Flow 1. **Files Service** discovers and indexes files in configured directories 2. **Configs Service** receives notifications about new .cfg and .pgv files 3. **Configs Service** processes the specialized file formats to extract metadata 4. Both services maintain their own databases but can cross-reference data ### Storage Options Both services support dual storage backends: - **MongoDB**: For production environments with high volume and performance requirements - **LiteDB**: For development, testing, or smaller deployments ### Communication - **REST APIs**: Both services expose comprehensive REST endpoints - **Notification System**: Services can notify each other about file changes - **Heartbeat Monitoring**: Health check endpoints for service monitoring --- ## Technology Stack ### Backend Framework - **ASP.NET Core**: Modern web API framework - **C# 11+**: Latest language features and performance improvements ### Scheduling - **Quartz.NET**: Robust job scheduling with cron expressions - **Background Tasks**: Asynchronous file scanning and processing ### Data Storage - **MongoDB**: Document database for high-performance scenarios - **LiteDB**: Embedded database for lightweight deployments ### Logging & Monitoring - **Serilog**: Structured logging with multiple sinks - **Health Checks**: Built-in health monitoring endpoints ### Configuration - **ASP.NET Core Configuration**: JSON-based with environment variable support - **Dependency Injection**: Built-in IoC container --- # API Documentation This document provides comprehensive API documentation for both the **MA.DataPlatforms.Indexing.Files** and **MA.DataPlatforms.Indexing.Configs** projects. ## Table of Contents 1. [Overview](#overview) 2. [Files Project APIs](#files-project-apis) 3. [Configs Project APIs](#configs-project-apis) 4. [Data Models](#data-models) 5. [Error Handling](#error-handling) ## Overview Both projects expose RESTful APIs for file and configuration indexing operations. The APIs support: - File scanning and indexing - Data retrieval with pagination - Real-time status monitoring - Configuration management **Base URLs:** - Files Project: `http://your-files-service:5000` - Configs Project: `http://your-configs-service:5050` ## Files Project APIs ### Reader Controller (`/api/read`) #### Get All Files ```http GET /api/read/files/all ``` **Description**: Retrieves all indexed files with pagination support. **Query Parameters:** | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `pageSize` | integer | No | 200 | Number of items per page | | `pageNumber` | integer | No | 1 | Page number (1-based) | | `sortAscending` | boolean | No | false | Sort order by file path | **Response:** ```json { "items": [ { "id": "unique-file-id", "fileName": "example.cfg", "filePath": "C:\\Data\\example.cfg", "directoryPath": "C:\\Data", "searchRootDirectory": "C:\\Data", "fileSize": 1024, "lastModifiedTimeTick": 637849536000000000, "status": 1, "updateDates": ["2023-01-01T00:00:00Z"], "lastUpdateTime": "2023-01-01T00:00:00Z", "fileChecksum": "abc123def456" } ], "pageNumber": 1, "pageSize": 200, "totalPages": 5, "totalItems": 1000 } ``` #### Get Files by Status ```http GET /api/read/files/status/{status} ``` **Description**: Retrieves files filtered by their processing status. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `status` | string | Yes | File status (e.g., "Exist", "Processed", "NotFound") | **Query Parameters:** Same as "Get All Files" **Response:** Same as "Get All Files" #### Find Files by Substring ```http GET /api/read/files/name/substring/{substr} ``` **Description**: Finds files whose names contain the specified substring. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `substr` | string | Yes | Substring to search for in file names | **Query Parameters:** Same as "Get All Files" **Response:** Same as "Get All Files" #### Find Files by Exact Name ```http GET /api/read/files/name/is/{fileName} ``` **Description**: Finds files with exact file name match. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `fileName` | string | Yes | Exact file name to search for | **Query Parameters:** Same as "Get All Files" **Response:** Same as "Get All Files" #### Query Files with Multiple Filters ```http GET /api/read/filter/query/all ``` **Description**: Advanced file search with multiple filter criteria. **Query Parameters:** | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `fileName` | string | No | null | Filter by file name | | `filePath` | string | No | null | Filter by full file path | | `directoryPath` | string | No | null | Filter by directory path | | `status` | string | No | null | Filter by file status | | `pageSize` | integer | No | 200 | Number of items per page | | `pageNumber` | integer | No | 1 | Page number | | `sortAscending` | boolean | No | false | Sort order | **Response:** Same as "Get All Files" #### Get File by Path ```http GET /api/read/files/path?filePath={filePath} ``` **Description**: Retrieves file information by exact file path. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `filePath` | string | Yes | Complete file path | **Response:** ```json [ { "id": "unique-file-id", "fileName": "example.cfg", "filePath": "C:\\Data\\example.cfg", // ... other FileInfoModel properties } ] ``` ### Scanner Controller (`/api/scanner`) #### Start Scanning (Default Configuration) ```http GET /api/scanner/start ``` **Description**: Starts file scanning using directories and extensions from appsettings. **Response:** ```json { "message": "Scan service started. id:abc123def456", "scanId": "abc123def456" } ``` **Error Response (already running):** ```json { "message": "Scanning service already running", "scanId": "" } ``` #### Start Custom Scanning ```http POST /api/scanner/scan ``` **Description**: Starts file scanning with custom directories and extensions. **Request Body:** ```json { "directories": [ "C:\\CustomPath1", "C:\\CustomPath2" ], "extensions": [ ".cfg", ".pgv" ] } ``` **Response:** Same as "Start Scanning" #### Stop Scanning ```http GET /api/scanner/stop ``` **Description**: Stops the currently running scan operation. **Response:** ```json { "message": "The service will stop", "isStopping": true } ``` **Error Response (not running):** ```json { "message": "No scan service running to stop", "isStopping": false } ``` #### Get Scanner Status ```http GET /api/scanner/status ``` **Description**: Returns the current status of the scanning service. **Response:** ```json { "message": "The service is running", "isRunning": true } ``` ### Writer Controller (`/api/writer`) #### Add File ```http POST /api/writer/add?filePath={filePath} ``` **Description**: Manually adds a file to the index. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `filePath` | string | Yes | Full path to the file to add | **Response:** ```json { "message": "File added successfully", "isSuccess": true } ``` #### Delete File ```http POST /api/writer/delete?filePath={filePath}&force={force} ``` **Description**: Removes a file from the index. **Query Parameters:** | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `filePath` | string | Yes | - | Full path to the file to delete | | `force` | boolean | No | false | Force deletion even if file exists | **Response:** ```json { "message": "File deleted successfully", "isSuccess": true } ``` ### Update Controller (`/api/update`) #### Prune Database ```http POST /api/update/prune ``` **Description**: Removes entries for files that no longer exist on disk. **Response:** `200 OK` (empty body) #### Mark File as Processed ```http POST /api/update/process?filePath={filePath} ``` **Description**: Updates a file's status to "Processed". **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `filePath` | string | Yes | Full path to the file | **Response:** `200 OK` (empty body) ### Config Controller (`/api/config`) #### Get Environment Variables Info ```http GET /api/config/vars ``` **Description**: Returns information about supported environment variables. **Response:** ```json { "directories": { "name": "Directories_0 .. *", "example": "C:\\Data" }, "extensions": { "name": "Extensions_0 .. *", "example": ".txt" }, // ... other configuration options } ``` ### Heartbeat Controller (`/api/heartbeat`) #### Health Check ```http GET /api/heartbeat ``` **Description**: Simple health check endpoint. **Response:** ```json { "status": "ok", "timestamp": "2023-01-01T12:00:00.000Z" } ``` ## Configs Project APIs ### CFG Read Controller (`/api/read/cfg`) #### Get CFG Files by Config ID ```http GET /api/read/cfg/{configId} ``` **Description**: Retrieves CFG files associated with a specific configuration ID. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `configId` | ulong | Yes | Configuration identifier | **Response:** ```json [ { "configId": 12345, "filePath": "C:\\Configs\\config1.cfg" }, { "configId": 12345, "filePath": "C:\\Configs\\config2.cfg" } ] ``` #### Get All CFG Files ```http GET /api/read/cfg/files/all ``` **Description**: Retrieves all indexed CFG files with pagination. **Query Parameters:** | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `pageSize` | integer | No | 200 | Number of items per page | | `pageNumber` | integer | No | 1 | Page number | | `sortAscending` | boolean | No | false | Sort order | **Response:** ```json { "items": [ { "configId": 12345, "filePath": "C:\\Configs\\example.cfg" } ], "pageNumber": 1, "pageSize": 200, "totalPages": 3, "totalItems": 500 } ``` #### Get All Config IDs ```http GET /api/read/cfg/ids/all ``` **Description**: Retrieves all unique configuration IDs. **Response:** ```json [12345, 12346, 12347, 12348] ``` ### PGV Read Controller (`/api/read/pgv`) #### Get PGV Files by App ID ```http GET /api/read/pgv/{appId} ``` **Description**: Retrieves PGV files for a specific application ID. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `appId` | ushort | Yes | Application identifier | **Response:** ```json [ { "appId": 100, "appVersion": 1, "filePath": "C:\\PGVs\\app100_v1.pgv" }, { "appId": 100, "appVersion": 2, "filePath": "C:\\PGVs\\app100_v2.pgv" } ] ``` #### Get PGV Files by App ID and Version ```http GET /api/read/pgv/{appId}/versions/{appVersion} ``` **Description**: Retrieves PGV files for specific application ID and version. **Path Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `appId` | ushort | Yes | Application identifier | | `appVersion` | ushort | Yes | Application version | **Response:** ```json [ { "appId": 100, "appVersion": 1, "filePath": "C:\\PGVs\\app100_v1.pgv" } ] ``` #### Get All PGV Files ```http GET /api/read/pgv/files/all ``` **Description**: Retrieves all indexed PGV files with pagination. **Query Parameters:** Same as CFG files **Response:** Similar to CFG files but with PGV structure ### CFG Scanner Controller (`/api/scanner/cfg`) #### Start CFG Scanning ```http POST /api/scanner/cfg/start ``` **Description**: Starts CFG file scanning and indexing process. **Response:** ```json { "message": "CFG scan Service started. id:abc123def456", "scanId": "abc123def456" } ``` #### Stop CFG Scanning ```http POST /api/scanner/cfg/stop ``` **Description**: Stops the CFG scanning process. **Response:** ```json { "message": "The service will stop", "isStopping": true } ``` #### Get CFG Scanner Status ```http GET /api/scanner/cfg/status ``` **Description**: Returns CFG scanner status. **Response:** ```json { "message": "The service is running", "isRunning": true } ``` ### PGV Scanner Controller (`/api/scanner/pgv`) #### Start PGV Scanning ```http POST /api/scanner/pgv/start ``` **Description**: Starts PGV file scanning and indexing process. **Response:** ```json { "message": "Pgv Scan Service started. id:abc123def456", "scanId": "abc123def456" } ``` #### Stop PGV Scanning ```http POST /api/scanner/pgv/stop ``` **Description**: Stops the PGV scanning process. **Response:** Similar to CFG scanner stop #### Get PGV Scanner Status ```http GET /api/scanner/pgv/status ``` **Description**: Returns PGV scanner status. **Response:** Similar to CFG scanner status ### Processing Info Controller (`/api/processing`) #### Get Processing Information ```http GET /api/processing/info?configId={configId} ``` **Description**: Retrieves processing information for a configuration. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `configId` | ulong | Yes | Configuration identifier | **Response:** ```json { "aggregate": { // Processing aggregate information }, "foundCfgInfos": [ { "configId": 12345, "filePath": "C:\\Configs\\config.cfg" } ], "foundPgvInfos": [ { "appId": 100, "appVersion": 1, "filePath": "C:\\PGVs\\app.pgv" } ] } ``` #### Get CFG Files for Config ```http GET /api/processing/files/cfg?configId={configId} ``` **Description**: Retrieves all CFG files for a specific configuration. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `configId` | ulong | Yes | Configuration identifier | **Response:** ```json [ { "configId": 12345, "filePath": "C:\\Configs\\config1.cfg" } ] ``` #### Get PGV Files for App ```http GET /api/processing/files/pgv?appId={appId}&appVersion={appVersion} ``` **Description**: Retrieves all PGV files for a specific application and version. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `appId` | ushort | Yes | Application identifier | | `appVersion` | ushort | Yes | Application version | **Response:** ```json [ { "appId": 100, "appVersion": 1, "filePath": "C:\\PGVs\\app.pgv" } ] ``` ### Indexing Controller (`/api/indexing`) #### Index File ```http POST /api/indexing/index?filePath={filePath} ``` **Description**: Manually indexes a specific CFG or PGV file. **Query Parameters:** | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `filePath` | string | Yes | Full path to the file (.cfg or .pgv) | **Response:** ```json "File example.cfg processed successfully." ``` **Error Responses:** - `404 Not Found`: "File not found." - `400 Bad Request`: "the file extension is not valid" ### Heartbeat Controller (`/api/heartbeat`) #### Health Check ```http GET /api/heartbeat ``` **Description**: Health check endpoint for the configs service. **Response:** ```json { "status": "ok", "timestamp": "2023-01-01T12:00:00.000Z" } ``` ## Data Models ### FileInfoModel ```typescript interface FileInfoModel { id: string; // Unique identifier fileName: string; // File name without path filePath: string; // Complete file path directoryPath: string; // Directory containing the file searchRootDirectory: string; // Root search directory fileSize: number; // File size in bytes lastModifiedTimeTick: number; // Last modified time (ticks) status: FileStatus; // Processing status updateDates: Date[]; // History of update dates lastUpdateTime: Date; // Most recent update fileChecksum: string; // File checksum (if enabled) } ``` ### FileStatus Enum ```typescript enum FileStatus { Exist = 0, Processed = 1, NotFound = 2, Error = 3 } ``` ### CfgFileDto ```typescript interface CfgFileDto { configId: number; // Configuration identifier (ulong) filePath: string; // Full path to CFG file } ``` ### PgvFileDto ```typescript interface PgvFileDto { appId: number; // Application identifier (ushort) appVersion: number; // Application version (ushort) filePath: string; // Full path to PGV file } ``` ### ProcessingInfoDto ```typescript interface ProcessingInfoDto { aggregate?: ProcessingAggregateInfoDto; foundCfgInfos: CfgFileDto[]; foundPgvInfos: PgvFileDto[]; } ``` ### PageSet ```typescript interface PageSet { items: T[]; // Current page items pageNumber: number; // Current page number (1-based) pageSize: number; // Items per page totalPages: number; // Total number of pages totalItems: number; // Total number of items across all pages } ``` ### ScanInfo ```typescript interface ScanInfo { directories: string[]; // List of directories to scan extensions: string[]; // List of file extensions to include } ``` ## Error Handling ### Standard Error Response ```json { "type": "https://tools.ietf.org/html/rfc7231#section-6.6.1", "title": "Unable to retrieve all files. Check server logs for details.", "status": 500, "traceId": "0HMVD1234567890" } ``` ### Common HTTP Status Codes | Code | Description | When Used | |------|-------------|-----------| | 200 | OK | Successful operation | | 400 | Bad Request | Invalid parameters or request format | | 404 | Not Found | Resource not found or scanner not running | | 500 | Internal Server Error | Server-side exceptions | ### Error Scenarios 1. **Scanner Already Running**: When trying to start a scan while one is already active 2. **Scanner Not Running**: When trying to stop a scanner that isn't running 3. **File Not Found**: When trying to index a file that doesn't exist 4. **Invalid File Extension**: When trying to index unsupported file types 5. **Database Errors**: Connection issues or query failures ## Rate Limiting and Performance - **Pagination**: All list endpoints support pagination to handle large datasets - **Default Page Size**: 200 items per page - **Maximum Page Size**: Not explicitly limited (consider implementing based on needs) - **Concurrent Operations**: Scanner operations run asynchronously to avoid blocking ## Authentication and Security Currently, the APIs do not implement authentication. Consider adding: - API key authentication - JWT token validation - IP whitelisting - HTTPS enforcement for production use --- # Configuration Guide This document provides comprehensive information about configuration options for both the **MA.DataPlatforms.Indexing.Files** and **MA.DataPlatforms.Indexing.Configs** projects. ## Table of Contents 1. [Overview](#overview) 2. [Files Project Configuration](#files-project-configuration) 3. [Configs Project Configuration](#configs-project-configuration) 4. [Common Configuration](#common-configuration) 5. [Environment Variable Mapping](#environment-variable-mapping) ## Overview Both projects use ASP.NET Core configuration system with JSON-based appsettings files. The configuration supports: - File and directory indexing settings - Database connections (MongoDB and LiteDB) - Scheduled job configuration (Quartz) - Logging and monitoring - API endpoints and networking ## Files Project Configuration ### Core File Indexing Settings #### `Directories` - **Type**: Array of strings - **Description**: List of directories to scan for files - **Example**: ```json "Directories": [ "C:\\Data\\ProjectFiles", "D:/Documents/DataFiles", "E:/Archive/ConfigFiles", "/mnt/storage/telemetry" ] ``` - **Environment Variable**: `Directories__0`, `Directories__1`, etc. #### `Extensions` - **Type**: Array of strings - **Description**: File extensions to include in indexing - **Example**: ```json "Extensions": [ ".pgv", ".cfg", ".pul" ] ``` - **Environment Variable**: `Extensions__0`, `Extensions__1`, etc. #### `ExcludedDirectories` - **Type**: Array of strings - **Description**: Directories to exclude from scanning - **Default**: `[]` (empty array) - **Example**: ```json "ExcludedDirectories": [ "C:\\Temp", "C:\\Windows" ] ``` - **Environment Variable**: `ExcludedDirectories__0`, `ExcludedDirectories__1`, etc. #### `AddFileCheckSum` - **Type**: Boolean - **Description**: Whether to calculate and store file checksums during indexing - **Default**: `true` - **Example**: `"AddFileCheckSum": true` - **Environment Variable**: `AddFileCheckSum` ### Storage Configuration #### `StorageType` - **Type**: Integer (enum) - **Description**: Database storage type to use - **Values**: - `0`: LiteDB - `1`: MongoDB - **Default**: `1` (MongoDB) - **Environment Variable**: `StorageType` #### MongoDB Configuration ```json "MongoDB": { "ConnectionString": "mongodb://your-server:27017", "DatabaseName": "YourFileIndexDB", "CollectionName": "YourIndexedFiles" } ``` - **`ConnectionString`** - **Type**: String - **Description**: MongoDB connection string - **Environment Variable**: `MongoDb__ConnectionString` - **`DatabaseName`** - **Type**: String - **Description**: MongoDB database name - **Environment Variable**: `MongoDb__DataBaseName` - **`CollectionName`** - **Type**: String - **Description**: MongoDB collection name for storing file indexes - **Environment Variable**: `MongoDb__CollectionName` #### LiteDB Configuration ```json "LiteDb": { "FileName": "YourFileIndexingDB" } ``` - **`FileName`** - **Type**: String - **Description**: LiteDB database file name (without extension) - **Environment Variable**: `LiteDb__FileName` ### Scheduling Configuration #### `QuartzSettings` ```json "QuartzSettings": { "CronExpression": "0 0 8 * * ?" } ``` - **`CronExpression`** - **Type**: String - **Description**: Cron expression for scheduled file scanning - **Example**: `"0 0 8 * * ?"` (Daily at 8:00 AM) - **Environment Variable**: `QuartzSettings__IndexingCronExpression` ## Configs Project Configuration ### File Indexer Endpoints #### `FileIndexerEndpoints` ```json "FileIndexerEndpoints": { "CfgEndpoint": "http://your-files-service:5000/", "PgvEndpoint": "http://your-files-service:5000/" } ``` - **`CfgEndpoint`** - **Type**: String - **Description**: Endpoint URL for CFG file indexing service - **Example**: `"http://your-files-service:5000/"` - **Environment Variable**: `FileIndexerEndpoints__CfgEndpoint` - **`PgvEndpoint`** - **Type**: String - **Description**: Endpoint URL for PGV file indexing service - **Example**: `"http://your-files-service:5000/"` - **Environment Variable**: `FileIndexerEndpoints__PgvEndpoint` ### Storage Configuration #### `StorageType` - **Type**: Integer (enum) - **Description**: Database storage type (same as Files project) - **Values**: - `0`: LiteDB - `1`: MongoDB - **Default**: `1` (MongoDB) - **Environment Variable**: `StorageType` #### MongoDB Configuration ```json "MongoDB": { "ConnectionString": "mongodb://your-mongo-server:27017", "DatabaseName": "YourConfigIndexDb" } ``` - **`ConnectionString`** - **Type**: String - **Description**: MongoDB connection string - **Environment Variable**: `MongoDb__ConnectionString` - **`DatabaseName`** - **Type**: String - **Description**: Database name for config indexing - **Environment Variable**: `MongoDb__DataBaseName` #### LiteDB Configuration ```json "LiteDb": { "FileName": "YourConfigIndexingDB" } ``` - **`FileName`** - **Type**: String - **Description**: LiteDB database file name for config storage - **Environment Variable**: `LiteDb__FileName` ### Scheduling Configuration #### `QuartzSettings` ```json "QuartzSettings": { "CfgCronExpression": "0 30 9 * * ?", "PgvCronExpression": "0 45 9 * * ?" } ``` - **`CfgCronExpression`** - **Type**: String - **Description**: Cron expression for CFG file scanning - **Example**: `"0 59 13 * * ?"` (Daily at 1:59 PM) - **Environment Variable**: `QuartzSettings__CfgCronExpression` - **`PgvCronExpression`** - **Type**: String - **Description**: Cron expression for PGV file scanning - **Example**: `"0 00 14 * * ?"` (Daily at 2:00 PM) - **Environment Variable**: `QuartzSettings__PgvCronExpression` ### Notification Configuration #### `NotificationUrls` ```json "NotificationUrls": ["http://your-files-service:5000/"] ``` - **Type**: Array of strings - **Description**: Callback URLs that this service registers with the Files indexer to receive notifications when .cfg and .pgv files are discovered - **Example**: `["http://your-files-service:5000/api/notification"]` - **Environment Variable**: `NotificationUrls__0`, `NotificationUrls__1`, etc. - **Note**: **Important** - To ensure configs are indexed immediately when files are discovered, set this to the Files service URL (e.g., `http://your-files-service:5000/`). Without this configuration, configs will only be indexed when the scheduled Quartz job runs. ## Common Configuration ### Web Server Configuration #### `Kestrel` ```json "Kestrel": { "Endpoints": { "Http": { "Url": "http://your-service:5000" // Files: 5000, Configs: 5050 } } } ``` - **Files Project**: Default port `5000` - **Configs Project**: Default port `5050` - **Environment Variable**: `Kestrel__Endpoints__Http__Url` #### `AllowedHosts` - **Type**: String - **Description**: Allowed host headers for the application - **Default**: `"*"` (all hosts allowed) - **Environment Variable**: `AllowedHosts` ### Logging Configuration #### Standard Logging ```json "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } } ``` #### Serilog Configuration ```json "Serilog": { "Using": ["Serilog.Sinks.Console", "Serilog.Sinks.File"], "WriteTo": [ { "Name": "Console", "Args": { "outputTemplate": "[{Timestamp:HH:mm:ss.fff} {Level:u3}] {Message:lj}{NewLine}{Exception}" } }, { "Name": "File", "Args": { "path": "Logs/log-.txt", "rollingInterval": "Day", "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u3}] {Message:lj}{NewLine}{Exception}" } } ] } ``` ## Environment Variable Mapping The application supports configuration through environment variables using the standard ASP.NET Core configuration hierarchy: ### Files Project Environment Variables | Configuration Path | Environment Variable | Example Value | |-------------------|---------------------|---------------| | `Directories:0` | `Directories__0` | `"C:\\Data"` | | `Directories:1` | `Directories__1` | `"D:\\Archive"` | | `Extensions:0` | `Extensions__0` | `".pgv"` | | `Extensions:1` | `Extensions__1` | `".cfg"` | | `ExcludedDirectories:0` | `ExcludedDirectories__0` | `"C:\\Temp"` | | `StorageType` | `StorageType` | `"1"` | | `AddFileCheckSum` | `AddFileCheckSum` | `"true"` | | `MongoDB:ConnectionString` | `MongoDb__ConnectionString` | `"mongodb://your-mongo-server:27017"` | | `MongoDB:DatabaseName` | `MongoDb__DataBaseName` | `"FileIndexDB"` | | `MongoDB:CollectionName` | `MongoDb__CollectionName` | `"IndexedFiles"` | | `LiteDb:FileName` | `LiteDb__FileName` | `"data.db"` | | `QuartzSettings:CronExpression` | `QuartzSettings__IndexingCronExpression` | `"0 0 * * * ?"` | ### Configs Project Environment Variables | Configuration Path | Environment Variable | Example Value | |-------------------|---------------------|---------------| | `FileIndexerEndpoints:CfgEndpoint` | `FileIndexerEndpoints__CfgEndpoint` | `"http://your-files-service:5000/"` | | `FileIndexerEndpoints:PgvEndpoint` | `FileIndexerEndpoints__PgvEndpoint` | `"http://your-files-service:5000/"` | | `StorageType` | `StorageType` | `"1"` | | `MongoDB:ConnectionString` | `MongoDb__ConnectionString` | `"mongodb://your-mongo-server:27017"` | | `MongoDB:DatabaseName` | `MongoDb__DataBaseName` | `"ConfigIndexDB"` | | `LiteDb:FileName` | `LiteDb__FileName` | `"config.db"` | | `QuartzSettings:CfgCronExpression` | `QuartzSettings__CfgCronExpression` | `"0 30 9 * * ?"` | | `QuartzSettings:PgvCronExpression` | `QuartzSettings__PgvCronExpression` | `"0 45 9 * * ?"` | | `NotificationUrls:0` | `NotificationUrls__0` | `"http://your-files-service:5000/"` | ### Common Environment Variables (Both Projects) | Configuration Path | Environment Variable | Example Value | |-------------------|---------------------|---------------| | `AllowedHosts` | `AllowedHosts` | `"*"` | | `Kestrel:Endpoints:Http:Url` | `Kestrel__Endpoints__Http__Url` | `"http://your-service:5000"` | | `Logging:LogLevel:Default` | `Logging__LogLevel__Default` | `"Information"` | | `Logging:LogLevel:Microsoft.AspNetCore` | `Logging__LogLevel__Microsoft.AspNetCore` | `"Warning"` | ## Configuration Examples ### Development Environment ```json { "Directories": ["C:\\Dev\\TestFiles"], "Extensions": [".cfg", ".pgv"], "StorageType": 0, "LiteDb": { "FileName": "development.db" }, "Kestrel": { "Endpoints": { "Http": { "Url": "http://your-service:5000" } } } } ``` ### Production Environment ```json { "Directories": [ "/data/production/configs", "/data/production/pgvs" ], "Extensions": [".cfg", ".pgv", ".pul"], "StorageType": 1, "MongoDB": { "ConnectionString": "mongodb://your-production-cluster:27017", "DatabaseName": "ProductionIndexDB" }, "QuartzSettings": { "CronExpression": "0 0 2 * * ?" } } ``` --- # Overview This section includes detailed guides for automating and enhancing your ATLAS Client workflows using various APIs defined. Get the libraries from our [GitHub packages feed](nuget/index.md). [Reading, Writing and Recording Data with the SQL Race API](./sql-race/index.md) : _Code Samples and API Reference, including:_ * _Reading SSN files and associated sessions_ * _Reading and writing SSN2 files and SQL Server databases_ * _Controlling recorders and handling live data_ * _Developing file loaders for third-party formats_ * _Developing custom function DLLs_ [Creating Custom ATLAS Displays with the ATLAS Display API](./display-api/index.md) : _Detailed developer guide to writing custom ATLAS displays, including:_ * _Concepts and Design Patterns_ * _Tutorials with Code Samples_ * _API Reference_ --- # Overview The Automation API is a WCF-based interface that enables seamless interaction with ATLAS and its data ecosystem. It supports reading and writing data, updating session metadata, and automating the creation of Workbooks, Pages, Displays, and Layouts. It also allows automated loading and unloading of data into ATLAS. ATLAS (Advanced Telemetry Linked Acquisition System) is a sophisticated software package developed by Motion Applied to capture, distribute, display, and analyze data from control systems—especially in motorsport and automotive applications. ## Capabilities === "Read" - Session metadata - Parameter metadata - Lap information - Timeseries data - Events and markers === "Write" - Session metadata - Transient data - Lap information - Timeseries data - Events and markers === "Automate" - Session loading - Workbook generation - Display properties - Parameter properties - Sets management !!! note While the SQL Race API offers more direct and performant access to telemetry data, the Automation API is ideal for automating ATLAS workflows. [Code Samples](https://github.com/mat-docs/Atlas.AutomationAPI.Examples){ .md-button } [API Reference](https://mat-docs.github.io/Atlas.DisplayAPI.Documentation/api/index.html){ .md-button } ## Classes and Methods The ATLAS Automation API is composed of several service clients, each responsible for a specific area of functionality such as session management, workbook automation, telemetry recording, and data access. These classes expose methods that allow developers to programmatically control and interact with the ATLAS environment, enabling powerful automation and integration scenarios. !!! tip inline end "Classes and Methods" This is a subsample of all the classes and methods available in the Automation API. [API Reference](https://mat-docs.github.io/Atlas.DisplayAPI.Documentation/api/index.html){ .md-button } ??? info "ATLAS Automation API – Classes and Methods" ### ApplicationServiceClient - Connect() - Disconnect() ### WorkbookServiceClient - LoadFromWorkbook() - GetRecorders() - CreateSet() - GetSets() - AssociateSetToWorkbook() ### RecorderServiceClient - GetRecorder() - StartRecorder() - StopRecorder() ### SetServiceClient - RemoveSet() - RenameSet() - GetList() - Disconnect() ### PageServiceClient - ActivatePage() - DuplicatePage() - NewPage() - RenamePage() ### DisplayServiceClient - GetDisplay() - AssociateControlToDisplay() - GetDisplaysProperties() ### SessionServiceClient - AppendSessions() - RemoveSessions() - TransferData() ### ParameterDataAccessServiceClient - GetNextSamples() - GetSamples() ### TimeBaseServiceClient - Connect() - Disconnect() --- # Extracting Sessions One of the fundamental operations in the ATLAS Automation API is retrieving an already loaded session from ATLAS. The example below demonstrates how to: - Obtain a list of Sets - Select a specific Set - Extract all sessions loaded into that Set This will return a list of sessions. In the example, basic information such as `Session.Name` and `Session.Id` is extracted from each session. === "C#" ``` csharp // Retrieve the list of sets from the workbook service var setsList = workbookServiceClient.GetSets(); // Get the ID of the first set var firstSetId = setsList[0].Id; // Retrieve the list of composite sessions for the first set var sessionsList = setServiceClient.GetCompositeSessions(firstSetId); // Print the name and ID of each session foreach (var session in sessionsList) { Console.WriteLine($"Session name: {session.Name}, Session ID: {session.Id}"); } ``` === "Python" ``` python # Retrieve the list of sets from the workbook service sets_list = workbookServiceClient.GetSets() # Get the ID of the first set first_set_id = sets_list[0].Id # Retrieve the list of composite sessions for the first set sessions_list = setServiceClient.GetCompositeSessions(first_set_id) # Print the name and ID of each session for session in sessions_list: print("Session name:", session.Name, "Session ID:", session.Id) ``` === "MATLAB" ``` matlab % Retrieve the list of sets from the workbook service setsList = workbookServiceClient.GetSets(); % Get the ID of the first set firstSetId = setsList(1).Id; % Retrieve the list of composite sessions for the first set sessionsList = setServiceClient.GetCompositeSessions(firstSetId); % Print the name and ID of each session for i = 1:length(sessionsList) fprintf('Session name: %s, Session ID: %s\n', sessionsList(i).Name, sessionsList(i).Id); end ``` --- # Loading Sessions Sessions can be programmatically loaded and unloaded into ATLAS, including associate sessions. Additional sessions can be appended. !!! warning SSN2 and SQLRace Sessions can be loaded to ATLAS using the method `LoadSqlRaceSessions()` from the ApplicationServiceClient Class. The session key is required and the only way to extract this is using the SQLRace API. SQLRace API ssn2_path = r'C:\\Users\\...\\my_session_file.ssn2' connection_string = rf"DbEngine=SQLite;Data Source={ssn2_path};" sessions = sessionManager.FindBySessionState(SessionState.Historical, connection_string) session = sessions[0] key = session.Key.ToString() Automation API applicationServiceClient.LoadSqlRaceSessions(mySetId, [key], [connection_string]) ## Usage Example ### Load SSN Session to ATLAS Basic example that shows how to load a single session into ATLAS. === "C#" ``` csharp // Get the list of sets from the workbook service var setsList = workbookServiceClient.GetSets(); // Select a specific set by index var mySet = setsList[set]; var mySetId = mySet.Id; // Load a session file into the selected set applicationServiceClient.LoadFileSessions(setsList[set].Id, new[] { filePath }); // Alternatively, load a session file using a direct path string ssnPath = @"C:\\Users\\...\\my_session_file.ssn"; applicationServiceClient.LoadFileSessions(mySetId, new[] { ssnPath }); ``` === "Python" ``` python # Get the list of sets from the workbook service sets_list = workbookServiceClient.GetSets() # Select a specific set by index my_set = sets_list[set] my_set_id = my_set.Id # Load a session file into the selected set applicationServiceClient.LoadFileSessions(sets_list[set].Id, [file_path]) # Alternatively, load a session file using a direct path ssn_path = r'C:\\Users\\...\\my_session_file.ssn' applicationServiceClient.LoadFileSessions(my_set_id, [ssn_path]) ``` === "MATLAB" ``` matlab % Get the list of sets from the workbook service setsList = workbookServiceClient.GetSets(); % Select a specific set by index mySet = setsList(set); mySetId = mySet.Id; % Load a session file into the selected set applicationServiceClient.LoadFileSessions(setsList(set).Id, {file_path}); % Alternatively, load a session file using a direct path ssn_path = 'C:\\Users\\...\\my_session_file.ssn'; applicationServiceClient.LoadFileSessions(mySetId, {ssn_path}); ``` !!! information "Improve Session Loading with Events" When working with session files in ATLAS, it's important to know **when a session is fully loaded**. Instead of relying on manual checks or delays, you can use the **`OnSessionLoaded` event** provided by the `ApplicationServiceClient`. // Subscribe to the event applicationServiceClient.OnSessionLoaded += SessionLoaded; // Define the event handler private void SessionLoaded(object sender, SessionEventArgs e) { Console.WriteLine($"Session loaded: {e.SessionId}"); // You can now safely interact with the loaded session } ### Load SSN Session To ATLAS With Associates Basic example that shows how to load a single session into ATLAS and then append additional associate sessions. === "C#" ``` csharp // Get the list of sets from the workbook service var setsList = workbookServiceClient.GetSets(); // Select a specific set by index var mySet = setsList[set]; var mySetId = mySet.Id; // Load a session file (.ssn) into the selected set applicationServiceClient.LoadFileSessions(setsList[set].Id, new[] { filePath }); // Define paths to session files string ssnPath = @"C:\\Users\\...\\my_session_file.ssn"; string ssvPath = @"C:\\Users\\...\\my_session_file.VTS.001.ssv"; // Load the .ssn session file applicationServiceClient.LoadFileSessions(mySetId, new[] { ssnPath }); // Retrieve the list of sessions in the set var sessions = setServiceClient.GetCompositeSessions(mySetId); // Append an associated .ssv session file to the first session sessionServiceClient.AppendSessionsWithAssociates(sessions[0].Id, new[] { ssvPath }, new[] { 1 }); ``` === "Python" ``` python # Get the list of sets from the workbook service sets_list = workbookServiceClient.GetSets() # Select a specific set by index my_set = sets_list[set] my_set_id = my_set.Id # Load a session file (.ssn) into the selected set applicationServiceClient.LoadFileSessions(sets_list[set].Id, [file_path]) # Define paths to session files ssn_path = r'C:\\Users\\...\\my_session_file.ssn' ssv_path = r'C:\\Users\\...\\my_session_file.VTS.001.ssv' # Load the .ssn session file applicationServiceClient.LoadFileSessions(my_set_id, [ssn_path]) # Retrieve the list of sessions in the set sessions = setServiceClient.GetCompositeSessions(my_set_id) # Append an associated .ssv session file to the first session sessionServiceClient.AppendSessionsWithAssociates(sessions[0].Id, [ssv_path], [1]) ``` === "MATLAB" ``` matlab % Get the list of sets from the workbook service setsList = workbookServiceClient.GetSets(); % Select a specific set by index mySet = setsList(set); mySetId = mySet.Id; % Load a session file (.ssn) into the selected set applicationServiceClient.LoadFileSessions(setsList(set).Id, {file_path}); % Define paths to session files ssn_path = 'C:\\Users\\...\\my_session_file.ssn'; ssv_path = 'C:\\Users\\...\\my_session_file.VTS.001.ssv'; % Load the .ssn session file applicationServiceClient.LoadFileSessions(mySetId, {ssn_path}); % Retrieve the list of sessions in the set sessions = setServiceClient.GetCompositeSessions(mySetId); % Append an associated .ssv session file to the first session sessionServiceClient.AppendSessionsWithAssociates(sessions(1).Id, {ssv_path}, {1}); ``` ### Unload All Sessions From Set Basic example that shows how to unload all the sessions from the selected Set. === "C#" ``` csharp // Get the list of sets from the workbook service var sets = workbookServiceClient.GetSets(); // Get the list of sessions in the selected set var sessionsList = setServiceClient.GetCompositeSessions(sets[set].Id); // Remove each session in the list foreach (var session in sessionsList) { applicationServiceClient.RemoveSession(session.Id); } ``` === "Python" ``` python # Get the list of sets from the workbook service sets = workbookServiceClient.GetSets() # Get the list of sessions in the selected set sessions_list = setServiceClient.GetCompositeSessions(sets[set].Id) # Remove each session in the list for session in sessions_list: applicationServiceClient.RemoveSession(session.Id) ``` === "MATLAB" ``` matlab % Get the list of sets from the workbook service sets = workbookServiceClient.GetSets(); % Get the list of sessions in the selected set sessionsList = setServiceClient.GetCompositeSessions(sets(set).Id); % Remove each session in the list for i = 1:length(sessionsList) applicationServiceClient.RemoveSession(sessionsList(i).Id); end ``` --- # Working With Displays ## Working with Displays This guide explains how to interact with **Displays** in a workbook using the provided service clients. You'll learn how to retrieve displays, filter them, and modify their properties such as color. ???+ tip Before working with displays, and depending on the needs, ensure you have: - A valid **session ID**, a session is loaded in ATLAS. - Access to the following service clients: - `WorkbookServiceClient` - `PageServiceClient` - `DisplayServiceClient` - `SessionServiceClient` ### Retrieve Pages from the Workbook === "C#" ```csharp var pages = workbookServiceClient.GetPages(); Console.WriteLine($"Total amount of pages to be processed: {pages.Count}"); ``` === "Python" ```python pages = workbookServiceClient.GetPages() print(f"Total amount of pages to be processed: {len(pages)}") ``` === "MATLAB" ```matlab pages = workbookServiceClient.GetPages(); fprintf('Total amount of pages to be processed: %d\n', length(pages)); ``` ### Iterate Through Pages and Displays === "C#" ```csharp foreach (var page in pages) { Console.WriteLine($"Processing Page: {page.Title}"); var displays = pageServiceClient.GetDisplaysOfPage(page.Id); } ``` === "Python" ```python for page in pages: print(f"Processing Page: {page.Title}") displays = pageServiceClient.GetDisplaysOfPage(page.Id) ``` === "MATLAB" ```matlab for i = 1:length(pages) page = pages(i); fprintf('Processing Page: %s\n', page.Title); displays = pageServiceClient.GetDisplaysOfPage(page.Id); end ``` ### Filter and Process Waveform Displays === "C#" ```csharp foreach (var display in displays) { if (display.Type != "Waveform") continue; Console.WriteLine($"Processing Display: {display.Name}"); } ``` === "Python" ```python for display in displays: if display.Type != "Waveform": continue print(f"Processing Display: {display.Name}") ``` === "MATLAB" ```matlab for j = 1:length(displays) display = displays(j); if ~strcmp(display.Type, 'Waveform') continue; end fprintf('Processing Display: %s\n', display.Name); end ``` ### Retrieve Display Parameters === "C#" ```csharp var displayParameters = displayServiceClient.GetDisplayParameters(display.Id); foreach (var p in displayParameters) { Console.WriteLine($"Parameter Name: {p.Name}"); } ``` === "Python" ```python displayParameters = displayServiceClient.GetDisplayParameters(display.Id) for p in displayParameters: print(f"Parameter Name: {p.Name}") ``` === "MATLAB" ```matlab displayParameters = displayServiceClient.GetDisplayParameters(display.Id); for k = 1:length(displayParameters) p = displayParameters(k); fprintf('Parameter Name: %s\n', p.Name); end ``` ### Modify Display Properties Check if each parameter is in your dictionary and modify its properties: ??? info "Available Display Parameter Properties" These are the standard properties that can be retrieved or modified for each display parameter: - AdditionalMarkers - ApplicationName - BitShift - BitShiftDirection - ByteOrder - ClipTrace - Color - ConversionFunctionName - CriticalMaximum - CriticalMinimum - DataBitMask - DataSource - DataType - Definition - FullDescription - DisplayMaximum - DisplayMinimum - DistributionMinimum - DistributionMaximum - ErrorBitMask - FdlFunctionCode - FilterFrequency - FilterType - FormatString - IsFunction - Identifier - InterpolateBetweenSamples - LineStyle - Name - SampleMode - ShowCriticalLimits - ShowUnits - ShowWarningLimits - TraceStyle - TraceWidth - TrendType - Units - IsVirtualParameter - WarningMaximum - WarningMinimum - WeightedStatistics - YScaleMode === "C#" ```csharp var parameters = new Dictionary> { { "vCar", new Dictionary { { "Color", "#0000FF" }, { "TraceWidth", "1" } } }, { "nEngine", new Dictionary { { "Color", "#FF0000" }, { "TraceWidth", "2" } } }, { "MEngine", new Dictionary { { "Color", "#008000" }, { "TraceWidth", "3" } } }, { "sLap", new Dictionary { { "Color", "#FFFF00" }, { "TraceWidth", "8" } } } }; if (parameters.ContainsKey(p.Name)) { var parameter = sessionServiceClient.GetSessionParameter(sessionId, p.Identifier); var displayParameterProperties = displayServiceClient.GetDisplayParameterProperties(display.Id, parameter.Identifier); foreach (var property in displayParameterProperties) { if (property.Name == "Color") { var value = new Value { Data = parameters[p.Name]["Color"] }; displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value); var color = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name); Console.WriteLine($"Parameter: {parameter.Identifier} Color set to: {color}"); } else if (property.Name == "TraceWidth") { var value = new Value { Data = parameters[p.Name]["TraceWidth"] }; displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value); var traceWidth = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name); Console.WriteLine($"Parameter: {parameter.Identifier} TraceWidth set to: {traceWidth}"); } } } ``` === "Python" ```python parameters = { "vCar": { "Color": "#0000FF", # Blue "TraceWidth": 1 }, "nEngine": { "Color": "#FF0000", # Red "TraceWidth": 2 }, "MEngine": { "Color": "#008000", # Green "TraceWidth": 3 }, "sLap": { "Color": "#FFFF00", # Yellow "TraceWidth": 8 } } if p.Name in parameters: parameter = sessionServiceClient.GetSessionParameter(sessionId, p.Identifier) displayParameterProperties = displayServiceClient.GetDisplayParameterProperties(display.Id, parameter.Identifier) for property in displayParameterProperties: if property.Name == "Color": value = Value() value.Data = parameters[p.Name]["Color"] displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value) color = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name) print(f"Parameter: {parameter.Identifier} Color set to: {color}") elif property.Name == "TraceWidth": value = Value() value.Data = str(parameters[p.Name]["TraceWidth"]) displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value) trace_width = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name) print(f"Parameter: {parameter.Identifier} TraceWidth set to: {trace_width}") ``` === "MATLAB" ```matlab parameters = struct( ... 'vCar', struct('Color', '#0000FF', 'TraceWidth', 1), ... 'nEngine', struct('Color', '#FF0000', 'TraceWidth', 2), ... 'MEngine', struct('Color', '#008000', 'TraceWidth', 3), ... 'sLap', struct('Color', '#FFFF00', 'TraceWidth', 8) ... ); if isfield(parameters, p.Name) parameter = sessionServiceClient.GetSessionParameter(sessionId, p.Identifier); displayParameterProperties = displayServiceClient.GetDisplayParameterProperties(display.Id, parameter.Identifier); for i = 1:length(displayParameterProperties) property = displayParameterProperties(i); if strcmp(property.Name, 'Color') value = Value(); value.Data = parameters.(p.Name).Color; displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value); color = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name); fprintf('Parameter: %s Color set to: %s\n', parameter.Identifier, color); elseif strcmp(property.Name, 'TraceWidth') value = Value(); value.Data = num2str(parameters.(p.Name).TraceWidth); displayServiceClient.SetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name, value); traceWidth = displayServiceClient.GetDisplayParameterPropertyValue(display.Id, parameter.Identifier, property.Name); fprintf('Parameter: %s TraceWidth set to: %s\n', parameter.Identifier, traceWidth); end end end ``` ??? tip "Other Display Properties" These properties apply to the display as a whole and control layout, appearance, and behavior: - AlignGridYScale - AllowHistoricReplay - BackgroundColor - CircuitDefinition - CursorMinimumPrecision - CursorMode - DrawLeftToRight - LegendFontSize - LeftToRightGapPercentage - HighlightAlpha - HighlightMode - BackgroundImageOffsetLeft - BackgroundImagePath - BackgroundImageOffsetTop - LeftToRightOverlay - MaskEventsAll - MaskEventsHigh - MaskEventsLow - MaskEventsMedium - LegendPosition - ReferenceCursorUses - ShowCircuitStripeLabels - CircuitStripeSource - HeaderVisible - ShowDateMarkers - ShowEventIndicators - LegendShowGrid - GridDisplayMode - ShowBackgroundImage - ShowLapMarkers - LegendVisible - ShowStatistics - ShowTimeAs - TimeDiffVisible - TitleBarVisible - LegendShowUnits - LegendShowWarningIndicators - TimeDiffStart - Statistics - Title - UseGlobalCircuitDefinition - LegendValueAlignment - ParameterZOrder --- # Timebase and Cursor Point The `TimeBaseServiceClient` class is used to interact with time-based automation services. It provides a rich set of methods for manipulating time bases and cursors in ATLAS. Both the cursor and the TimeBase can be adjusted, set and retrieved. !!! info "Understanding the TimeBase in ATLAS" The **TimeBase** defines the time range that ATLAS will display on screen. You can adjust the TimeBase to: - **Zoom in** on a specific time range - **Zoom out** to view a broader segment - **Display a complete lap** or the entire session To set the full session TimeBase, use `Session.StartTime` and `Session.EndTime` with the `SetTimeBase` method. ### Time Base Manipulation
![Image title](assets/TimeBase.png){ width="800" }
ATLAS TimeBase
- **`SetTimeBase(ObjectId, Int64, Int64)`** Sets the time base using start and end ticks. - **`SetTimeBaseByLap(ObjectId, Int16)`** Sets the time base to a specific lap. - **`SetTimeBaseByLaps(ObjectId, Int16, Int16)`** Sets the time base between two laps. - **`SetTimeBaseFastestLap(ObjectId)`** Sets the time base to the fastest lap. - **`SetTimeBaseVba(ObjectId, Double, Double)`** Sets the time base using VBA (Virtual Best Average) values. ### Cursor Control - **`SetCursor(ObjectId, Int64)`** Sets the cursor to a specific tick. - **`SetCursorToDistance(ObjectId, Double)`** Sets the cursor to a specific distance. - **`SetCursorToDistanceIntoLap(ObjectId, Double, Int16)`** Sets the cursor to a distance into a specific lap. - **`SetCursorVba(ObjectId, Double)`** Sets the cursor using a VBA value. ### Time Base Retrieval - **`GetTimeBase(ObjectId)`** Retrieves the current time base for the specified object. ## Usage Example ### Setting ATLAS Cursor Point The ATLAS cursor point can be set to a specific timestamp or Distance using the methods below. === "C#" ``` csharp var timebase = sessionServiceClient.GetSessionTimeBase(session.Id); var timebaseService = new TimeBaseServiceClient(timebase.Id); timebaseService.SetCursor(timebase.Id, session.StartTime); timebaseService.SetCursorToDistance(timebase.Id, 1500.5); timebaseService.SetCursorToDistanceIntoLap(timebase.Id, 300.0, 2); timebaseService.SetCursorVba(timebase.Id, 12.5); ``` === "Python" ``` python timebase = sessionServiceClient.GetSessionTimeBase(session.Id) timebaseService = TimeBaseServiceClient(timebase.Id) timebaseService.SetCursor(timebase.Id, session.StartTime) timebaseService.SetCursorToDistance(timebase.Id, 1500.5) timebaseService.SetCursorToDistanceIntoLap(timebase.Id, 300.0, 2) timebaseService.SetCursorVba(timebase.Id, 12.5) ``` === "MATLAB" ``` matlab timebase = GetSessionTimeBase(session.Id); timebaseService = TimeBaseServiceClient(timebase.Id); SetCursor(timebase.Id, session.StartTime); SetCursorToDistance(timebase.Id, 1500.5); SetCursorToDistanceIntoLap(timebase.Id, 300.0, 2); SetCursorVba(timebase.Id, 12.5); ``` ### Setting ATLAS TimeBase ATLAS TimeBase can be adjusted using the methods below. === "C#" ``` csharp var timebase = sessionServiceClient.GetSessionTimeBase(session.Id); var timebaseService = new TimeBaseServiceClient(timebase.Id); timebaseService.SetTimeBase(timebase.Id, 1000000, 2000000); timebaseService.SetTimeBaseByLap(timebase.Id, 3); timebaseService.SetTimeBaseByLaps(timebase.Id, 2, 5); timebaseService.SetTimeBaseFastestLap(timebase.Id); timebaseService.SetTimeBaseVba(timebase.Id, 12.5, 45.3); ``` === "Python" ``` python timebase = sessionServiceClient.GetSessionTimeBase(session.Id) timebaseService = TimeBaseServiceClient(timebase.Id) timebaseService.SetTimeBase(timebase.Id, 1000000, 2000000) timebaseService.SetTimeBaseByLap(timebase.Id, 3) timebaseService.SetTimeBaseByLaps(timebase.Id, 2, 5) timebaseService.SetTimeBaseFastestLap(timebase.Id) timebaseService.SetTimeBaseVba(timebase.Id, 12.5, 45.3) ``` === "MATLAB" ``` matlab timebase = GetSessionTimeBase(session.Id); timebaseService = TimeBaseServiceClient(timebase.Id); SetTimeBase(timebase.Id, 1000000, 2000000); SetTimeBaseByLap(timebase.Id, 3); SetTimeBaseByLaps(timebase.Id, 2, 5); SetTimeBaseFastestLap(timebase.Id); SetTimeBaseVba(timebase.Id, 12.5, 45.3); ``` --- # Working With Data The `ParameterDataAccessServiceClient` class is the entry point for accessing parameter data from a session. It provides a variety of methods to retrieve, navigate, and analyze time-series data. ## Available Methods ### **Statistics Retrieval** | Method | Description | |-------|-------------| | `GetDataStatistics(SessionId, String, Int64, Int64, StatisticOption)` | Retrieves statistical data over a specified time range. | | `GetDataStatisticsVba(SessionId, String, Double, Double, StatisticOption)` | VBA-compatible version of `GetDataStatistics`. | | `GetWeightedDataStatistics(SessionId, String, Int64, Int64, StatisticOption)` | Retrieves weighted statistics over a time range. | | `GetWeightedDataStatisticsVba(SessionId, String, Double, Double, StatisticOption)` | VBA-compatible version of `GetWeightedDataStatistics`. | ### **Data Navigation** | Method | Description | |--------|-------------| | `Goto(SessionId, String, Int64)` | Moves the data pointer to a specific timestamp. | | `GotoVba(SessionId, String, Double)` | VBA-compatible version of `Goto`. | | `SetSampleIncrement(SessionId, String, Int64)` | Sets the increment between samples. | | `SetSampleIncrementVba(SessionId, String, Double)` | VBA-compatible version of `SetSampleIncrement`. | ### **Sample Retrieval** | Method | Description | |--------|-------------| | `GetNextData(SessionId, String, Int64, SampleMode, Boolean)` | Retrieves the next data point(s) using a specific mode. | | `GetNextDataVba(SessionId, String, Int32, SampleMode, Boolean)` | VBA-compatible version of `GetNextData`. | | `GetNextSamples(SessionId, String, Int64)` | Retrieves the next N samples. | | `GetNextSamplesVba(SessionId, String, Int32)` | VBA-compatible version of `GetNextSamples`. | | `GetNextSamplesToTime(SessionId, String, Int64)` | Retrieves samples up to a specific time. | | `GetNextSamplesToTimeVba(SessionId, String, Double)` | VBA-compatible version of `GetNextSamplesToTime`. | | `GetPreviousSamples(SessionId, String, Int64)` | Retrieves the previous N samples. | | `GetPreviousSamplesVba(SessionId, String, Int32)` | VBA-compatible version of `GetPreviousSamples`. | ### **Sample Count Estimation** | Method | Description | |--------|-------------| | `GetSamplesCountEstimate(SessionId, String, Int64, Int64)` | Estimates the number of samples in a time range. | | `GetSamplesCountEstimateVba(SessionId, String, Double, Double)` | VBA-compatible version of `GetSamplesCountEstimate`. | ## Usage Example ### Extracting Data Statistics per Lap === "C#" ``` csharp var sessionId = sessionsList[0].Id; var parameterName = "vCar:Chassis"; var startTime = session.StartTime; var endTime = session.EndTime; var statisticsOption = StatisticOption.Max; // Get overall maximum value var statistics = parameterDataAccessServiceClient.GetDataStatistics( sessionId, parameterName, startTime, endTime, statisticsOption ); Console.WriteLine($"{parameterName} - Maximum value: {statistics.Maximum}"); // Get maximum value per lap var laps = sessionServiceClient.GetSessionLaps(sessionId); foreach (var lap in laps) { var lapStats = parameterDataAccessServiceClient.GetDataStatistics( sessionId, parameterName, lap.StartTime, lap.EndTime, statisticsOption ); Console.WriteLine($"Lap Name: {lap.Name}, Start Time: {lap.StartTime}, End Time: {lap.EndTime}, " + $"{parameterName} - Maximum value: {lapStats.Maximum}"); } ``` === "Python" ``` python # Retrieve session and parameter information session_id = sessionsList[0].Id parameter_name = 'vCar:Chassis' start_time = session.StartTime end_time = session.EndTime statistics_option = StatisticOption.Max # Get overall maximum value for the parameter statistics = parameterDataAccessServiceClient.GetDataStatistics( session_id, parameter_name, start_time, end_time, statistics_option ) print(f'{parameter_name} - Maximum value: {statistics.Maximum}') # Get maximum value per lap laps = sessionServiceClient.GetSessionLaps(session_id) for lap in laps: lap_stats = parameterDataAccessServiceClient.GetDataStatistics( session_id, parameter_name, lap.StartTime, lap.EndTime, statistics_option ) print(f'Lap Name: {lap.Name}, Start Time: {lap.StartTime}, End Time: {lap.EndTime}, ' f'{parameter_name} - Maximum value: {lap_stats.Maximum}') ``` === "MATLAB" ``` matlab sessionId = sessionsList(1).Id; parameterName = 'vCar:Chassis'; startTime = session.StartTime; endTime = session.EndTime; statisticsOption = StatisticOption.Max; % Get overall maximum value statistics = parameterDataAccessServiceClient.GetDataStatistics( ... sessionId, parameterName, startTime, endTime, statisticsOption); fprintf('%s - Maximum value: %f\n', parameterName, statistics.Maximum); % Get maximum value per lap laps = sessionServiceClient.GetSessionLaps(sessionId); for i = 1:length(laps) lap = laps(i); lapStats = parameterDataAccessServiceClient.GetDataStatistics( ... sessionId, parameterName, lap.StartTime, lap.EndTime, statisticsOption); fprintf('Lap Name: %s, Start Time: %f, End Time: %f, %s - Maximum value: %f\n', ... lap.Name, lap.StartTime, lap.EndTime, parameterName, lapStats.Maximum); end ``` ### Extracting Time Series Data for the Fastest Lap This example demonstrates how to extract time series data for the parameter `vCar` during the **fastest lap** of a session. It involves the following steps: 1. **Identifying the fastest lap** Use the session service to retrieve the fastest lap of the session. 2. **Moving the data pointer to the start of that lap** Position the data access service at the start time of the fastest lap using the `Goto` method. 3. **Estimating how many samples are available in that time range** Use `GetSamplesCountEstimate` to determine how many data points exist between the start and end of the lap. 4. **Retrieving the corresponding data and timestamps** Use `GetNextSamples` to extract the actual time series data and timestamps for the parameter during the lap. === "C#" ``` csharp // Get the fastest lap of the session var fastestLap = sessionServiceClient.GetSessionFastestLap(session.Id); // Define the parameter to extract string parameter = "vCar"; // Retrieve parameter metadata (optional) var parameterInfo = sessionServiceClient.GetSessionParameter(session.Id, parameter); // Move to the start of the fastest lap parameterDataAccessServiceClient.Goto(session.Id, parameter, fastestLap.StartTime); // Estimate the number of samples in the lap long sampleCount = parameterDataAccessServiceClient.GetSamplesCountEstimate( session.Id, parameter, fastestLap.StartTime, fastestLap.EndTime ); // Retrieve the time series data var parameterValues = parameterDataAccessServiceClient.GetNextSamples(session.Id, parameter, sampleCount); var values = parameterValues.Data.ToList(); var times = parameterValues.Time.ToList(); ``` === "Python" ``` python # Get the fastest lap of the session fastest_lap = sessionServiceClient.GetSessionFastestLap(session.Id) # Define the parameter to extract parameter = 'vCar' # Retrieve parameter metadata (optional) parameter_info = sessionServiceClient.GetSessionParameter(session.Id, parameter) # Move to the start of the fastest lap parameterDataAccessServiceClient.Goto(session.Id, parameter, fastest_lap.StartTime) # Estimate the number of samples in the lap sample_count = parameterDataAccessServiceClient.GetSamplesCountEstimate( session.Id, parameter, fastest_lap.StartTime, fastest_lap.EndTime ) # Retrieve the time series data parameter_values = parameterDataAccessServiceClient.GetNextSamples(session.Id, parameter, sample_count) values = list(parameter_values.Data) times = list(parameter_values.Time) ``` === "MATLAB" ``` matlab % Get the fastest lap of the session fastestLap = sessionServiceClient.GetSessionFastestLap(session.Id); % Define the parameter to extract parameter = 'vCar'; % Retrieve parameter metadata (optional) parameterInfo = sessionServiceClient.GetSessionParameter(session.Id, parameter); % Move to the start of the fastest lap parameterDataAccessServiceClient.Goto(session.Id, parameter, fastestLap.StartTime); % Estimate the number of samples in the lap sampleCount = parameterDataAccessServiceClient.GetSamplesCountEstimate( ... session.Id, parameter, fastestLap.StartTime, fastestLap.EndTime); % Retrieve the time series data parameterValues = parameterDataAccessServiceClient.GetNextSamples(session.Id, parameter, sampleCount); values = parameterValues.Data; times = parameterValues.Time; ``` --- # Transient Parameters This guide demonstrates how to add a transient parameter to an ATLAS session, retrieve data from an existing parameter, perform a transformation, and write the result to the new transient parameter. Overview: 1. Create a transient parameter. 2. Retrieve the time base and source parameter (`vCar`). 3. Estimate the number of samples. 4. Read the data and apply a transformation (`vCar * 2`). 5. Write the transformed data to the transient parameter. **Parameters** - **DemoDisplayName**: `"Demo Waveform"` - **ParameterName**: `"vCar:Chassis"` - **TransientParameterName**: `"vCar2 Demo"` - **Range**: `0` to `800` ## Define Transient Parameter Metadata This section sets up the necessary metadata for the transient parameter. It includes the display name, identifier, description, value range, and grouping information. These values are used to register the parameter in the ATLAS session and define how it will appear and behave. === "C#" ``` csharp string demoDisplayName = "Demo Waveform"; string parameterName = "vCar:Chassis"; string transientParameterDescription = "vCar2 Demo"; string transientParameterGroups = ""; string transientParameterIdentifier = "vCar2 Demo"; double transientParameterMaximum = 800; double transientParameterMinimum = 0; string transientParameterName = "vCar2 Demo"; ``` === "Python" ``` python DemoDisplayName = "Demo Waveform" ParameterName = "vCar:Chassis" TransientParameterDescription = "vCar2 Demo" TransientParameterGroups = "" TransientParameterIdentifier = "vCar2 Demo" TransientParameterMaximum = 800 TransientParameterMinimum = 0 TransientParameterName = "vCar2 Demo" ``` === "MATLAB" ``` matlab demoDisplayName = 'Demo Waveform'; parameterName = 'vCar:Chassis'; transientParameterDescription = 'vCar2 Demo'; transientParameterGroups = ''; transientParameterIdentifier = 'vCar2 Demo'; transientParameterMaximum = 800; transientParameterMinimum = 0; transientParameterName = 'vCar2 Demo'; ``` ## Create Transient Parameter The transient parameter is created and registered within the current ATLAS session. A transient parameter is temporary and used to store calculated or transformed data during the session. This step ensures the system is ready to receive new data. === "C#" ``` csharp var transientParameter = sessionServiceClient.AddTransientParameter( sessionId, transientParameterIdentifier, demoDisplayName, transientParameterDescription, transientParameterGroups.Split(';'), transientParameterMinimum, transientParameterMaximum ); ``` === "Python" ``` python transient_parameter = sessionServiceClient.AddTransientParameter( sessionId, TransientParameterIdentifier, DemoDisplayName, TransientParameterDescription, TransientParameterGroups.split(";"), TransientParameterMinimum, TransientParameterMaximum ) ``` === "MATLAB" ``` matlab transientParameter = sessionServiceClient.AddTransientParameter( ... sessionId, ... transientParameterIdentifier, ... demoDisplayName, ... transientParameterDescription, ... strsplit(transientParameterGroups, ';'), ... transientParameterMinimum, ... transientParameterMaximum); ``` !!! tip A **transient parameter** is **temporary**—it exists **only while the session is active** in ATLAS. Once the session is **closed or disposed**, the transient parameter and its data are **permanently lost**. ✅ Use transient parameters for: - Intermediate calculations - Temporary visualizations - Session-specific data transformations ❌ Do **not** use them for storing data that needs to persist beyond the session. ## Retrieve Time Base and Source Parameter This step retrieves two critical pieces of information: The time base, which defines the start and end times for the session data. The source parameter (vCar), which contains the original data that will be transformed. These are needed to align the new data with the existing session timeline and to access the correct input values. === "C#" ``` csharp var timebase = sessionServiceClient.GetSessionTimeBase(sessionId); var parameter = sessionServiceClient.GetSessionParameter(sessionId, "vCar"); ``` === "Python" ``` python timebase = sessionServiceClient.GetSessionTimeBase(sessionId) parameter = sessionServiceClient.GetSessionParameter(sessionId, 'vCar') ``` === "MATLAB" ``` matlab timebase = sessionServiceClient.GetSessionTimeBase(sessionId); parameter = sessionServiceClient.GetSessionParameter(sessionId, 'vCar'); ``` ## Estimate Sample Count and Navigate to Start Before reading data, we estimate how many samples exist for the selected parameter within the time base. This helps optimize memory and performance. We also move the internal cursor to the start of the time base to ensure data is read from the beginning. === "C#" ``` csharp int sampleCount = parameterDataAccessServiceClient.GetSamplesCountEstimate( sessionId, parameter.Identifier, timebase.StartTime, timebase.EndTime ); parameterDataAccessServiceClient.Goto(sessionId, parameter.Identifier, timebase.StartTime); ``` === "Python" ``` python sample_count = parameterDataAccessServiceClient.GetSamplesCountEstimate( sessionId, parameter.Identifier, timebase.StartTime, timebase.EndTime ) parameterDataAccessServiceClient.Goto(sessionId, parameter.Identifier, timebase.StartTime) ``` === "MATLAB" ``` matlab sampleCount = parameterDataAccessServiceClient.GetSamplesCountEstimate( ... sessionId, ... parameter.Identifier, ... timebase.StartTime, ... timebase.EndTime); parameterDataAccessServiceClient.Goto(sessionId, parameter.Identifier, timebase.StartTime); ``` ## Retrieve and Transform Data In this step, the original data samples are retrieved from the source parameter. Each sample is checked for validity, and a transformation is applied—in this case, multiplying each value by 2. The transformed values and their corresponding timestamps are stored for later use. === "C#" ``` csharp var timestamps2 = new List(); var values2 = new List(); var values = parameterDataAccessServiceClient.GetNextSamples(sessionId, parameter.Identifier, sampleCount); for (int i = 0; i < values.Data.Length; i++) { if (values.DataStatus[i] == DataStatusType.Sample) { timestamps2.Add(values.Time[i]); values2.Add(values.Data[i] * 2); } } ``` === "Python" ``` python timestamps2 = [] values2 = [] values = parameterDataAccessServiceClient.GetNextSamples(sessionId, parameter.Identifier, sample_count) for status, data, time in zip(values.DataStatus, values.Data, values.Time): if status == DataStatusType.Sample: timestamps2.append(time) values2.append(data * 2) ``` === "MATLAB" ``` matlab timestamps2 = []; values2 = []; values = parameterDataAccessServiceClient.GetNextSamples(sessionId, parameter.Identifier, sampleCount); for i = 1:length(values.Data) if values.DataStatus(i) == DataStatusType.Sample timestamps2(end+1) = values.Time(i); values2(end+1) = values.Data(i) * 2; end end ``` ## Write Transformed Data to Transient Parameter Finally, the transformed data is written to the transient parameter. This makes the new data available within the session for visualization, analysis, or further processing. The timestamps ensure the data aligns correctly with the session timeline. === "C#" ``` csharp sessionServiceClient.AddTimeDataToTransientParameter( sessionId, transientParameter.Identifier, timestamps2.ToArray(), values2.ToArray() ); ``` === "Python" ``` python sessionServiceClient.AddTimeDataToTransientParameter( sessionId, transient_parameter.Identifier, timestamps2, values2 ) ``` === "MATLAB" ``` matlab sessionServiceClient.AddTimeDataToTransientParameter( ... sessionId, ... transientParameter.Identifier, ... timestamps2, ... values2); ```
![Image title](assets/Transient.png){ width="1200" }
ATLAS Transient Parameter
--- # Overview The ATLAS Data layer is provided by the SQL Race API, which provides: === "Data Model" * Session metadata management — including search * Configuration model — which describes available data * Data samples, events, errors and lap markers * Composite Session management === "I/O" * Read ATLAS SSN files * Read and Write SSN2 files and shared MS SQL Server telemetry databases * Record telemetry from Motion Applied hardware, simulations, and third-party protocols * Monitor and extract the live telemetry * Augment the recording with additional data * Read data from third-party files * Pull data from RTA web services === "Compute" * Resampling and Interpolation * Statistics * Virtual Parameters * Functions (C# and FDL, including [processor routines](../../../key-functionality/analyse/viewer/functions/processing.md)) !!! note The ATLAS Automation API also provides data access, but the SQL Race API is more direct and will yield better performance. [Code Samples](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples){ .md-button } [API Reference](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/){ .md-button } ## Prerequisites * Visual Studio 2019 or later * .NET 8 or later ## Code Samples Index [GettingStarted](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/projects/csharp/GettingStarted) : * Searching and loading sessions * Loading a live session recording into a shared database * Loading multiple sessions together as a composite * Loading SSN files and associate sessions * Reading and writing events * Reading and writing data * Calculating lap statistics * Adding and reading [Markers](../../../key-functionality/analyse/viewer/handling-data/laps-markers/markers.md) [StandaloneRecorder](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/projects/csharp/StandaloneRecorder) : * Embedding the ATLAS Data Server Telemetry (DST) recorder * Monitoring live data from the recorder * Writing augmented data back into the live session [FunctionLibrary](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/projects/csharp/FunctionLibrary) : * Writing a [Function DLL](../../../key-functionality/analyse/viewer/functions/using-dlls.md), for use in the API and in ATLAS [MAT.SQLRace.FileLoaderSample](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples) : * Writing a Session Loader DLL to read a third-party file format (CSV example) * This sample was retired as part of the examples repository restructure and no longer has a direct equivalent [Python](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/snippets/python) : * Loading a session * Reading events * Reading data --- # Getting Started ## Prerequisites * Visual Studio 2022 or later (for C#) * MATLAB 2020b or later (for MATLAB) * Python 3.7 or later (for Python) * Python IDE such as PyCharm or Visual Studio Code (for Python) * .NET 8 or later * ATLAS 10 installed on your machine * A valid ATLAS and SQLRace license !!! tip For more information on licenses, please get in touch with your Motion Applied representative or [contact us](mailto:sales@motionapplied.com). ## Environment Setup Select the programming language you want to use for your project. The SQLRace API supports multiple languages, including C#, Python, and MATLAB. Below are the steps to set up your environment for each language. === "C#" 1. Open Visual Studio 2. Create a new project 3. Add the NuGet package: `MAT.OCS.SQLRace.Domain`. For more information on how to add NuGet packages, refer to the [NuGet documentation](../nuget/setup.md). 4. Create a new C# file in the root of your project and copy the sample code 5. Or clone the sample code into the root 6. Change the variables at the top of the script to connect to your database === "Python" 1. Open PyCharm 2. Create a new project 3. Add the package dependency: 'pythonnet' 4. Create a new Python file main.py in the root of your project and copy the sample code 5. Or clone the sample code into the root 6. Change the variables at the top of the script to connect to your database ## Importing the APIs ### C# NuGet Packages The SQLRace API is available as a NuGet package to registered users found here: [Motion Applied NuGet Repository](https://github.com/mat-docs/packages). ### Python and MATLAB For Python and MATLAB, the API DLLs can be directly imported and declared in the code. === "Python" ``` python import os from pythonnet import load load("coreclr", runtime_config=r"C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.Atlas.Host.runtimeconfig.json") import clr # The path to the main SQL Race DLL. This is the default location when installed with Atlas 10 sql_race_dll_path = r"C:\Program Files\McLaren Applied Technologies\ATLAS 10\MESL.SqlRace.Domain.dll" # Configure Pythonnet and reference the required assemblies for dotnet and SQL Race clr.AddReference("System.Collections") clr.AddReference("System.Core") clr.AddReference("System.IO") if not os.path.isfile(sql_race_dll_path): raise Exception("Couldn't find SQL Race DLL at " + sql_race_dll_path + " please check that Atlas 10 is installed") clr.AddReference(sql_race_dll_path) from System.Collections.Generic import * from System.Collections.ObjectModel import * from System import * from MAT.OCS.Core import * from MESL.SqlRace.Domain import * from MESL.SqlRace.Enumerators import * from MESL.SqlRace.Domain.Infrastructure.DataPipeline import * ``` === "MATLAB" ``` matlab NET.addAssembly('C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.OCS.Core.dll'); NET.addAssembly('C:\Program Files\McLaren Applied Technologies\ATLAS 10\MESL.SqlRace.Domain.dll'); NET.addAssembly('C:\Program Files\McLaren Applied Technologies\ATLAS 10\MESL.SqlRace.Matlab.dll'); NET.addAssembly('System.Core'); import System.Collections.Generic.* import System.Collections.ObjectModel.* import System.* import MAT.OCS.Core.* import MESL.SqlRace.Matlab.* import MESL.SqlRace.Domain.* import MESL.SqlRace.Enumerators.* import MESL.SqlRace.Domain.Infrastructure.DataPipeline.* ``` ## Example Code Several example projects are available to help you get started along with walkthrough explanations. These examples cover a range of functionalities provided by the SQLRace API, including reading and writing data, session management, and using functions. For easy access, the examples are hosted on GitHub in the [MAT.OCS.SQLRace.Examples](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples) repository. Each example is structured to demonstrate specific features of the SQLRace API, making it easier for you to understand how to implement them in your own projects. --- # Object Model The SQL Race API is organised around a **`Session`** — the container for time-series data, its configuration, and the structure (laps, markers, events) layered on top. The interactive map below shows the key classes and, crucially, **the actual methods and properties that connect them**, so you can see how a typical workflow flows from one class to the next. !!! tip "How to use the map" * **Click a class** to see what it does, its key members, and links to the API reference and a worked example. * **Hover** a class to highlight its connections. * Use the **legend** to highlight a stage — Connect, Configure, Structure or Read. * Arrow colours: **navigation** (a property), **write** (adds data), and plain **method call**.
> The map is generated from the official > [SQL Race API reference](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/index.html) > (the DocFX `xrefmap`), so every class and every connecting member maps to a real, > current API type and method. ## The four stages A typical end-to-end workflow moves through four stages, which map to the colours in the graph: 1. **Connect** — `SessionManager.CreateSession(...)` or `Load(...)` returns an `IClientSession`; its `.Session` property gives you the `Session`. 2. **Configure** (writing) — build a `ConfigurationSet` (via `Session.CreateConfiguration()` or `ConfigurationSetManager`), add `Channel`s, `Parameter`s, `Conversion`s and groups, then commit it. 3. **Structure** — a `Session` organises data into `Lap`s, `Marker`s and `Event`s, each with their own time ranges. 4. **Read** — open a `ParameterDataAccess` with `Session.CreateParameterDataAccess(...)`, then call `GetNextSamples`, `GetNextData` or `GetLapStatistics` to get `ParameterValues` / statistics back. ## Where to go next | Stage | Worked example | |-------|----------------| | Connect | [Session Loading](examples/session-loading.md) | | Configure | [Create Session](examples/create-session.md) · [Synchro Data](examples/synchro-data.md) | | Read | [Parameter Data Access](examples/parameter-data-access.md) | | Export | [Session Export](examples/session-export.md) | | Calculated channels | [Functions](examples/functions.md) | --- # Create Session This guide walks you through the process of creating a new **session file (`.ssn2`)** using the **SQLRace API**. You'll learn how to initialize the environment, create a session, add metadata, generate parameter data, and define laps. ???+ tip Make sure you have: - Installed **.NET 8** and are targeting **x64 architecture**. - Added the following NuGet packages: - `MESL.SQLRace.API` - Configured your NuGet source to include the **McLaren Applied GitHub packages**: [https://github.com/mat-docs/packages](https://github.com/mat-docs/packages) ## Step 1: Initialize SQLRace This is a required step and scripts will work without an active SQLRace License. You can check that you have the correct license by opening License Manager from your start menu. === "C#" ```csharp Core.LicenceProgramName = "SQLRace"; Core.Initialize(); ``` === "Python" ```python Core.LicenceProgramName = "SQLRace" Core.Initialize() ``` === "MATLAB" ```matlab NET.setStaticProperty('MESL.SqlRace.Domain.Core.LicenceProgramName', 'SQLRace'); Core.Initialize(); ``` ## Step 2: Create a New Session Creates a unique session at the specified location. === "C#" ```csharp var sessionKey = SessionKey.NewKey(); var clientSession = SessionManager.CreateSessionManager() .CreateSession(connectionString, sessionKey, sessionDescription, DateTime.Now, "TAG-310"); ``` === "Python" ```python session_key = SessionKey.NewKey() client_session = SessionManager.CreateSessionManager().CreateSession(connection_string, session_key, session_desc, DateTime.Now, "TAG-310") ``` === "MATLAB" ```matlab sessionKey = SessionKey.NewKey(); clientSession = SessionManager.CreateSessionManager().CreateSession(connectionString, sessionKey, sessionDescription, DateTime.Now, "TAG-310"); ``` [SessionManager.CreateSession()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.SessionManager.html#MESL_SqlRace_Domain_SessionManager_CreateSession_System_String_MAT_OCS_Core_SessionKey_System_String_System_DateTime_System_String_) !!! info Use the following connection strings to create your session, whether it's SSN2, SSNDB, or SQLRace: connectionStringSSN2 = r"DbEngine=SQLite;Data Source=C:\Path\To\session.ssn2;" connectionStringSSNDB = r"DbEngine=SQLite;Data Source=C:\Path\To\local_db.ssndb;PRAGMA journal_mode=WAL;" connectionStringSQLRACE = r"Data Source=MCLA-42765X3\LOCAL;Initial Catalog=SQLRACE01;Integrated Security=True;" ## Step 3: Create Session Configuration Create a configuration set for the session. Configuration will be saved when Commit() is called on this configuration set. If a configuration set is not created then the parameters will not be part of any structured groups. !!! tip Configuration can be created independently and shared across multiple sessions (non-session specific) using the `ConfigurationSetManager`. This improves session creation and load time. !!! warning "Deprecated Method" Configuration can also be added directly to a session using `CreateConfiguration()`. This method is **deprecated** and should **not** be used. ???+ "Configuration Set Components" A **Configuration Set** is composed of the following entities: - Parameters - Channels - Conversions - Rational conversions - Table conversions - Text conversions - Application Groups - Parameter Groups - Error Definitions - Event Definitions The following is an example of using the ConfigurationSetManager to add a shared configuration set: === "C#" ```csharp // Define names and IDs string parameterName = "myParam"; string applicationGroupName = "myApp"; string parameterIdentifier = $"{parameterName}:{applicationGroupName}"; string conversionFunctionName = $"CONV_{parameterName}:{applicationGroupName}"; string parameterGroupIdentifier = "myParamGroup"; string parameterSubGroupIdentifier = "myParamSubGroup"; uint parameterNameChannelId = 999998; int applicationGroupId = 998; // Create configuration set ConfigurationSetManager configurationSetManager = ConfigurationSetManager.CreateConfigurationSetManager(); var configurationSet = configurationSetManager.Create(connectionString, "Configuration Set Name", "Configuration Set Description"); // Create parameter group and add sub group var parameterGroup = new ParameterGroup(parameterGroupIdentifier, "Parameter group description"); parameterGroup.AddSubGroup(new ParameterGroup(parameterSubGroupIdentifier, "Sub group description")); // Create application group var applicationGroup = new ApplicationGroup(applicationGroupName, "Application group description", applicationGroupId, new List { parameterGroup.Identifier }); // Add application group and parameter group to configuration set configurationSet.AddGroup(applicationGroup); configurationSet.AddParameterGroup(parameterGroup); // Add a conversion function to the configuration set configurationSet.AddConversion(RationalConversion.CreateSimple1To1Conversion(conversionFunctionName, "Units", "5.2f")); // Create a frequency for the parameter's channel var myParamFrequency = new Frequency(1000, FrequencyUnit.Hz); // Create a channel for the parameter var myParamChannel = new Channel(parameterNameChannelId, "Parameter channel description", myParamFrequency.ToInterval(), DataType.Double64Bit, ChannelDataSourceType.Periodic); // Create the parameter var myParam = new Parameter(parameterIdentifier, parameterName, $"{parameterName} Description", 1, -1, 0, 0, 0, 255, 0, conversionFunctionName, new List { parameterSubGroupIdentifier }, myParamChannel.Id, applicationGroupName); // Add the channel and parameter to the configuration set configurationSet.AddChannel(myParamChannel); configurationSet.AddParameter(myParam); // Commit the configuration set to the database and handle exception try { configurationSet.Commit(); Console.WriteLine($"Configuration Set '{configurationSet.Identifier}' committed to database."); } catch (ConfigurationSetAlreadyExistsException ex) { Console.WriteLine($"Commit of a Configuration Set '{configurationSet.Identifier}' failed. The config set already exists."); } ``` === "Python" ```python # Define names and IDs parameter_name = "myParam" application_group_name = "myApp" parameter_identifier = f"{parameter_name}:{application_group_name}" conversion_function_name = f"CONV_{parameter_name}:{application_group_name}" parameter_group_identifier = "myParamGroup" parameter_sub_group_identifier = "myParamSubGroup" parameter_name_channel_id = 999998 # Unique to each channel application_group_id = 998 # Unique to each application group # Define .NET Lists parameter_group_identifier_list = List[str]() parameter_sub_group_identifier_list = List[str]() # Add variables to the .NET identifier lists parameter_group_identifier_list.Add(parameter_group_identifier) parameter_sub_group_identifier_list.Add(parameter_sub_group_identifier) # Create configuration set configuration_set = configuration_set_manager.Create(connection_string, "Configuration Set Name", "Configuration Set Description") # Create parameter group and add sub-group parameter_group = ParameterGroup(parameter_group_identifier, "Parameter group description") parameter_group.AddSubGroup(ParameterGroup(parameter_sub_group_identifier, "Sub group description")) # Create the application group appGroup = ApplicationGroup(application_group_name, "Application group description", application_group_id, parameter_group_identifier_list) # Add the application group and parameter group to the configuration set configuration_set.AddGroup(appGroup) configuration_set.AddParameterGroup(parameter_group) # Add the conversion function to the configuration set configuration_set.AddConversion(RationalConversion.CreateSimple1To1Conversion(conversion_function_name, "Unit", "5.2f")) # Create a frequency for the parameter's channel myParam_frequency = Frequency(1000, FrequencyUnit.Hz) # Create a channel for the parameter myParam_channel = Channel(parameter_name_channel_id, "Parameter channel description", FrequencyExtensions.ToInterval(myParam_frequency), DataType.Double64Bit, ChannelDataSourceType.Periodic) # Create the parameter myParam = Parameter(parameter_identifier, parameter_name, f"{parameter_name} Description", 1, -1, 0, 0, 0, 255, 0, conversion_function_name, parameter_sub_group_identifier_list, myParam_channel.Id, application_group_name) # Add the channel and parameter to the configuration set configuration_set.AddChannel(myParam_channel) configuration_set.AddParameter(myParam) # Commit the configuration set to the database and handle exception try: configuration_set.Commit() print(f"Configuration Set '{configuration_set.Identifier}' committed to database.") except ConfigurationSetAlreadyExistsException as ex: print(f"Commit of a Configuration Set '{configuration_set.Identifier}' failed. The config set already exists.") ``` === "MATLAB" ```matlab %% Create configuration set % Define Names and IDs parameterName = "myParam"; applicationGroupName = "myApp"; parameterIdentifier = sprintf("%s:%s", parameterName, applicationGroupName); conversionFunctionName = sprintf("CONV_%s:%s", parameterName, applicationGroupName); parameterGroupIdentifier = "myParamGroup"; parameterSubGroupIdentifier = "MyParamSubGroup"; parameterNameChannelId = 999998; applicationGroupId = 998; % Define .NET lists parameterGroupIdentifierList = NET.createGeneric('System.Collections.Generic.List', {'System.String'}); parameterSubGroupIdentifierList = NET.createGeneric('System.Collections.Generic.List', {'System.String'}); % Add variable to the .NET identifier lists parameterGroupIdentifierList.Add(parameterGroupIdentifier); parameterSubGroupIdentifierList.Add(parameterSubGroupIdentifier); % Create a configuration set configurationSetManager = ConfigurationSetManager.CreateConfigurationSetManager(); configurationSet = configurationSetManager.Create(connectionString, "Configuration Set Name", "Configuration Set Description"); % Create a parameter group and sub group parameterGroup = ParameterGroup(parameterGroupIdentifier, "Parameter group description"); parameterGroup.AddSubGroup(ParameterGroup(parameterSubGroupIdentifier, "Sub group description")); % Create application group applicationGroup = ApplicationGroup(applicationGroupName, "Application Group Name", applicationGroupId, parameterGroupIdentifierList); % Add application group and parameter group to configuration set configurationSet.AddGroup(applicationGroup); configurationSet.AddParameterGroup(parameterGroup); % Add a conversion function to the configuration set configurationSet.AddConversion(RationalConversion.CreateSimple1To1Conversion(conversionFunctionName, "Units", "5.2f")); % Create a frequency for the parameter's channel myParamFrequency = Frequency(1000, FrequencyUnit.Hz); % Create a channel for the parameter myParamChannel = Channel(parameterNameChannelId, "Parameter channel description", FrequencyExtensions.ToInterval(myParamFrequency), DataType.Double64Bit, ChannelDataSourceType.Periodic); % Create the parameter myParam = Parameter(parameterIdentifier, parameterName, sprintf("%s Description", parameterName), 1, -1, 0, 0, 0, 255, 0, conversionFunctionName, parameterSubGroupIdentifierList, myParamChannel.Id, applicationGroupName); % Add the channel and parameter to the configuration set configurationSet.AddChannel(myParamChannel); configurationSet.AddParameter(myParam); % Commit the configuration set to the database and handle exception try configurationSet.Commit(); fprintf("Configuration Set '%s' committed to database.\n", configurationSet.Identifier); catch errorMessage fprintf("Encountered an error: %s\n", errorMessage.message); end ``` [ConfigurationSetManager.Create()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSetManager.html#MESL_SqlRace_Domain_ConfigurationSetManager_Create_MAT_OCS_Core_SessionKey_System_String_System_String_System_String_System_Boolean_System_Boolean_) ### UseConfigurationSets Method === "C#" ```csharp var metadata = session.UseConfigurationSets(new List> { new("Configuration Set Name", configSetOffset) }); ``` === "Python" ```python kvp = List[KeyValuePair[str, UInt32]]() kvp.Add(KeyValuePair[str, UInt32]("Configuration Set Name", config_set_offset)) metadata = session.UseConfigurationSets(kvp) ``` === "MATLAB" ```matlab kvpType = NET.GenericClass('System.Collections.Generic.KeyValuePair', 'System.String', 'System.UInt32'); kvpList = NET.createGeneric('System.Collections.Generic.List', {kvpType}); kvpList.Add(NET.createGeneric('System.Collections.Generic.KeyValuePair', {'System.String', 'System.UInt32'}, "Configuration Set Name", configSetOffset)); metadata = session.UseConfigurationSets(kvpList); ``` [Session.UseConfigurationSets()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Session.html#MESL_SqlRace_Domain_Session_UseConfigurationSets_System_Collections_Generic_IEnumerable_System_Collections_Generic_KeyValuePair_System_String_System_UInt32___) ## Step 4: Add Session Metadata Metadata allows tracking of essential session attributes (e.g., driver, event, date, etc.). Session metadata can be configured adding **SessionDataItem** objects into the **session.Items**. !!! tip The **SessionDataItem** supports the following data types for values: - String – For textual data - Double – For floating-point numbers - Long – For large integer values - Boolean – For true/false flags - DateTime – For date and time values - Byte[] – For binary data === "C#" ```csharp session.Items.Add(new SessionDataItem("Driver Name", "Driver xxxxx")); session.Items.Add(new SessionDataItem("Race", "Silverstone GP")); session.Items.Add(new SessionDataItem("Date", DateTime.Now.AddDays(7))); session.Items.Add(new SessionDataItem("Fuel level in gallons", 56.45)); ``` === "Python" ```python session.Items.Add(SessionDataItem("Driver Name", "Driver xxxxx")) session.Items.Add(SessionDataItem("Race", "Silverstone GP")) session.Items.Add(SessionDataItem("Date", DateTime.Now.AddDays(7))) session.Items.Add(SessionDataItem("Fuel level in gallons", 56.45)) ``` === "MATLAB" ```matlab session.Items.Add(SessionDataItem("Driver Name", "Driver xxxxx")); session.Items.Add(SessionDataItem("Race", "Silverstone GP")); session.Items.Add(SessionDataItem("Date", DateTime.Now.AddDays(7))); session.Items.Add(SessionDataItem("Fuel level in gallons", 56.45)); ``` [Class SessionDataItemsCollection](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Collections.SessionDataItemsCollection-1.html) ## Step 5: Configure Conversions Add conversion logic (e.g., rational and table conversions) for later use by parameters. === "C#" ```csharp var conversions = new List { new RationalConversion("Rational_conv", "mph", "%5.2f", 0, 1, 0, 0, 0, 1), new TableConversion("Table_conv", "mph", "%5.2f", rawValues, calibratedValues, false) }; config.AddConversions(conversions); ``` === "Python" ```python numvalues = Array[Double]([0.0, 1.0]) stringvalues = Array[String](['Yes', 'No']) textConvertion = TextConversion.Create(ConversionFunctionName, "kph", "%s", [0.0, 1.0], ['Yes', 'No'], "No") config.AddConversion(RationalConversion.CreateSimple1To1Conversion(ConversionFunctionName, "kph", "%5.2f")) formula = '(floor($fconv_dummy/60))+((($fconv_dummy/60)-(floor($fconv_dummy/60)))*0.6)' expresion = IExpressionAdapter config.AddConversion(FormulaConversion.Create('CONV_MYCONVERSION_NAME:TAG320BIOS', 'kph', "%s", '%2.2f', formula, IExpressionAdapter)) ``` === "MATLAB" ```matlab configurationSet.AddConversion( RationalConversion.CreateSimple1To1Conversion( conversionFunctionName, "Units", "5.2f")); ``` [ConfigurationSet.AddConversion()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSet.html#MESL_SqlRace_Domain_ConfigurationSet_AddConversion_MESL_SqlRace_Domain_ConversionBase_) [Class ConversionBase](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConversionBase.html) ## Step 6: Add Channels and Parameters Set up channels and map them to parameters that will receive time-series data. For more information on channels and parameters, see [Understanding Channels and Parameters](../../../../key-functionality/configure/system-monitor/data-acquisition/logging/logging-config.md). === "C#" ```csharp var channel = new Channel(1, "lowFrequencyPeriodic1", 100_000_000, DataType.FloatingPoint32Bit, ChannelDataSourceType.Periodic); var parameter = new Parameter("lowFrequency_10Hz", "lowFrequency_10Hz", "Low frequency parameter", 200, 0, 200, 0, 0, 0xffffffff, 0, "Rational_conv", new List { "chassis" }, channel.Id); config.AddChannels(new List { channel }); config.AddParameters(new List { parameter }); ``` === "Python" ```python # Create a channel for the parameter myParam_channel = Channel( parameter_name_channel_id, "Parameter channel description", FrequencyExtensions.ToInterval(myParam_frequency), DataType.Double64Bit, ChannelDataSourceType.Periodic) # Create the parameter myParam = Parameter( parameter_identifier, parameter_name, f"{parameter_name} Description", 1, # Max value -1, # Min value 0, # Max Warning value 0, # Min Warning value 0, # Offset value 255, # Data Bit Mask 0, # Error Bit Mask conversion_function_name, parameter_sub_group_identifier_list, myParam_channel.Id, application_group_name) # Add the channel and parameter to the configuration set configuration_set.AddChannel(myParam_channel) configuration_set.AddParameter(myParam) ``` === "MATLAB" ```matlab % Create a channel for the parameter myParamChannel = Channel( parameterNameChannelId, "Parameter channel description", FrequencyExtensions.ToInterval(myParamFrequency), DataType.Double64Bit, ChannelDataSourceType.Periodic); % Create the parameter myParam = Parameter( parameterIdentifier, parameterName, sprintf("%s Description", parameterName), 1, -1, 0, 0, 0, 255, 0, conversionFunctionName, parameterSubGroupIdentifierList, myParamChannel.Id, applicationGroupName); % Add the channel and parameter to the configuration set configurationSet.AddChannel(myParamChannel); configurationSet.AddParameter(myParam); ``` [Class Channel](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Channel.html) [Class Parameter](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Parameter.html) [ConfigurationSet.AddChannel()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSet.html#MESL_SqlRace_Domain_ConfigurationSet_AddChannel_MESL_SqlRace_Domain_IChannel_) [ConfigurationSet.AddParameter()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSet.html#MESL_SqlRace_Domain_ConfigurationSet_AddParameter_MESL_SqlRace_Domain_ParameterBase_) ??? note "Available Data Types" | **Data Type** | **Description** | |------------------------------|--------------------------------------| | `Double64Bit` | Double 64 bits of data. | | `FloatingPoint32Bit` | Floating point 32 bits of data. | | `Signed16Bit` | Signed 16 bits of data. | | `Signed32Bit` | Signed 32 bits of data. | | `Signed8Bit` | Signed 8 bits of data. | | `TripleFloatingPoint32Bit` | Triple Floating point values. | | `Unsigned16Bit` | Unsigned 16 bits of data. | | `Unsigned32Bit` | Unsigned 32 bits of data. | | `Unsigned8Bit` | Unsigned 8 bits of data. | ## Step 7: Define Event Metadata Events let you log significant occurrences during a session (e.g., start, end, pit stops). === "C#" ```csharp var eventDefinitions = new List { new EventDefinition(1, "FirstEvent", EventPriorityType.High, new List { "Rational_conv" }, "Group1"), new EventDefinition(2, "SecondEvent", EventPriorityType.Low, new List { "Rational_conv" }, "Group1"), new EventDefinition(3, "LastEvent", EventPriorityType.Medium, new List { "Rational_conv" }, "Group2") }; config.AddEventDefinitions(eventDefinitions); ``` === "Python" ```python event_definitions = List[EventDefinition]() event_definitions.Add(EventDefinition(1, "FirstEvent", EventPriorityType.High, List[String]().Add("Rational_conv"), "Group1")) event_definitions.Add(EventDefinition(2, "SecondEvent", EventPriorityType.Medium, List[String]().Add("Rational_conv"), "Group1")) event_definitions.Add(EventDefinition(3, "ThirdEvent", EventPriorityType.Low, List[String]().Add("Rational_conv"), "Group2")) configuration_set.AddEventDefinitions(event_definitions) ``` === "MATLAB" ```matlab % Get EventDefinition type eventDefinitionType = System.Type.GetType('MESL.SqlRace.Domain.EventDefinition, MESL.SqlRace.Domain'); % Add EventDefinition as argument for List listOfEventDefinitionTypeArgs = NET.createArray('System.Type', 1); listOfEventDefinitionTypeArgs.Set(0, eventDefinitionType); listOfEventDefinitionType = listTypeDef.MakeGenericType(listOfEventDefinitionTypeArgs); % Create List and List eventDefinitions = System.Activator.CreateInstance(listOfEventDefinitionType); convFuncNames = System.Activator.CreateInstance(listOfStringType); % Add conversion function name to List convFuncNames.Add("Rational_conv"); % Add event definitions to List eventDefinitions.Add(EventDefinition(1, "FirstEvent", EventPriorityType.High, convFuncNames, "Group1")); eventDefinitions.Add(EventDefinition(2, "SecondEvent", EventPriorityType.Medium, convFuncNames, "Group1")); eventDefinitions.Add(EventDefinition(3, "ThirdEvent", EventPriorityType.Low, convFuncNames, "Group2")); % Add event definitions to configuration set configurationSet.AddEventDefinitions(eventDefinitions); ``` [Class EventDefinition](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.EventDefinition.html) [ConfigurationSet.AddEventDefinition()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSet.html#MESL_SqlRace_Domain_ConfigurationSet_AddEventDefinition_MESL_SqlRace_Domain_EventDefinition_) ## Step 8: Write Parameter and Event Data Adds time-stamped numeric data to your session’s channels and logs events using IDs and timestamps. === "C#" ```csharp session.AddChannelData(channel.Id, timestamp, sampleCount, doubleDataBytes); session.Events.AddEventData(1, "Group1", timestamp, new List { 300, 200, 100 }, true, "Status Text"); ``` === "Python" ```python session.AddChannelData( channel.Id, timestamp, sample_count, struct.pack('](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Session.html#MESL_SqlRace_Domain_Session_Constants) [Class Constant](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Constant.html) ## Step 11: Add CAN Data CAN Data can be added thogh the CAN Collection object. CAN Collection only accepts RAW CAN messages. === "C#" ```csharp long timeStamp = 1627845123456; byte isReceived = 1; ushort busId = 99; uint canId = 123456; byte[] message = new byte[] { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 }; session.CanData.AddCanData(timeStamp, isReceived, busId, canId, message); ``` === "Python" ```python timeStamp = Int64(1627845123456) isReceived = Byte(1) # 1 for received, 0 for transmitted busId = UInt16(99) canId = UInt32(123456) message = Array.CreateInstance(Byte, 8) message[0] = Byte(0x01) message[1] = Byte(0x02) message[2] = Byte(0x03) message[3] = Byte(0x04) message[4] = Byte(0x05) message[5] = Byte(0x06) message[6] = Byte(0x07) message[7] = Byte(0x08) session.CanData.AddCanData(timeStamp, isReceived, busId, canId, message) ``` === "MATLAB" ```matlab timeStamp = int64(1627845123456); isReceived = uint8(1); busId = uint16(99); canId = uint32(123456); message = NET.createArray('System.Byte', 8); message(1) = uint8(1); message(2) = uint8(2); message(3) = uint8(3); message(4) = uint8(4); message(5) = uint8(5); message(6) = uint8(6); message(7) = uint8(7); message(8) = uint8(8); session.CanData.AddCanData(timeStamp, isReceived, busId, canId, message); ``` [Session.CanData](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Session.html#MESL_SqlRace_Domain_Session_CanData) [Session.CanData.AddCanData()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Collections.CanCollection.html#MESL_SqlRace_Domain_Collections_CanCollection_AddCanData_System_Int64_System_Byte_System_UInt16_System_UInt32_System_Byte___) !!! warning Only SSN2 and SQLRace Sessions support CAN data, SSN sessions do not contain any CAN data. !!! tip Sessions generated from a TAG-320 or TAG-700 will only contain CAN data from the PCU-8D or PCU-700. ## Step 12: Finalize Session Close the session to ensure all data is written and saved to the `.ssn2` file. === "C#" ```csharp session.EndData(); clientSession.Close(); ``` === "Python" ```python session.EndData() client_session.Close() ``` === "MATLAB" ```matlab session.EndData(); clientSession.Close(); ``` [Session.EndData()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Session.html#MESL_SqlRace_Domain_Session_EndData) [ClientSession.Close()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ClientSession.html#MESL_SqlRace_Domain_ClientSession_Close) !!! warning EndData() is needed for the session's state to be properly updated to 'Historical', otherwise it would be stuck as 'LiveNotInServer' or 'Live'. Close() or Dispose() is used to free system resources and reduce memory leaks. ## Summary By following these steps, you can build a comprehensive `.ssn2` session file with: - Structured metadata - Parameter/channel setup - Event definitions and logging - Sampled and synchronized data - Lap and constant annotation Explore further via [SQLRace API documentation](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/index.html). Additional examples in the GitHub repository [create-session-write-data.cs](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/blob/master/snippets/csharp/getting-started/04-create-session-write-data.cs) & [GettingStarted](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/projects/csharp/GettingStarted). --- # Synchro Data Synchro (engine-synchronous) channels store one sample per engine event — typically once per revolution or once per firing cycle — rather than at a fixed clock rate. Because engine speed varies continuously, the time between samples changes from one sample to the next, so the timing of every sample must be carried *in the data itself* rather than implied by a fixed channel interval. This guide explains how synchro data is modelled and written with the **SQLRace API**: how to create a synchro channel, how the variable timing is encoded (inter-sample intervals, the *delta scale*, and the binary packet format), how to write packets with `AddSynchroChannelData`, and how to do all of this efficiently at scale. For the general session/configuration lifecycle (initialising SQLRace, creating a session, metadata, laps, finalising), see [Creating an SSN2 Session from Scratch](create-session.md) — this page focuses only on what is different for synchro channels. --- ## When to Use Synchro In ATLAS, **synchro** channels are fundamentally different from time-based (**row**/periodic) channels. They are sampled in the *angular* domain — one value per engine revolution, per TDC, or per firing event — so the effective sample rate rises and falls with RPM. | Scenario | Use Synchro? | Reason | |----------|:---:|--------| | Cylinder pressure (per-revolution) | :material-check: | One sample per crank revolution — rate tracks RPM | | Exhaust gas temperature (per-cylinder firing) | :material-check: | Sampled at TDC of each cylinder — irregular in time | | Lambda / AFR from wideband sensor | :material-check: | Logged once per firing cycle, tied to engine speed | | Knock intensity per cylinder | :material-check: | One measurement per combustion event | | Injector pulse width | :material-check: | One value per injection event, rate varies with RPM | | CAN-triggered gear position | :material-check: | Messages arrive only on gear change — completely aperiodic | | Chassis accelerometers at fixed 1 kHz | :material-close: | Constant rate — use a standard row channel | | Damper potentiometers at fixed 500 Hz | :material-close: | Constant rate — use a standard row channel | A single session can freely mix row channels (fixed-rate chassis sensors) and synchro channels (per-revolution engine sensors); ATLAS interpolates between the two domains automatically when overlaying traces. --- ## The Synchro Write Pipeline Writing a synchro parameter involves two distinct phases: 1. **Configuration** (once per parameter) — create a configuration set containing a parameter group, application group, conversion, a **synchro** channel, and the parameter, then commit and activate it. This is the same machinery as any other parameter, with one difference: the channel uses `ChannelDataSourceType.Synchro`. 2. **Data** (repeated) — for each block of samples: difference the timestamps into intervals, compute a *delta scale*, pack samples and scaled intervals into a binary payload, and write it with `AddSynchroChannelData`. ### Step 1 — Create the configuration set Every parameter belongs to a configuration set — a collection of channels, conversions, and groups that describes the structure of the data. === "C#" ```csharp var configMgr = ConfigurationSetManager.CreateConfigurationSetManager(); var config = configMgr.Create(session.ConnectionString, configId, "Synchro config"); ``` === "Python" ```python config_mgr = ConfigurationSetManager.CreateConfigurationSetManager() config = config_mgr.Create(session.ConnectionString, config_id, "Synchro config") ``` === "MATLAB" ```matlab configMgr = ConfigurationSetManager.CreateConfigurationSetManager(); config = configMgr.Create(session.ConnectionString, configId, "Synchro config"); ``` ### Step 2 — Define the parameter group and application group A **parameter group** is a logical grouping of related parameters (e.g. `CombustionGroup`). An **application group** ties one or more parameter groups to an application context (e.g. `Engine`) and controls how ATLAS organises parameters in its browser tree. Synchro channels do not use RDA, so set `SupportsRda = false`. === "C#" ```csharp var group = new ParameterGroup("CombustionGroup", "CombustionGroup"); config.AddParameterGroup(group); var groupIds = new List { group.Identifier }; var app = new ApplicationGroup("Engine", "Engine", null, groupIds) { SupportsRda = false }; config.AddGroup(app); ``` === "Python" ```python group = ParameterGroup("CombustionGroup", "CombustionGroup") config.AddParameterGroup(group) group_ids = List[String]() group_ids.Add(group.Identifier) app = ApplicationGroup("Engine", "Engine", None, group_ids) app.SupportsRda = False # synchro channels do not use RDA config.AddGroup(app) ``` === "MATLAB" ```matlab group = ParameterGroup("CombustionGroup", "CombustionGroup"); config.AddParameterGroup(group); groupIds = NET.createGeneric('System.Collections.Generic.List', {'System.String'}); groupIds.Add(group.Identifier); app = ApplicationGroup("Engine", "Engine", [], groupIds); app.SupportsRda = false; % synchro channels do not use RDA config.AddGroup(app); ``` ### Step 3 — Add a conversion A **conversion** maps raw channel values to engineering units. For synchro channels carrying `float64` data, a simple 1:1 rational conversion is sufficient — the raw value equals the engineering value. The conversion also carries the display format and unit. === "C#" ```csharp var conversion = RationalConversion.CreateSimple1To1Conversion( "CONV_CylPressure:Engine", // conversion name "bar", // engineering unit "%5.2f"); // display format string config.AddConversion(conversion); ``` === "Python" ```python conversion = RationalConversion.CreateSimple1To1Conversion( "CONV_CylPressure:Engine", # conversion name "bar", # engineering unit "%5.2f") # display format string config.AddConversion(conversion) ``` === "MATLAB" ```matlab conversion = RationalConversion.CreateSimple1To1Conversion( ... "CONV_CylPressure:Engine", ... % conversion name "bar", ... % engineering unit "%5.2f"); % display format string config.AddConversion(conversion); ``` ### Step 4 — Create the synchro channel This is the one step unique to synchro data. The channel is created with **`ChannelDataSourceType.Synchro`** (rather than `Periodic`/`RowData`). The interval argument is unused for synchro channels — pass `0`. Synchro channels typically store `Double64Bit` values. === "C#" ```csharp uint channelId = session.ReserveNextAvailableRowChannelId(); var channel = new Channel( channelId, "CylPressure_SynchroChannel", 0, // interval — unused for synchro DataType.Double64Bit, ChannelDataSourceType.Synchro); // marks this as a synchro channel config.AddChannel(channel); ``` === "Python" ```python channel_id = session.ReserveNextAvailableRowChannelId() channel = Channel( channel_id, "CylPressure_SynchroChannel", 0, # interval — unused for synchro DataType.Double64Bit, ChannelDataSourceType.Synchro) # marks this as a synchro channel config.AddChannel(channel) ``` === "MATLAB" ```matlab channelId = session.ReserveNextAvailableRowChannelId(); channel = Channel( ... channelId, ... "CylPressure_SynchroChannel", ... 0, ... % interval — unused for synchro DataType.Double64Bit, ... ChannelDataSourceType.Synchro); % marks this as a synchro channel config.AddChannel(channel); ``` ### Step 5 — Create the parameter The **parameter** binds the channel, the conversion, and the group hierarchy together, and carries the display/warning limits ATLAS uses for scaling traces. === "C#" ```csharp var parameter = new Parameter( "CylPressure:Engine", // identifier (Name:AppGroup) "CylPressure", // display name "Peak cylinder pressure", // description 120, 0, // display max, display min 120, 0, // warning max, warning min 0, // offset 0xffffffff, // data bit mask 0, // error bit mask "CONV_CylPressure:Engine", // conversion name new List { "CombustionGroup" }, channelId); config.AddParameter(parameter); ``` === "Python" ```python parameter = Parameter( "CylPressure:Engine", # identifier (Name:AppGroup) "CylPressure", # display name "Peak cylinder pressure", # description 120, 0, # display max, display min 120, 0, # warning max, warning min 0, # offset 0xffffffff, # data bit mask 0, # error bit mask "CONV_CylPressure:Engine", # conversion name group_ids, # parameter group identifiers channel_id) config.AddParameter(parameter) ``` === "MATLAB" ```matlab parameter = Parameter( ... "CylPressure:Engine", ... % identifier (Name:AppGroup) "CylPressure", ... % display name "Peak cylinder pressure", ... % description 120, 0, ... % display max, display min 120, 0, ... % warning max, warning min 0, ... % offset uint32(hex2dec('ffffffff')), ... % data bit mask 0, ... % error bit mask "CONV_CylPressure:Engine", ... % conversion name groupIds, ... channelId); config.AddParameter(parameter); ``` ### Step 6 — Commit and activate Commit the configuration set to the session store, then activate it so subsequent data writes target the correct channels. === "C#" ```csharp config.Commit(); session.UseLoggingConfigurationSet(config.Identifier); ``` === "Python" ```python config.Commit() session.UseLoggingConfigurationSet(config.Identifier) ``` === "MATLAB" ```matlab config.Commit(); session.UseLoggingConfigurationSet(config.Identifier); ``` [Class Channel](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Channel.html) · [Class Parameter](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Parameter.html) · [ConfigurationSetManager](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.ConfigurationSetManager.html) --- ## Encoding the Timing With row data, the interval is constant and declared once on the channel — 1 ms for 1 kHz, 2 ms for 500 Hz. With synchro data, every gap between consecutive samples can differ, and that timing must be encoded into the payload. The chart below makes this concrete. Each sample is drawn as a **stem**, with a **Δt ruler** measuring the gap between consecutive samples. Switch between the **time domain** and the **per-rev (event) domain**, and try the different RPM profiles: in time the stems **bunch together as RPM rises** and spread out as it falls, while in the per-rev domain they stay perfectly uniform. The inter-sample intervals and the per-packet **delta scale** (both covered below) update live.
### Inter-sample intervals Difference the absolute timestamp array to obtain the gap (in nanoseconds) between each pair of consecutive samples. For `N` samples there are `N − 1` intervals. === "C#" ```csharp // timestampsNs: absolute nanosecond timestamps, one per sample long[] intervalsNs = new long[timestampsNs.Length - 1]; for (int i = 0; i < intervalsNs.Length; i++) { intervalsNs[i] = timestampsNs[i + 1] - timestampsNs[i]; } ``` === "Python" ```python # timestamps_ns: absolute nanosecond timestamps, one per sample intervals_ns = np.diff(timestamps_ns) ``` === "MATLAB" ```matlab % timestampsNs: absolute nanosecond timestamps, one per sample intervalsNs = diff(timestampsNs); ``` **Concrete example** — five engine revolutions during acceleration from 6 000 to 10 000 RPM: | Revolution | RPM | Period (60/RPM) | Timestamp (ns) | Interval to next (ns) | |:---:|:---:|:---:|:---:|:---:| | 0 | 6 000 | 10.00 ms | 0 | 9 000 000 | | 1 | 6 667 | 9.00 ms | 9 000 000 | 8 000 000 | | 2 | 7 500 | 8.00 ms | 17 000 000 | 7 000 000 | | 3 | 8 571 | 7.00 ms | 24 000 000 | 6 000 000 | | 4 | 10 000 | 6.00 ms | 30 000 000 | — | The intervals for this packet are `[9000000, 8000000, 7000000, 6000000]`. ### The delta scale Within a packet, intervals are stored as **`uint16`** values (0–65 535) to keep the payload compact. To fit nanosecond intervals into two bytes, each interval is divided by a per-packet **delta scale** — the *greatest common divisor* (GCD) of all intervals in that packet. ``` intervals_ns = [9 000 000, 8 000 000, 7 000 000, 6 000 000] delta_scale = gcd(9000000, 8000000, 7000000, 6000000) = 1 000 000 scaled = intervals_ns / delta_scale = [9, 8, 7, 6] ← fit easily in uint16 ``` SQLRace receives the `delta_scale` alongside the packet and reconstructs the original timing by multiplying each stored interval back up: `9 × 1 000 000 = 9 000 000 ns`. Every packet is self-describing — **each packet has its own delta scale.** === "C#" ```csharp static long Gcd(long a, long b) { while (b != 0) { (a, b) = (b, a % b); } return a; } long deltaScale = intervalsNs[0]; foreach (long v in intervalsNs) { deltaScale = Gcd(deltaScale, v); } if (deltaScale == 0) { deltaScale = 1; } ``` === "Python" ```python # np.gcd.reduce is a vectorised C-level reduction (far faster than a Python loop) delta_scale = int(np.gcd.reduce(intervals_ns)) if delta_scale == 0: delta_scale = 1 ``` === "MATLAB" ```matlab deltaScale = intervalsNs(1); for v = intervalsNs deltaScale = gcd(deltaScale, v); end if deltaScale == 0 deltaScale = 1; end ``` **Three scenarios that show how the delta scale behaves:** === "Constant RPM" At steady 8 000 RPM, every revolution takes exactly 7 500 000 ns: ``` intervals = [7500000, 7500000, 7500000, 7500000] delta_scale = 7 500 000 scaled = [1, 1, 1, 1] ← maximum compression ``` === "Smoothly varying RPM" Acceleration from 6 000 to 10 000 RPM (the running example): ``` intervals = [9000000, 8000000, 7000000, 6000000] delta_scale = 1 000 000 scaled = [9, 8, 7, 6] ``` === "Highly irregular (misfire)" A misfire causes one abnormally long interval: ``` intervals = [7500000, 7500000, 22173000, 7500000] delta_scale = gcd(...) = 3000 scaled = [2500, 2500, 7391, 2500] ← larger, but still < 65 535 ``` !!! warning "When the delta scale collapses to 1" If the GCD of a packet's intervals is 1 nanosecond (the intervals share no common factor), then `delta_scale = 1` and the scaled intervals equal the raw nanosecond values — which will almost certainly exceed 65 535 and make the packet impossible to encode. This is common with timestamps derived from floating-point arithmetic (e.g. `60e9 / rpm`). **Quantise intervals to microsecond resolution before computing the GCD** to restore a healthy common factor (see [Interval quantisation](#interval-quantisation) below). ### Splitting into packets SQLRace does not accept a single monolithic block of synchro data — it expects **packets**, each with its own base timestamp, delta scale, and byte payload. Two constraints force packet boundaries: - **Payload size** — keep each payload comfortably under ~1 MB (see the [silent-discard warning](#performance-at-scale)). A good target is 16 000–48 000 samples per packet. - **Encoding** — within a packet, `max_interval / delta_scale` must be ≤ 65 535. If a packet spans an extreme RPM range (e.g. a multi-second stall mixed with high-RPM data), this can overflow; split that packet until every one satisfies the constraint. A simple, robust algorithm: ``` 1. Divide all N samples into chunks of ≤ packet_size (e.g. 32 000) 2. For each chunk: a. Compute its intervals and their GCD (delta scale) b. If max_interval / delta_scale ≤ 65 535 → emit as a packet c. Otherwise → split the chunk in half and re-check both halves 3. Repeat until every chunk passes ``` --- ## The Binary Payload Format Each packet is serialised into a contiguous byte buffer that **interleaves** sample values and scaled intervals in strict alternation. The final sample has no trailing interval (there is no "next" sample to measure a gap to). ``` Byte offset Content Size ──────────── ──────────────────────────────── ──── 0 sample₀ (float64, little-endian) 8 bytes 8 interval₀ (uint16, little-endian) 2 bytes 10 sample₁ (float64, little-endian) 8 bytes 18 interval₁ (uint16, little-endian) 2 bytes ... (N-1)×10 sampleₙ₋₁ (float64, little-endian) 8 bytes ← LAST sample, NO interval ``` **Total payload size** for `N` samples: `(N − 1) × 10 + 8` bytes. **Worked byte-level example** with the 5-revolution data (pressures `[45.2, 52.1, 58.7, 63.4, 70.0]`, scaled intervals `[9, 8, 7, 6]`): ``` Bytes 0–7: 45.2 float64 LE → 9A 99 99 99 99 99 46 40 Bytes 8–9: 9 uint16 LE → 09 00 Bytes 10–17: 52.1 float64 LE → CD CC CC CC CC 0C 4A 40 Bytes 18–19: 8 uint16 LE → 08 00 Bytes 20–27: 58.7 float64 LE → 66 66 66 66 66 56 4D 40 Bytes 28–29: 7 uint16 LE → 07 00 Bytes 30–37: 63.4 float64 LE → CD CC CC CC CC CC 4F 40 Bytes 38–39: 6 uint16 LE → 06 00 Bytes 40–47: 70.0 float64 LE → 00 00 00 00 00 80 51 40 Total: 48 bytes (5 samples × 10 − 2 = 48) ``` Building the payload: === "C#" ```csharp // samples: double[N], scaled: ushort[N-1] using var ms = new MemoryStream((samples.Length - 1) * 10 + 8); using var bw = new BinaryWriter(ms); // BinaryWriter is little-endian for (int i = 0; i < samples.Length - 1; i++) { bw.Write(samples[i]); // 8-byte float64 bw.Write(scaled[i]); // 2-byte uint16 } bw.Write(samples[^1]); // final sample, no interval byte[] payload = ms.ToArray(); ``` === "Python" ```python # Vectorised packing with a NumPy structured array — no per-sample loop. paired_dtype = np.dtype([("sample", " ~1 MB | > ~100 000 | :material-close: **Silently dropped** | Keep packets at 16 000–48 000 samples to stay well clear of this threshold. ### Optimise packet size Each `AddSynchroChannelData` call carries fixed .NET interop overhead, so fewer, larger packets mean higher throughput — up to a point. Measured on a local SQL Server, single channel, 10 M `float64` samples: | Packet size | Payload | Packets (10 M) | Throughput (measured) | |:---:|:---:|:---:|:---:| | 16 000 | ~156 KB | 625 | ~1.2 M/s | | **32 000** | **~312 KB** | **313** | **~1.5 M/s** | | 48 000 | ~469 KB | 209 | ~1.5 M/s | | 64 000 | ~625 KB | 157 | ~1.35 M/s | | 96 000 | ~938 KB | 105 | **~0.55 M/s** (near the 1 MB cliff) | The sweet spot is **32 000–48 000 samples**. Going larger gives no benefit and degrades sharply as the payload approaches 1 MB — the 96 000-sample case measured roughly **2–2.5× slower** than the sweet spot across runs. ### Interval quantisation (critical) !!! warning "Quantise float-derived intervals or throughput collapses" When timestamps come from floating-point arithmetic (`60e9 / rpm`), the nanosecond intervals often share no common factor, giving `delta_scale = 1`. That forces the splitter to halve every packet down to 2–3 samples — millions of tiny packets, throughput dropping to ~90–100 k samples/s. Round intervals to microsecond resolution **before** computing the delta scale: === "C#" ```csharp for (int i = 0; i < intervalsNs.Length; i++) { intervalsNs[i] = Math.Max((intervalsNs[i] / 1000) * 1000, 1000); } ``` === "Python" ```python intervals_ns = np.maximum((intervals_ns // 1000) * 1000, 1000) ``` === "MATLAB" ```matlab intervalsNs = max(floor(intervalsNs / 1000) * 1000, 1000); ``` For a 10 M-sample write this is the difference between ~108 s (≈3.9 M tiny packets) and ~9 s (≈1 250 packets) — a **12× improvement**. ### Flush periodically on large sessions On small sessions SQLRace buffers everything and writes on close. On very large sessions the buffer can exhaust memory and the close may fail **silently**. Call `Flush()` every few hundred packets, and `EndData()` before closing. === "C#" ```csharp int sinceFlush = 0; foreach (var pkt in packets) { session.AddSynchroChannelData(/* ... */); if (++sinceFlush >= 500) { session.Flush(); sinceFlush = 0; } } session.EndData(); ``` === "Python" ```python since_flush = 0 for pkt in packets: session.AddSynchroChannelData(...) since_flush += 1 if since_flush >= 500: session.Flush() since_flush = 0 session.EndData() ``` === "MATLAB" ```matlab sinceFlush = 0; for k = 1:numel(packets) session.AddSynchroChannelData(); % arguments elided for brevity sinceFlush = sinceFlush + 1; if sinceFlush >= 500 session.Flush(); sinceFlush = 0; end end session.EndData(); ``` ### Where the time goes At production scale the `.NET` write call dominates — it is ~95% of the total. A representative split for a 1 M-sample single-channel write: | Phase | Share | |-------|:---:| | `AddSynchroChannelData` (the DB write) | **~95%** | | One-time configuration commit | ~3% | | Interval diff + GCD + binary packing | ~1–2% | | `EndData` flush / close | <1% | The practical consequences: minimise the number of calls (right-size packets), don't micro-optimise the packing once it is vectorised, and create each parameter's configuration **once** — check whether the parameter already exists (`session.ContainsParameter`) before rebuilding its config. ### Parallelise across channels The write is latency-bound on the database round-trip, not CPU-bound. Both the .NET runtime and pythonnet **release the GIL/thread during the `AddSynchroChannelData` call**, so writing **different channels** from a small thread pool overlaps their round-trips: - **Threads across channels** — benchmarked at roughly **2× throughput** with 4 worker threads writing distinct channels into one session. The gain is sub-linear because the database is the shared ceiling. - **Within a single channel, use one writer.** A channel's packet sequence numbers must be emitted in order, so never split one channel across threads. - **Multiple processes** scale further still — each writes its own session, and you overlay them in ATLAS. !!! note "Create configuration serially, then fan out" Configuration commits mutate session state and are not thread-safe. Create all channel configurations on one thread first, then write the data for different channels concurrently. ### SQL Server vs SQLite For bulk ingestion prefer a (local) SQL Server instance over SQLite: SQLite is single-writer and flushes on every transaction, so it is measurably slower for sustained high-throughput synchro writes. !!! note "First-connection timeout" Some network configurations cause the *first* SQLRace connection to a remote SQL Server to take ~35 s. If you hit this, retry — the second attempt usually connects immediately. It is a known initialisation behaviour, not a fault. --- ## Common Mistakes !!! danger "Pitfalls to avoid" - **Using `ChannelDataSourceType.Periodic`/`RowData` for synchro data** — `AddSynchroChannelData` throws if the channel was not created as `Synchro`. The channel type must match the write method. - **Packing absolute timestamps instead of scaled intervals** — the payload carries *scaled intervals*, not timestamps. Packing raw timestamps makes ATLAS interpret enormous gaps and the trace looks completely wrong. - **One delta scale for the whole dataset** — the delta scale is **per packet**. A single global GCD can leave some packets with intervals that overflow `uint16`. Compute the GCD per packet. - **Non-monotonic timestamps** — SQLRace assumes timestamps are sorted ascending. Unsorted input produces garbled data with no error at write time; sort before writing. - **Skipping `Flush()`/`EndData()` on huge sessions** — the most common "data disappeared" report. Flush periodically and call `EndData()` before close. --- # Functions There different ways to work with functions in the SQLRace API. Existing FDL functions can be exposed through the API as FunctionParameters. Functions can also be defined from scratch in code. In addition to functions written in ATLAS's Function Definition Language (FDL) ATLAS 10 is able to call external interpreters to evaluate functions written in alternative languages. ## Exposing ATLAS Functions Through the API The API's License Program Name will need to be changed to the name of your ATLAS license. This will either be 'ATLAS 10 Premium' or 'ATLAS 10 Data Platform.' By default this is set to 'SQLRace.' The `LicenceProgramName` property needs to be set prior to the `Core.Initialize()` statement. Upon changing the license program name, the functions you have defined will be exposed through the session and can be used as any other parameter would. === "C#" ``` csharp Core.LicenceProgramName = "ATLAS 10 Premium"; Core.Initialize(); ``` === "Python" ``` python Core.LicenceProgramName = "ATLAS 10 Premium" Core.Initialize() ``` === "MATLAB" ``` matlab NET.setStaticProperty('MESL.SqlRace.Domain.Core.LicenceProgramName', 'ATLAS 10 Premium'); Core.Initialize(); ``` ## Implementing Functions From Scratch === "C#" ``` csharp // Some additional import statements using MESL.SqlRace.Enumerators; using MESL.SqlRace.Domain.Functions; using MESL.SqlRace.Domain.Functions.Fdl; using MESL.SqlRace.Functions.Interfaces.Enums; var functionManager = FunctionManagerFactory.Create(); // Create function const string CarSpeedDoubledInFdl = "return ($GPSSpeed:ECU * 2)"; var functionDefinition = functionManager.CreateFunctionDefinition(FdlFunctionConstants.UniqueId); // The implementation of a function is completely agnostic. We requested a function definition from // the FDL runtime, so the generic function implementation definition in the definition returned can // be safely cast to the more specific FDL implementation definition. var fdlFunctionImplementationDefinition = (IFdlFunctionImplementationDefinition)functionDefinition.ImplementationDefinition; fdlFunctionImplementationDefinition.FunctionCode = CarSpeedDoubledInFdl; // FDL doesn't define the output for a single parameter - it's simply the return value of the // function - so we have to tell the function executor what to do with the output it generates var outputParameter = FunctionOutputParameterDefinition .Create("GPSSpeedDoubled", "FunctionParameters", "Double car speed!") .Units("MPH") .FormatOverride("%.2f") .MinimumValue("0.0") .MaximumValue("700.0"); functionDefinition.OutputParameterDefinitions.Add(outputParameter); // Functions in ATLAS 9 have a number of different properties that define how, where, when and on // what a function executes. ATLAS 10 will add more of these properties. Each value here is the // default for its property, shown for illustration. functionDefinition.Name = $"DoubleCarSpeed_{Environment.TickCount}"; functionDefinition.FunctionMode = FunctionMode.Instantaneous; functionDefinition.CalculationModeInfoDefinition.Mode = CalculationMode.EachSamplePoint; functionDefinition.InterpolateBetweenSamples = false; functionDefinition.JoinGapsAroundNull = true; functionDefinition.CalculateOverWholeSession = false; functionDefinition.StoreInSession = false; functionDefinition.ShouldHide = false; functionDefinition.ShouldPersist = false; var buildResults = functionManager.Build(functionDefinition); if (buildResults.Errors.Any()) { Console.WriteLine("Function build failed."); } // Use this outputFunctionParameter to create PDA var outputFunctionParameter = functionDefinition.OutputParameterDefinitions.First(); ``` === "Python" ``` python # Some additional import statements from MESL.SqlRace.Enumerators import * from MESL.SqlRace.Domain.Functions import * from MESL.SqlRace.Domain.Functions.Fdl import * from MESL.SqlRace.Functions.Interfaces.Enums import * function_manager = FunctionManagerFactory.Create() # Create function car_speed_doubled_in_fdl = "return ($GPSSpeed:ECU * 2)" function_definition = function_manager.CreateFunctionDefinition(FdlFunctionConstants.UniqueId) # The implementation of a function is completely agnostic. We requested a function definition from # the FDL runtime, so the generic function implementation definition in the definition returned can # be safely cast to the more specific FDL implementation definition. fdl_function_implementation_definition = IFdlFunctionImplementationDefinition(function_definition.ImplementationDefinition) fdl_function_implementation_definition.FunctionCode = car_speed_doubled_in_fdl # FDL doesn't define the output for a single parameter - it's simply the return value of the # function - so we have to tell the function executor what to do with the output it generates output_parameter = FunctionOutputParameterDefinition.Create("GPSSpeedDoubled", "FunctionParameters", "Double car speed!") output_parameter.Units = "MPH" output_parameter.FormatOverride = "%.2f" output_parameter.MinimumValue = "0.0" output_parameter.MaximumValue = "700.0" function_definition.OutputParameterDefinitions.Add(output_parameter) # Functions in ATLAS 9 have a number of different properties that define how, where, when and on # what a function executes. ATLAS 10 will add more of these properties. Each value here is the # default for its property, shown for illustration. function_definition.Name = f"DoubleCarSpeed_{Environment.TickCount}" function_definition.FunctionMode = FunctionMode.Instantaneous function_definition.CalculationModeInfoDefinition.Mode = CalculationMode.EachSamplePoint function_definition.InterpolateBetweenSamples = False function_definition.JoinGapsAroundNull = True function_definition.CalculateOverWholeSession = False function_definition.StoreInSession = False function_definition.ShouldHide = False function_definition.ShouldPersist = False build_results = function_manager.Build(function_definition) if (build_results.Errors): print(f"Function build failed with error: {build_results.Errors}") # Use this output_function_parameter to create PDA output_function_parameter = function_definition.OutputParameterDefinitions[0] ``` === "MATLAB" ``` matlab % Some additional import statements import MESL.SqlRace.Enumerators.* import MESL.SqlRace.Domain.Functions.* import MESL.SqlRace.Domain.Functions.Fdl.* import MESL.SqlRace.Functions.Interfaces.Enums.* %% Create Function functionManager = FunctionManagerFactory.Create(); % Define function in FDL carSpeedDoubledInFdl = "return ($GPSSpeed:ECU * 2)"; functionDefinition = functionManager.CreateFunctionDefinition(FdlFunctionConstants.UniqueId); % The implementation of a function is completely agnostic. We requested a function definition from % the FDL runtime, so the generic function implementation definition in the definition returned can % be safely cast to the more specific FDL implementation definition. fdlFunctionImplementationDefinition = functionDefinition.ImplementationDefinition; fdlFunctionImplementationDefinition.FunctionCode = carSpeedDoubledInFdl; % FDL doesn't define the output for a single parameter - it's simply the return value of the % function - so we have to tell the function executor what to do with the output it generates outputParameter = FunctionOutputParameterDefinition.Create("GPSSpeedDoubled", "FunctionParameters", "Double car speed!"); outputParameter.Units = "MPH"; outputParameter.FormatOverride = "%.2f"; outputParameter.MinimumValue = "0.0"; outputParameter.MaximumValue = "700.0"; functionDefinition.OutputParameterDefinitions.Add(outputParameter); % Functions in ATLAS 9 have a number of different properties that define how, where, when and on % what a function executes. ATLAS 10 will add more of these properties. Each value here is the % default for its property, shown for illustration. functionDefinition.Name = sprintf("DoubleCarSpeed_%s", Environment.TickCount); functionDefinition.FunctionMode = FunctionMode.Instantaneous; functionDefinition.CalculationModeInfoDefinition.Mode = CalculationMode.EachSamplePoint; functionDefinition.InterpolateBetweenSamples = false; functionDefinition.JoinGapsAroundNull = true; functionDefinition.CalculateOverWholeSession = false; functionDefinition.StoreInSession = false; functionDefinition.ShouldHide = false; functionDefinition.ShouldPersist = false; buildResults = functionManager.Build(functionDefinition); if buildResults.Errors.Count > 0 fprintf("Function build failed with error: %s", buildResults.Errors.Item(0).ErrorText); end %% Retrieve data % Use this outputFunctionParameter to create PDA outputFunctionParameter = functionDefinition.OutputParameterDefinitions.Item(0); ``` ## Build a .NET Function .dll !!! tip "The following modules must be referenced:" - OCS.Core - SQLRace.Domain - SQLRace.Enumerators - SQLRace.Functions.Interfaces - SQLRace.Functions.Types Each function is implemented in a class that implements IDotNetFunction and is exported via the following attribute: ``` csharp [Export(typeof(IDotNetFunction))] ``` With the following namespaces: ``` csharp using MESL.SqlRace.Domain.Functions; using MESL.SqlRace.Domain.Functions.DotNet; using MESL.SqlRace.Enumerators; using MESL.SqlRace.Functions.Interfaces.Enums; using MAT.OCS.Core; ``` The function is defined and build in: ``` csharp public void Initialize(IFunctionManager functionManager) ``` The function properties match those you would specify when creating a function from the editor. Remember to assign “this” to the ImplementationDefinition. !!! note RelativePath can be set to override the location of the function within the parameter browser. and executed in: ``` csharp public void Execute(IExecutionContext executionContext) ``` where the timestamp of each input(s) is accessed by: ``` csharp executionContext.FunctionInput.Timestamps[]; ``` and values for each input via: ``` csharp executionContext.FunctionInput.Values[][]; // (input parameter index, input value) ``` The indexes of input parameters can be looked up by name: ``` csharp executionContext.FunctionInput.InputParameterIndexes; ``` The combined status of the inputs is accessed by: ``` csharp executionContext.FunctionInput.Statuses[]; ``` NB: if the status is `DataStatusType.Missing`, then assign `double.NaN` to the corresponding output values to indicate no value. Output values are assigned to each output parameter for each input value, via: ``` csharp executionContext.FunctionOutput.OutputParametersValues[][]; // (output parameter index, output value) ``` The indexes of output parameters can be looked up by name: ``` csharp executionContext.FunctionOutput.OutputParameterIndexes; ``` Copy just the output module to: `C:\Users\\Documents\McLaren Electronic Systems\SQL Race\Functions\` or sub folder. Restart ATLAS10 and on inspecting the parameter browser, you should see the function(s) populated. ### Code Example ``` csharp using MESL.SqlRace.Domain.Functions; using MESL.SqlRace.Domain.Functions.DotNet; using MESL.SqlRace.Enumerators; using MESL.SqlRace.Functions.Interfaces.Enums; using System; using System.ComponentModel.Composition; using System.Linq; namespace MAT.SqlRace.Functions.HelloDotNet { [Export(typeof(IDotNetFunction))] [Serializable] public class HelloFunction : IDotNetFunction { internal const string FunctionName = "HelloDotNet"; internal const string ParameterName = "vCarDoubledParam"; internal const string ApplicationName = "DotNetGroup"; internal const string ParameterIdentifier = ParameterName + ":" + ApplicationName; public string Name => FunctionName; /// /// This method will be called by SqlRace framework. /// /// public void Initialize(IFunctionManager functionManager) { // create the function definition var functionDefinition = functionManager.CreateFunctionDefinition(DotNetFunctionConstants.UniqueId, FunctionMode.Instantaneous, CalculationModeInfoDefinition.EachSamplePoint()); //var functionDefinition = functionManager.CreateFunctionDefinition(DotNetFunctionConstants.UniqueId); functionDefinition.CalculateOverWholeSession = false; functionDefinition.InterpolateBetweenSamples = false; functionDefinition.JoinGapsAroundNull = false; functionDefinition.Name = FunctionName; functionDefinition.StoreInSession = false; // set the implementation var implementationDefinition = (DotNetFunctionImplementationDefinition)functionDefinition.ImplementationDefinition; implementationDefinition.Function = this; // create the single output parameter functionDefinition.OutputParameterDefinitions.Add( new FunctionOutputParameterDefinition { ApplicationName = ApplicationName, ByteOrder = ByteOrderType.BigEndian, Description = "HelloDotNet.vCarDoubled", FormatOverride = @"%5.2f", Name = ParameterName, Units = "km/h", MaximumValue = "800", MinimumValue = "0", ShowInBrowser = true }); functionDefinition.InputParameterIdentifiers.Add("vCar:Chassis"); // build the function var buildResults = functionManager.Build(functionDefinition); // make sure we have no build errors if (buildResults.Errors.Count > 0) { Console.WriteLine($"Error building DotNet function '{FunctionName}': {buildResults.Errors.FirstOrDefault()?.ErrorText}."); } else { Console.WriteLine($"DotNet function '{FunctionName}' created successfully."); } } /// /// This method is called once per data request. /// /// public void Execute(IExecutionContext executionContext) { var timestamps = executionContext.FunctionInput.Timestamps; var timestampsCount = timestamps.Length; var index = executionContext.FunctionOutput.OutputParameterIndexes[ParameterIdentifier]; var parameterOutput = executionContext.FunctionOutput.OutputParametersValues[index]; var input = executionContext.FunctionInput.Values[index]; for (int i = 0; i < timestampsCount; i++) { parameterOutput[i] = input[i] * 2; //Console.WriteLine($"{data.Timestamp[i]},{new DateTime(data.Timestamp[i] / 100):HH:mm:ss.fff},{data.Data[i]} = {result[i]}"); } } } } ``` **Additional Notes** ``` csharp functionDefinition.CalculationModeInfoDefinition.Mode = CalculationMode.FixedFrequency; ``` `FixedFrequency` can be replaced by `EachSamplePoint` or `HighestRate` It is also possible to use corresponding numbers: ``` csharp functionDefinition.CalculationModeInfoDefinition.Mode = 0; ``` !!! tip "Numbers Corresponding To Their Calculation Mode" - 0 = EachSamplePoint - 1 = FixedFrequency - 2 = HighestRate If you use `FixedFrequency`, the following line is used to set the Frequency (Example for 1Hz): ``` csharp functionDefinition.CalculationModeInfoDefinition.Frequency = 1; ``` [GitHub Example](https://github.com/mat-docs/MAT.OCS.SQLRace.Examples/tree/master/projects/csharp/FunctionLibrary) ## C# Embedded .XFNs !!! warning "Licensed Feature" Please note that C# embedded XFN functions are a licensed feature and require the **.NET & MATLAB Functions** ATLAS license option. There are two methods to call .NET functions, one way is to write a Dynamic Link Library (.DLL) containing the functions' implementations the other is to embed the C# code directly into an .XFN file. A C# .NET function can be implemented directly in a specially constructed .XFN file. Each .XFN function file can implement multiple function outputs. The .XFN file must be in the ATLAS 10 Functions directory: `%HOMEPATH%\Documents\McLaren Electronic Systems\SQL Race\Functions` To try this example: 1. Download the SinAndCosOfSteeringWheelAngle.xfn file attached to this example and place it in the functions directory. 2. Start ATLAS 10 (or if it is already running, restart it). 3. Only one input parameter is referenced by the function aSteerWheel:Chassis so for the function to calculate, data containing this parameter must be loaded. 4. Two output parameters are calculated `steerWheelSin:Functions` and `steerWheelCos:Functions`. These should now be available to add to any display from the ATLAS 10 Parameter Browser or Quick Access shortcut. Firstly the InputParameters section defines which ATLAS parameter identifiers are to be sourced by the .NET function. There should be an Identifier line for each input parameter to the function. The OutputParameters section defines the output function parameter properties for each value the .NET function calculates. Each OutputParameter must have a Name, Description, Units, Format, DisplayMin and DisplayMax. Your .NET function can define more than one output parameter. The calculation defaults should be left as they are. The FunctionCode section defines the actual function implementation in C#. At the top of the implementation the three using statements bring the required types into the namespace without using their fully qualified names. The IFunction interface comes from MESL.SqlRace.Domain.Functions so this must be imported. In this example System is needed for the maths functions and System.Collections.Generic is not actually needed as no collections defined by this are being used (List<> for example). The function must define a class which implements the IFunction interface. It may be named anything you want but it is best practice to have it match the name of the XFN file which contains the class. Because each function is compiled separately there should be no name clashes though. The interface requires an Execute() method which is called by ATLAS to calculate the function values. It is inside this method where the named output parameters get assigned their calculated values. The class can contain other methods which are called by Execute() to perform the actual calculations. ### Code Example ```xml aSteering:TAG400NApp1 steerWheelSin ° %.2f -1.00 1.00 steerWheelCos ° %.2f -1.00 1.00 XFN_Instantaneous XFN_EachSample false true false false ``` ### Lap-reset integral example (C# embedded .XFN) The following `FunctionCode` example shows how to build an integral that resets whenever the lap counter changes. - `aLongAcc:Chassis` is the signal to integrate. - `nLap:Chassis` is an integer-like lap counter in the input data. - `aLongAccIntegralPerLap:Functions` is the output parameter identifier. ```csharp using System; using MESL.SqlRace.Domain.Functions; public sealed class LapResetIntegral : IFunction { public void Execute(IExecutionContext executionContext) { var functionInput = executionContext.FunctionInput; var functionOutput = executionContext.FunctionOutput; var timestamps = functionInput.Timestamps; var timestampsCount = timestamps.Length; if (timestampsCount == 0) { return; } var signalIndex = functionInput.InputParameterIndexes["aLongAcc:Chassis"]; var lapIndex = functionInput.InputParameterIndexes["nLap:Chassis"]; var integralIndex = functionOutput.OutputParameterIndexes["aLongAccIntegralPerLap:Functions"]; var signal = functionInput.Values[signalIndex]; var lapNumber = functionInput.Values[lapIndex]; var integral = functionOutput.OutputParametersValues[integralIndex]; var runningIntegral = 0.0; var previousLap = Convert.ToInt32(Math.Round(lapNumber[0])); for (var i = 0; i < timestampsCount; i++) { var currentLap = Convert.ToInt32(Math.Round(lapNumber[i])); if (i == 0 || currentLap != previousLap) { runningIntegral = 0.0; } else { // SQLRace timestamps are 100 ns units, so multiply by 1e-7 to get seconds. var dtSeconds = (timestamps[i] - timestamps[i - 1]) * 1e-7; runningIntegral += signal[i] * dtSeconds; } integral[i] = runningIntegral; previousLap = currentLap; } } } ``` ## MATLAB Embedded .XFNs !!! warning "Licensed Feature" Please note that MATLAB embedded XFN functions are a licensed feature and require the **.NET & MATLAB Functions** ATLAS license option. A MATLAB function comprises at least two parts; an .XFN file which gives ATLAS 10 an XML interface to the MATLAB function and a .M file which contains the actual MATLAB script implementation. .M files can call one another so it is possible to have multiple .M files making up a single definition. The .XFN file must be in the ATLAS 10 Functions directory: `%HOMEPATH%\Documents\McLaren Electronic Systems\SQL Race\Functions` The .M files are referenced by absolute path from the .XFN file so can be located anywhere. All .M files which make up a single function definition however must be in the same folder or in folders on the MATLAB path. In this example they will be placed in the ATLAS 10 Functions directory like the .XFN file above. 1. The three files in this example are CalculateSlipRatios.xfn, calculateslipratios.m and slipratio.m, download these and place them all in the functions directory. 2. Edit the CalculateSlipRatios.xfn file in any normal text editor, find the ... line and edit it so the correct, absolute path to the calculateslipratios.m file is in the inner pair of square brackets. 3. Start ATLAS 10 (or if it is already running, restart it). 4. Four ATLAS input parameters are referenced; `vCar:Chassis`, `vWheelFL:Chassis`, `vWheelFR:Chassis`, `vWheelRL:Chassis` and `vWheelRR:Chassis` so for the function to calculate, data containing these parameters must be loaded. 5. Four output parameters are calculated: `rSlipRatioFL:Functions`, `rSlipRatioFR:Functions`, `rSlipRatioRL:Functions` and `rSlipRatioRR:Functions`. These should now be available to add to any display from the ATLAS 10 Parameter Browser or Quick Access shortcut. ### The .XFN File Firstly the InputParameters section defines which ATLAS parameter identifiers are to be sourced by the MATLAB function. There should be an Identifier line for each input parameter to the function. The OutputParameters section defines the output function parameter properties for each value the MATLAB function calculates. Each OutputParameter must have a Name, Description, Units, Format, DisplayMin and DisplayMax. Your MATLAB function can define more than one output parameter. The calculation defaults should be left as they are, these will be described in more detail in later Help documents. The FilePath line is the link between ATLAS and the MATLAB .M file which contains the implementation in MATLAB script. This is an absolute path. The InputMappings section defines how the ATLAS input parameter identifiers appear to MATLAB in the .M script. In this example the input parameter `vCar:Chassis` is translated to a MATLAB variable named simply vCar which is referenced in the code. There must be a Mapping entry for each of the Identifiers lines in the InputParameters section. The OutputMappings section defines how the MATLAB output parameters appear to ATLAS 10 in the Parameter Browser. In this example the MATLAB variable output by the script rSlipRatioFL will appear with the `rSlipRatioFL:Functions` identifier in ATLAS. There must be a Mapping entry for each of the OutputParameter entries in the OutputParameters section. ### The .M File The absolute basic .M file just assigns values to the OutputParameter Mapping MATLAB variable names. Our slightly more complicated example uses a MATLAB function call, referencing the input variables as arguments and assigns the function return to the output variable names. Any valid MATLAB code can appear in the .M files. So in our example the .XFN function definition references calculateslipratios.m ![MATLAB1](../assets/MATLABImplementation1.png) This calls the MATLAB function `slipratio()` using the input parameter names as the function arguments and assigns the function return to the four output variables. The actual calculation is done inside slipratio.m which defines a standard function in MATLAB ![MATLAB2](../assets/MATLABImplementation2.png) ### Code Example **CalculateSlipRatio.xfn** ```xml vCar:Chassis vWheelFL:Chassis vWheelFR:Chassis vWheelRL:Chassis vWheelRR:Chassis rSlipRatioFL % %.2f -5.00 90.00 rSlipRatioFR % %.2f -5.00 90.00 rSlipRatioRL % %.2f -5.00 90.00 rSlipRatioRR % %.2f -5.00 90.00 XFN_LeadingEdge XFN_EachSample false true false false vCar:Chassis vCar vWheelFL:Chassis vWheelFL vWheelFR:Chassis vWheelFR vWheelRL:Chassis vWheelRL vWheelRR:Chassis vWheelRR rSlipRatioFL rSlipRatioFL:Functions rSlipRatioFR rSlipRatioFR:Functions rSlipRatioRL rSlipRatioRL:Functions rSlipRatioRR rSlipRatioRR:Functions ``` **calculateslipratios.m** ```matlab %vCar = [10, 20, 30, 40, 50]; %vWheelFL = [9, 18, 27, 36, 45]; rSlipRatioFL = slipratio(vCar, vWheelFL); rSlipRatioFR = slipratio(vCar, vWheelFR); rSlipRatioRL = slipratio(vCar, vWheelRL); rSlipRatioRR = slipratio(vCar, vWheelRR); ``` **slipratio.m** ```matlab function slipRatio = slipratio(vCar, vWheel) diff = vCar-vWheel; ratio = diff./vCar; slipRatio = ratio.*100; end ``` --- # Parameter Data Access **ParameterDataAccess objects (PDAs)** are used to defines how parameter values are sampled or interpolated via SQL Race. PDA can be created by the `Session.CreateParameterDataAccess(parameterIdentifier)` method and is associated with both the Session and a Parameter. More than one PDA can exist within a session as each pda can be used to retrieve values from a different parameter. Methods on the PDA allow you to: * Move to specific point in time * Set the sample time * Read sub-sampled data * Read interpolated data * Read the measured sample values !!! info "Best practice" * Opening a PDA uses up memory until the session is closed and disposed. It is best to open a PDA for a parameter once and cache it. Then you can reuse it throughout the session. An example of when this is particularly useful is when iterating through laps. It is better to open each PDA once than to open and dispose of them inside the lap iteration. * Calls to a PDA is not thread safe. If you are wanting to access data using the same Parameter in different threads, it is better to consider opening a small pool of PDAs for the threads to use. You have to be extra careful when using method that require the PDA's current position to ensure that it is only being used by one thread. ## Samples and Data Within the context of PDA methods, *samples* refers to the values as logged by the datalogger, whereas *data* refers to sub-sampled, or interpolated values at a predefined fixed frequency. A parameter can contain several channels logged at different rates, and the two families of read methods reflect this: | | `GetNextSamples` / `GetSamplesUntil` | `GetNextData` / `GetData` | |---|---|---| | **Returns** | Raw logged **samples** | Resampled **data** on a fixed grid | | **Timestamps** | Each channel's *native* timestamps | A uniform `SampleTime` grid shared by every channel | | **`SampleTime`** | Ignored | Defines the grid interval (ns) | | **Sample mode** | — | `First` / `Minimum` / `Maximum` / `Mean` | | **Interpolation** | — | Optional linear interpolation when up-sampling | | **Direction** | `StepDirection.Forward` / `Reverse` | Forward only | | **Best for** | Maximum fidelity — exactly what was recorded | Aligning channels, plotting, consistent time steps | !!! note `SampleTime` only affects `GetNextData`/`GetData`. It has **no effect** on `GetNextSamples`/`GetSamplesUntil`, which always return the raw logged samples. Toggle the two read styles below. **GetNextSamples** keeps each channel on its own native timestamps (the highest-rate channel sets the cadence, and the slower channel has no value in between); **GetNextData** snaps everything onto one uniform `SampleTime` grid so the channels line up — drag **SampleTime**, change the mode, and toggle interpolation to see the effect.
## Choosing the Output Frequency With `GetNextData` you choose the output rate by setting the PDA's `SampleTime` (in **nanoseconds**). The returned frequency is `1e9 / SampleTime`: | `SampleTime` (ns) | Output frequency | |-------------------|------------------| | `2,000,000` | 500 Hz | | `10,000,000` | 100 Hz | | `20,000,000` | 50 Hz | The overlay below compares the true signal against `GetNextSamples` (every logged sample) and `GetNextData` at 500 Hz and 50 Hz. Drag the slider to request any output frequency and watch the trade-off: a lower frequency averages each window and loses the peaks, while a higher frequency can't recover detail beyond the logged rate.
### `GetNextSamples` — every logged sample Raw samples at the channel's native timestamps. Not affected by `SampleTime`. === "C#" ``` csharp pda.GoTo(session.StartTime); long n = pda.GetSamplesCount(session.StartTime, session.EndTime); ParameterValues samples = pda.GetNextSamples((int)n, StepDirection.Forward); double[] values = samples.Data; ``` === "Python" ``` python parameter_pda = session.CreateParameterDataAccess("vCar") parameter_pda.GoTo(session.StartTime) n_samples = parameter_pda.GetSamplesCount(session.StartTime, session.EndTime) samples = parameter_pda.GetNextSamples(n_samples, StepDirection.Forward) values = list(samples.Data) ``` === "MATLAB" ``` matlab pda.GoTo(session.StartTime); n = pda.GetSamplesCount(session.StartTime, session.EndTime); samples = pda.GetNextSamples(int32(n), StepDirection.Forward); values = double(samples.Data); ``` ### `GetNextData` — resampled at a fixed frequency Set `SampleTime` to choose the output rate, then read `n` aligned samples (default mode is `Mean`). === "C#" ``` csharp pda.SampleTime = 2_000_000; // 2 ms -> 500 Hz long n = pda.GetSamplesCount(lap.StartTime, lap.EndTime); ParameterValues data = pda.GetNextData((int)n); double[] values500Hz = data.Data; ``` === "Python" ``` python parameter_pda = session.CreateParameterDataAccess("vCar") parameter_pda.SampleTime = 10000000 # 10 ms -> 100 Hz n_samples = parameter_pda.GetSamplesCount(lap.StartTime, lap.EndTime) getnextdata = parameter_pda.GetNextData(n_samples) getnextdata_values_100Hz = list(getnextdata.Data) ``` === "MATLAB" ``` matlab parameter_pda = session.CreateParameterDataAccess("vCar"); parameter_pda.SampleTime = 10000000; % 10 ms -> 100 Hz n = parameter_pda.GetSamplesCount(lap.StartTime, lap.EndTime); getnextdata = parameter_pda.GetNextData(int32(n)); values_100Hz = double(getnextdata.Data); ``` ## Sample Modes Sample modes define the behaviour when sub-sampling. | Sample Mode | Behaviour | |-------------|-----------| | First | Return the first sample value in the range. | | Maximum | Return the maximum sample value in the range. | | Mean | Return the average sample value across the range. | | Minimum | Return the minimum sample value in the range. | The chart below applies all four modes to the same logged signal. Drag **SampleTime** to widen the sub-sample interval and watch `Maximum` track the peaks, `Minimum` the troughs, and `Mean` smooth through the middle — the coarser the interval, the more the modes diverge.
## Reading a Specific Channel By default a PDA reads across **all** channels of a parameter. To read a single channel (or a subset), pass a list of channel IDs to `CreateParameterDataAccess`. Inspect the available channels first via `Parameter.Channels`, then build the ID list: === "C#" ``` csharp var vCar = session.GetParameter("vCar"); // Inspect the channels that make up the parameter foreach (var c in vCar.Channels) { Console.WriteLine($"Name: {c.Name}, Id: {c.Id}, DataSource: {c.DataSource}, " + $"DataType: {c.DataType}, Interval: {c.Interval} ns, " + $"SampleSize: {c.SampleSizeInBytes} bytes"); } // Build a list containing just the channel(s) you want var channelIds = new List { vCar.Channels[0].Id }; // Create a PDA scoped to those channels using (var pda = session.CreateParameterDataAccess(vCar.Identifier, channelIds)) { pda.GoTo(session.StartTime); long nSamples = pda.GetSamplesCount(session.StartTime, session.EndTime); var values = pda.GetNextSamples((int)nSamples, StepDirection.Forward); } ``` === "Python" ``` python from System import UInt32 from System.Collections.Generic import List vCar = session.GetParameter("vCar") # Inspect the channels that make up the parameter for c in vCar.Channels: print("Channel Name:", c.Name) print(" Id:", c.Id) print(" DataSource:", c.DataSource.ToString()) print(" DataType:", c.DataType.ToString()) print(" Interval (ns):", c.Interval) if c.Interval != 0: # nanoseconds -> kHz print(" Frequency (kHz):", 1 / (float(c.Interval) * 1e-6)) print(" Sample Size (bytes):", c.SampleSizeInBytes) # Build a List[UInt32] with just the channel id(s) you want channel_ids_list = List[UInt32]() channel_ids_list.Add(vCar.Channels[0].Id) # Create a PDA scoped to those channels parameter_pda = session.CreateParameterDataAccess(vCar.Identifier, channel_ids_list) parameter_pda.GoTo(session.StartTime) n_samples = parameter_pda.GetSamplesCount(session.StartTime, session.EndTime) ``` === "MATLAB" ``` matlab vCar = session.GetParameter("vCar"); % Inspect the channels that make up the parameter channels = vCar.Channels; for i = 1:channels.Count c = channels.Item(i - 1); fprintf('Name: %s, Id: %d, DataSource: %s, DataType: %s, Interval: %d ns, SampleSize: %d bytes\n', ... char(c.Name), c.Id, char(c.DataSource.ToString()), ... char(c.DataType.ToString()), c.Interval, c.SampleSizeInBytes); end % Build a List with just the channel id(s) you want channelIds = NET.createGeneric('System.Collections.Generic.List', {'System.UInt32'}); channelIds.Add(uint32(channels.Item(0).Id)); % Create a PDA scoped to those channels pda = session.CreateParameterDataAccess(vCar.Identifier, channelIds); pda.GoTo(session.StartTime); nSamples = pda.GetSamplesCount(session.StartTime, session.EndTime); values = double(pda.GetNextSamples(int32(nSamples), StepDirection.Forward).Data); ``` !!! tip "Why scope to a channel?" A parameter such as `vCar` may aggregate several channels — logged at different rates or from different sources. Scoping the PDA to one channel avoids reading data you don't need and lets you walk a single source at its native rate. ## Lap Statistics `GetLapStatistics` computes aggregate statistics for a parameter over a single [`Lap`](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.Lap.html)'s time range. Pass one or more `StatisticOption` flags to choose what to compute; the results come back in an `IParameterDataStatistics` object. It takes a *"show your workings"* approach — requesting `StandardDeviation`, for example, also returns the sum of values and the mean. Samples with `NaN` values are skipped. Pick a parameter and toggle the statistics below. Each lap's Min, Max, Mean and ±σ band are drawn over the trace, and the table mirrors the `IParameterDataStatistics` fields you would read back per lap.
=== "C#" ``` csharp using var pda = session.CreateParameterDataAccess("vCar:Chassis"); foreach (var lap in session.LapCollection) { IParameterDataStatistics stats = pda.GetLapStatistics( lap, false, // weightStatistics StatisticOption.Min | StatisticOption.Max | StatisticOption.Mean | StatisticOption.StandardDeviation); Console.WriteLine( $"{lap.Name}: min={stats.MinimumValue:F1} max={stats.MaximumValue:F1} " + $"mean={stats.MeanValue:F1} sd={stats.StandardDeviation:F2} " + $"n={stats.NumberOfSamples}"); } ``` === "Python" ``` python pda = session.CreateParameterDataAccess("vCar:Chassis") for lap in session.LapCollection: stats = pda.GetLapStatistics( lap, False, # weightStatistics StatisticOption.Min | StatisticOption.Max | StatisticOption.Mean | StatisticOption.StandardDeviation) print(f"{lap.Name}: min={stats.MinimumValue:.1f} max={stats.MaximumValue:.1f} " f"mean={stats.MeanValue:.1f} sd={stats.StandardDeviation:.2f} " f"n={stats.NumberOfSamples}") ``` === "MATLAB" ``` matlab pda = session.CreateParameterDataAccess("vCar:Chassis"); opts = bitor(int32(StatisticOption.Min), int32(StatisticOption.Max)); opts = bitor(opts, int32(StatisticOption.Mean)); opts = bitor(opts, int32(StatisticOption.StandardDeviation)); laps = session.LapCollection; for i = 1:laps.Count lap = laps.Item(i - 1); stats = pda.GetLapStatistics(lap, false, opts); fprintf('%s: min=%.1f max=%.1f mean=%.1f sd=%.2f n=%d\n', ... char(lap.Name), stats.MinimumValue, stats.MaximumValue, ... stats.MeanValue, stats.StandardDeviation, stats.NumberOfSamples); end ``` For an arbitrary time range rather than a lap, use `GetDataStatistics(startTimeNs, durationNs, weightStatistics, statisticOption)` with the same `StatisticOption` flags. ### Reading more statistics `StatisticOption` is a flag enum, so a single call can compute many statistics at once. This example requests the full set and reads back the populated fields: === "C#" ``` csharp var stats = pda.GetLapStatistics(lap, false, StatisticOption.Min | StatisticOption.Max | StatisticOption.Mean | StatisticOption.StandardDeviation | StatisticOption.AbsoluteMean | StatisticOption.GeometricMean | StatisticOption.Delta); Console.WriteLine($"start={stats.StartValue:F1} end={stats.EndValue:F1} delta={stats.DeltaValue:F1}"); Console.WriteLine($"min={stats.MinimumValue:F1} @ {stats.MinimumTime} ns ({stats.MinimumStatus})"); Console.WriteLine($"max={stats.MaximumValue:F1} @ {stats.MaximumTime} ns ({stats.MaximumStatus})"); Console.WriteLine($"mean={stats.MeanValue:F1} absMean={stats.AbsoluteMeanValue:F1} geoMean={stats.GeometricMeanValue:F1}"); Console.WriteLine($"sd={stats.StandardDeviation:F2} n={stats.NumberOfSamples} NaNs={stats.NumberOfNansSkipped}"); ``` === "Python" ``` python stats = pda.GetLapStatistics(lap, False, StatisticOption.Min | StatisticOption.Max | StatisticOption.Mean | StatisticOption.StandardDeviation | StatisticOption.AbsoluteMean | StatisticOption.GeometricMean | StatisticOption.Delta) print(f"start={stats.StartValue:.1f} end={stats.EndValue:.1f} delta={stats.DeltaValue:.1f}") print(f"min={stats.MinimumValue:.1f} @ {stats.MinimumTime} ns ({stats.MinimumStatus})") print(f"max={stats.MaximumValue:.1f} @ {stats.MaximumTime} ns ({stats.MaximumStatus})") print(f"mean={stats.MeanValue:.1f} absMean={stats.AbsoluteMeanValue:.1f} geoMean={stats.GeometricMeanValue:.1f}") print(f"sd={stats.StandardDeviation:.2f} n={stats.NumberOfSamples} NaNs={stats.NumberOfNansSkipped}") ``` === "MATLAB" ``` matlab opts = int32(StatisticOption.Min); opts = bitor(opts, int32(StatisticOption.Max)); opts = bitor(opts, int32(StatisticOption.Mean)); opts = bitor(opts, int32(StatisticOption.StandardDeviation)); opts = bitor(opts, int32(StatisticOption.AbsoluteMean)); opts = bitor(opts, int32(StatisticOption.GeometricMean)); opts = bitor(opts, int32(StatisticOption.Delta)); stats = pda.GetLapStatistics(lap, false, opts); fprintf('start=%.1f end=%.1f delta=%.1f\n', ... stats.StartValue, stats.EndValue, stats.DeltaValue); fprintf('min=%.1f @ %d ns (%s)\n', stats.MinimumValue, stats.MinimumTime, ... char(stats.MinimumStatus.ToString())); fprintf('max=%.1f @ %d ns (%s)\n', stats.MaximumValue, stats.MaximumTime, ... char(stats.MaximumStatus.ToString())); fprintf('mean=%.1f absMean=%.1f geoMean=%.1f\n', ... stats.MeanValue, stats.AbsoluteMeanValue, stats.GeometricMeanValue); fprintf('sd=%.2f n=%d NaNs=%d\n', ... stats.StandardDeviation, stats.NumberOfSamples, stats.NumberOfNansSkipped); ``` **Trend (linear regression).** `SlopeIntercept` and `SlopeRSquared` fit a line to the lap's samples, where *X* is the sample number (1, 2, 3, …) and *Y* is the sample value — useful for spotting drift, e.g. brake or tyre temperature creeping up over a lap. Toggle **Trend (slope)** in the chart above to see the fitted line per lap. === "C#" ``` csharp var stats = pda.GetLapStatistics(lap, false, StatisticOption.SlopeIntercept | StatisticOption.SlopeRSquared); Console.WriteLine($"slope={stats.RegressionSlopeValue:F4} per sample, " + $"intercept={stats.RegressionInterceptValue:F2}, R2={stats.RegressionRSquaredValue:F3}"); ``` === "Python" ``` python stats = pda.GetLapStatistics(lap, False, StatisticOption.SlopeIntercept | StatisticOption.SlopeRSquared) print(f"slope={stats.RegressionSlopeValue:.4f} per sample, " f"intercept={stats.RegressionInterceptValue:.2f}, R2={stats.RegressionRSquaredValue:.3f}") ``` === "MATLAB" ``` matlab opts = bitor(int32(StatisticOption.SlopeIntercept), int32(StatisticOption.SlopeRSquared)); stats = pda.GetLapStatistics(lap, false, opts); fprintf('slope=%.4f per sample, intercept=%.2f, R2=%.3f\n', ... stats.RegressionSlopeValue, stats.RegressionInterceptValue, stats.RegressionRSquaredValue); ``` **Weighted statistics.** Set `weightStatistics = true` to weight the mean, absolute mean and standard deviation by each sample's interval — important for variable-rate (synchro) data where samples are not evenly spaced in time: === "C#" ``` csharp var weighted = pda.GetLapStatistics(lap, true, StatisticOption.Mean | StatisticOption.StandardDeviation); ``` === "Python" ``` python weighted = pda.GetLapStatistics(lap, True, StatisticOption.Mean | StatisticOption.StandardDeviation) ``` === "MATLAB" ``` matlab opts = bitor(int32(StatisticOption.Mean), int32(StatisticOption.StandardDeviation)); weighted = pda.GetLapStatistics(lap, true, opts); ``` ### Statistic options `StatisticOption` is a flag enum — combine options with `|` to compute several in a single call. Each option populates specific result fields: | `StatisticOption` | Populates | |-------------------|-----------| | `Min` | `MinimumValue`, `MinimumTime`, `MinimumStatus` | | `Max` | `MaximumValue`, `MaximumTime`, `MaximumStatus` | | `Mean` | `MeanValue`, `SumOfSampleValues` | | `StandardDeviation` | `StandardDeviation`, `MeanValue`, `SumOfSampleValues`, `SumOfSquaredSampleValues` | | `AbsoluteMean` | `AbsoluteMeanValue`, `SumOfAbsoluteSampleValues` | | `GeometricMean` | `GeometricMeanValue` (positive values only) | | `Delta` | `DeltaValue` (last − first sample) | | `SlopeIntercept` / `SlopeRSquared` | `RegressionSlopeValue`, `RegressionInterceptValue`, `RegressionRSquaredValue` | `StartValue`, `EndValue`, `NumberOfSamples`, `NumberOfNansSkipped` and `TimeRange` are always populated; the default (`None`) returns only start/end values. !!! note * Statistics are **cached** per lap for fast repeat access. * The lap must have an end time and a start ≥ 0, otherwise `ArgumentNullException` / `ArgumentException` is thrown. * If the lap's range falls outside the available data, `NumberOfSamples` is `0`. * Set `weightStatistics = true` to weight the mean, absolute mean and standard deviation by each sample's interval. * `StatisticOption` lives in `MESL.SqlRace.Domain.Infrastructure.DataPipeline`. ## Data Status `ParameterValues.DataStatus` provides additional information about the values returned in `ParameterValues.Data`. When retrieving values via a PDA, the arrays within `ParameterValues` will always be the length of the requested sample size, regardless of the number of values requested. As an example, when there are only 100 samples left between the cursor and the end of the session, and the sample size requested is larger than 100. The first 100 data status will be `Sample`, and the remaining `DataStatus` will be `AfterEnd`. The property `ParameterValues.SampleCount` can be used to check the number of valid samples within the returned array. * Samples that represents values as measured by the data logger will have data status `Sample`. * Data that are interpolated will have data status `Interpolated`. | **DataStatusType** | **Description** | |--------------------|-----------------| | `Missing` | No sample is available. *(Should be excluded from processing)* | | `Sample` | A valid sample exists within the current sample period. *(Recommended for processing)* | | `Default` | The sensor failed and the ECU returned a default value. | | `BeforeStart` | Data was requested before the first available sample. | | `AfterEnd` | Data was requested after the last available sample. | | `Incomplete` | The sample interval includes missing samples. | | `Interpolated` | The sample has been interpolated. | | `Pending` | The sample is being processed by a function processor. | | `Gap` | There is a gap in the data (e.g., in repeating lap compare mode). | !!! warning The length of `ParameterValues.Data` is not an indication of the amount of valid sample returned. Use `ParameterValues.SampleCount` to get the number of valid samples. ## Examples ### Getting Samples from a Historical Session `GetNextSamples` is **stateful**: `GoTo` positions a cursor, then each call returns the next `count` samples and advances the cursor — so repeated calls walk the session in order until the data runs out (the trailing values come back with `DataStatus = AfterEnd`). === "C#" ``` csharp // Local database connection string and session key for data already recorded string connectionString = @"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; SessionKey sessionKey = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC"); // Initialise SQLRace API Core.Initialize(); var sessionManager = SessionManager.CreateSessionManager(); // Load existing session using (var clientSession = sessionManager.Load(sessionKey, connectionString)) { // Obtain the session var session = clientSession.Session; // Use PDA to get logged data across all channels in parameter vCar using (ParameterDataAccessBase pda = session.CreateParameterDataAccess("vCar:Chassis")) { // Go to the start of the session pda.GoTo(session.StartTime); // Get 10 samples var parameterValues = pda.GetNextSamples(10, StepDirection.Forward); // Print data and status to console window for (int i = 0; i < 10; i++) { Console.WriteLine($"Data: {parameterValues.Data[i]}, Timestamp: {parameterValues.Timestamp[i]}, Status: {parameterValues.DataStatus[i]}"); } } } } ``` === "Python" ``` python # Local database connection string and session key for data already recorded connection_string = r"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;" session_key = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC") # Initialise SQLRace API Core.Initialize() session_manager = SessionManager.CreateSessionManager() # Load existing session client_session = session_manager.Load(session_key, connection_string) # Obtain the session session = client_session.Session # Create a PDA to get logged data across all channels in parameter vCar pda = session.CreateParameterDataAccess("vCar:Chassis") # Go to the start of the session pda.GoTo(session.StartTime) # Get 10 samples parameter_values = pda.GetNextSamples(10, StepDirection.Forward) # Print data and status to console window for i in range(10): print(f"Data: {parameter_values.Data[i]}, TimeStamp: {parameter_values.Timestamp[i]} Status: {parameter_values.DataStatus[i]}") # Dispose objects once we are finish with it pda.Dispose() client_session.Close() ``` === "MATLAB" ``` matlab % Local database connection string and session key for data already recorded connectionString = "DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; sessionKey = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC"); % Initialise SQLRace API Core.Initialize() sessionManager = SessionManager.CreateSessionManager(); % Load existing session clientSession = sessionManager.Load(sessionKey, connectionString); % Obtain the session session = clientSession.Session; % Create a PDA to get logged data across all channels in parameter vCar pda = session.CreateParameterDataAccess("vCar:Chassis"); % Go to the start of the session pda.GoTo(session.StartTime); % Get 10 samples parameterValues = pda.GetNextSamples(10, StepDirection.Forward); % Print data and status to console window for i=1:10 fprintf("Data: %f, Timestamp: %i, Status: %s\n",parameterValues.Data(i),parameterValues.Timestamp(i),parameterValues.DataStatus(i)) end % Dispose objects once we are finish with it pda.Dispose() clientSession.Close() ``` ### Getting Samples from a Live Session PDA contains the property `CurrentTime`, which keeps track of the location of the cursor. This cursor is specific to each instance of PDA and can be used in combination with `GetNextSamples` to get new samples on the leading edge during a live session. After the samples have been retrieved, the `CurrentTime` gets updated to 1ns after the timestamp of the last sample retrieved. === "C#" ``` csharp string connectionString = @"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; // Initialise SQLRace API Core.Initialize(); var sessionManager = SessionManager.CreateSessionManager(); // Find the latest live session var sessionSummaries = sessionManager.FindBySessionState(SessionState.Live, connectionString); var liveSessionSummary = sessionSummaries.OrderBy(x => x.TimeOfRecording).Last(); // Load the live session using (var clientSession = sessionManager.Load(liveSessionSummary.Key, liveSessionSummary.GetConnectionString())) { // Obtain the session var session = clientSession.Session; // Use PDA to get logged data across all channels in parameter vCar using (ParameterDataAccessBase pda = session.CreateParameterDataAccess("vCar:Chassis")) { // Go to the start of the session pda.GoTo(session.StartTime); while (session.State == SessionState.Live) { // Get 10 samples var parameterValues = pda.GetNextSamples(10, StepDirection.Forward); // Print data and status to console window for (int i = 0; i < parameterValues.SampleCount; i++) { Console.WriteLine($"Data: {parameterValues.Data[i]}, Timestamp: {parameterValues.Timestamp[i]}, Status: {parameterValues.DataStatus[i]}"); } Console.WriteLine($"New Cursor Position: {pda.CurrentTime}"); // Wait for a second Thread.Sleep(1000); } } } } ``` === "Python" ```python connection_string = r"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;" # Initialise SQLRace API Core.Initialize() session_manager = SessionManager.CreateSessionManager() # Find the latest live session session_summaries = session_manager.FindBySessionState(SessionState.Live, connection_string) latest_live_session = max(session_summaries, key=lambda x: x.TimeOfRecording) client_session = session_manager.Load(latest_live_session.Key, latest_live_session.GetConnectionString()) session = client_session.Session # Create a PDA to get logged data across all channels in parameter vCar pda = session.CreateParameterDataAccess("vCar:Chassis") # Go to the start of the session pda.GoTo(session.StartTime) while session.State == SessionState.Live: # Get 10 samples parameter_values = pda.GetNextSamples(10, StepDirection.Forward) # Print data and status to console window for i in range(parameter_values.SampleCount): print( f"Data: {parameter_values.Data[i]}, TimeStamp: {parameter_values.Timestamp[i]} Status: {parameter_values.DataStatus[i]}") print(f"New Cursor Position: {pda.CurrentTime}") # Wait for a second time.sleep(1) # Dispose objects once we are finish with it pda.Dispose() client_session.Close() ``` === "MATLAB" ```matlab connectionString = "DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; % Initialise SQLRace API Core.Initialize() sessionManager = SessionManager.CreateSessionManager(); % Find the latest live session sessionSummaries = sessionManager.FindBySessionState(SessionState.Live, connectionString); latestLiveSession = sessionSummaries.Item(0); % Array formation and indexing are not allowed on .NET objects in MATLAB for i = 0:sessionSummaries.Count-1 disp(sessionSummaries.Item(i).Key.ToString()) if (sessionSummaries.Item(i).TimeOfRecording > latestLiveSession.TimeOfRecording) latestLiveSession = sessionSummaries.Item(i); end end clientSession = sessionManager.Load(latestLiveSession.Key, latestLiveSession.GetConnectionString()); % Obtain the session session = clientSession.Session; % Create a PDA to get subampled data across all channels in parameter vCar pda = session.CreateParameterDataAccess("vCar:Chassis"); % Go to the start of the session pda.GoTo(session.StartTime); while (session.State == SessionState.Live) % Get 10 samples parameterValues = pda.GetNextSamples(10, StepDirection.Forward); % Print data and status to console window for i=1:10 fprintf("Data: %f, Timestamp: %i, Status: %s\n",parameterValues.Data(i),parameterValues.Timestamp(i),parameterValues.DataStatus(i)) end fprintf("New Cursor Position: %i\n", pda.CurrentTime.Value) % Wait for a second pause(1) end % Dispose objects once we are finish with it pda.Dispose() clientSession.Close() ``` ### Caching PDA to Improve Efficiency It is advisable to reuse PDAs regardless of whether multiple parameters are in use. A dictionary with the parameter identifier can be used to achieve this. === "C#" ``` csharp // Local database connection string and session key for data already recorded string connectionString = @"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; SessionKey sessionKey = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC"); // List of parameter identifiers to obtain samples from List parameterIdentifiers = new List { "vCar:Chassis", "nEngine:FIA" }; // Initialise SQLRace API Core.Initialize(); var sessionManager = SessionManager.CreateSessionManager(); // Load existing session using (var clientSession = sessionManager.Load(sessionKey, connectionString)) { // Create the pda cache Dictionary pdaCache = new Dictionary(); // Obtain the session var session = clientSession.Session; foreach (var lap in session.LapCollection) { foreach (var parameterIdentifier in parameterIdentifiers) { // Open and cache PDA ParameterDataAccessBase pda; if (pdaCache.ContainsKey(parameterIdentifier)) { pda = pdaCache[parameterIdentifier]; } else { pda = session.CreateParameterDataAccess(parameterIdentifier); pdaCache[parameterIdentifier] = pda; } // Go to the start of the lap pda.GoTo(lap.StartTime); // Get 10 samples var parameterValues = pda.GetNextSamples(10, StepDirection.Forward); Console.WriteLine($"Parameter: {parameterIdentifier}"); // Print data and status to console window for (int i = 0; i < 10; i++) { Console.WriteLine($"Data: {parameterValues.Data[i]}, Timestamp: {parameterValues.Timestamp[i]}, Status: {parameterValues.DataStatus[i]}"); } } } // Dispose the PDAs afterwards foreach (var parameterDataAccessBase in pdaCache) { parameterDataAccessBase.Value.Dispose(); } } ``` === "Python" ```python # Local database connection string and session key for data already recorded connection_string = r"DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;" session_key = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC") # List of parameter identifiers to obtain samples from parameter_identifiers = ["vCar:Chassis", "nEngine:FIA"] # Initialise SQLRace API Core.Initialize() session_manager = SessionManager.CreateSessionManager() # Load existing session client_session = session_manager.Load(session_key, connection_string) # Create the pda cache pda_cache = {} # Obtain the session session = client_session.Session for lap in session.LapCollection: for parameter_identifier in parameter_identifiers: # Open and cache PDA if parameter_identifier in pda_cache: pda = pda_cache[parameter_identifier] else: pda = session.CreateParameterDataAccess(parameter_identifier) pda_cache[parameter_identifier] = pda # Go to the start of the session pda.GoTo(lap.StartTime) # Get 10 samples parameter_values = pda.GetNextSamples(10, StepDirection.Forward) # Print data and status to console window print(f"Parameter: {parameterIdentifier}") for i in range(10): print( f"Data: {parameter_values.Data[i]}, TimeStamp: {parameter_values.Timestamp[i]} Status: {parameter_values.DataStatus[i]}") for pda in pda_cache.values(): pda.Dispose() # Dispose objects once we are finish with it pda.Dispose() client_session.Close() ``` === "MATLAB" ```matlab % Local database connection string and session key for data already recorded connectionString = "DbEngine=SQLite;Data Source=C:\session01.ssndb;Pooling=false;"; sessionKey = SessionKey.Parse("7DD05707-EAA2-4A36-BB8A-E2327AA52BFC"); % List of parameter identifiers to obtain samples from parameterIdentifiers = ["vCar:Chassis", "nEngine:FIA"]; % Initialise SQLRace API Core.Initialize() sessionManager = SessionManager.CreateSessionManager(); % Load existing session clientSession = sessionManager.Load(sessionKey, connection_string); % Create the pda cache pdaCache = configureDictionary("string","MESL.SqlRace.Domain.ParameterDataAccess"); % Obtain the session session = clientSession.Session; for i=0:session.LapCollection.Count()-1 for parameterIdentifier = parameterIdentifiers % Open and cache PDA if isKey(pdaCache,parameterIdentifier) pda = pdaCache(parameterIdentifier); else pda = session.CreateParameterDataAccess(parameterIdentifier); pdaCache(parameterIdentifier) = pda; end % Go to the start of the session pda.GoTo(session.LapCollection.Item(i).StartTime); % Get 10 samples parameterValues = pda.GetNextSamples(10, StepDirection.Forward); % Print data and status to console window fprintf("Parameter: %s",parameterIdentifier) for j=1:10 fprintf("Data: %f, Timestamp: %i, Status: %s\n",parameterValues.Data(j),parameterValues.Timestamp(j),parameterValues.DataStatus(j)) end end end % Dispose objects once we are finish with it pdas = values(pdaCache,"cell"); for i = 1:length(pdas) pdas{i}.Dispose() end clientSession.Close() ``` --- # Session Export Sessions can be exported to local SSN2 files using the **SQLRace API** and the `SessionManager` object. For instance, a SQLRace Session stored in a SQLRace Database can generate a local SSN2 or SSNDB session file. ### **SessionManager** The `SessionManager` class provides functionality to export sessions into local **SSN2** or **SSNDB** files using the SQLRace API. #### **Export Process Overview** 1. The session is first exported into a `MESL.SqlRace.Common.FragmentedMemoryStream` object using the `ExportSession` method. 2. This `FragmentedMemoryStream` contains all the necessary configuration and data required to reconstruct the session. 3. The resulting stream can then be used with the `ImportSession` method to generate and populate the session in its final form. #### **Key Methods** [SessionManager.ExportSession()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.SessionManager.html#MESL_SqlRace_Domain_SessionManager_ExportSession_MAT_OCS_Core_SessionKey_System_String_) [SessionManager.ImportSession()](https://mat-docs.github.io/Atlas.SQLRaceAPI.Documentation/api/MESL.SqlRace.Domain.SessionManager.html#MESL_SqlRace_Domain_SessionManager_ImportSession_System_String_System_IO_Stream_MAT_OCS_Core_SessionKey_) !!! Failure "Important: Do not use ExportSession alone to create a session file" Calling `SessionManager.ExportSession` with a file path will generate an SSN2 session file, but this file will be unreadable by ATLAS and ignored by the SQLRace API. ##### To create a valid and usable session file: 1. Export the session into a `FragmentedMemoryStream` using `ExportSession`. 2. Then use `ImportSession` to properly populate and finalize the session file. ### Example SQLRace Session to SSN2 Basic example that shows how to export to SSN2 from a SQLRace Database session. Load a session from a SQLRace Database: === "C#" ``` csharp class Program { static void Main() { string connectionStringSQLRACE = @"Data Source=SERVER_NAME\LOCAL;Initial Catalog=SQLRACE01;Integrated Security=True"; SessionManager sessionManager = SessionManager.CreateSessionManager(); SessionState sessionState = SessionState.Historical; var clientSessions = sessionManager.FindBySessionState(sessionState, connectionStringSQLRACE); var session = clientSessions[5]; var clientSession = sessionManager.Load(session.Key, connectionStringSQLRACE); var loadedSession = clientSession.Session; loadedSession.LoadConfiguration(); Console.WriteLine($"Session loaded: {loadedSession.Identifier}"); clientSession.Close(); clientSession.Dispose(); } } ``` === "Python" ``` python connectionStringSQLRACE = f"Data Source=SERVER_NAME\LOCAL;Initial Catalog=SQLRACE01;Integrated Security=True" sessionManager = SessionManager.CreateSessionManager() available_states = {"Created": SessionState.Created, "Exporting": SessionState.Exporting, "Historical": SessionState.Historical, "Live": SessionState.Live, "LiveNotInServer": SessionState.LiveNotInServer, "Indexed": SessionState.Indexed} sessionState = available_states["Historical"] clientSession_filter_state = sessionManager.FindBySessionState(sessionState, connectionStringSQLRACE) session = clientSession_filter_state[5] clientSession = sessionManager.Load(session.Key, connectionStringSQLRACE) session = clientSession.Session session.LoadConfiguration() print(f"Session loaded: {session.Identifier}") clientSession.Close() clientSession.Dispose() ``` === "MATLAB" ``` matlab connectionStringSQLRACE = 'Data Source=MCLA-525Q374\LOCAL;Initial Catalog=SQLRACE01;Integrated Security=True'; sessionManager = NET.invokeGenericMethod('MESL.SqlRace.Domain.SessionManager', 'CreateSessionManager', {}); % Define session state sessionState = MESL.SqlRace.Domain.SessionState.Historical; % Find sessions by state clientSessions = sessionManager.FindBySessionState(sessionState, connectionStringSQLRACE); % Select the 6th session (MATLAB is 1-based) session = clientSessions.Get(6); % Load the session clientSession = sessionManager.Load(session.Key, connectionStringSQLRACE); loadedSession = clientSession.Session; % Load configuration loadedSession.LoadConfiguration(); % Display session identifier disp(['Session loaded: ', char(loadedSession.Identifier)]); % Close and dispose clientSession.Close(); clientSession.Dispose(); ``` Use the **SessionManager** to store the session into memory and then import the session into a local file using the ImportSession method. === "C#" ``` csharp // Session key from the original session var sessionkey = session.Key; // Create the connection string to the SSN2 var localSsn2ConncetionString = $"DbEngine=SQLite;Data Source={ssn2FileName}"; // Export/store session from memory stream memoryStream = sessionManager.ExportSession(sessionKey, connectionString); // Import session from memory stream sessionManager.ImportSession(localSsn2ConncetionString, memoryStream, sessionKey); ``` === "Python" ``` python # Session key from the original session sessionkey = session.Key # Define file path where to export the session filePath = r"C:\ssn2\ExportSession.ssn2" # Create the connection string to the SSN2 localSsn2ConncetionString = f"DbEngine=SQLite;Data Source={filePath}" # Export/store session from memory stream memoryStream = sessionManager.ExportSession(sessionkey, connectionStringSQLRACE) # Import session from memory stream sessionManager.ImportSession(localSsn2ConncetionString, memoryStream, SessionKey.NewKey()) ``` === "MATLAB" ``` matlab % Session key from the original session sessionkey = session.Key; % Define file path where to export the session filePath = 'C:\ssn2\ExportSession.ssn2'; % Create local connection string localSsn2ConnectionString = sprintf('DbEngine=SQLite;Data Source=%s', filePath); % Export/store session to memory stream memoryStream = sessionManager.ExportSession(sessionkey, connectionStringSQLRACE); % Import session from memory stream sessionManager.ImportSession(localSsn2ConnectionString, memoryStream, SessionKey.NewKey()); --- # Session Loading The SQLRace API can load sessions either locally, from an SSNDB, or from an SQLRace database. !!! info "SSN File Loading" SSN files are read-only. ## Session States Session States are used to identify whether a session is currently live and having data recorded to it or historical where data is loaded from the database. | Session State | Description | |-------------|-----------| | Created | The session has been indexed and exists within the database. However, it is not available yet to be loaded either live or historical. This is a temporary state and should not be used to find and load sessions. | | Exporting | The (SQLite) session is exporting. | | Historical | The session is historical. | | Indexed | The session has been added to the database waiting to be indexed. | | Live | Indicates a session which is live and available within an SQLRace server. | | LiveNotInServer | Indicates a session which is live, but not available within an SQLRace server. | !!! warning "LiveNotInServer" Sessions with this session state should not be loaded until its state has changed to historical. This is due to the data in the database not being up-to-date until the recording has ended and all data has been flushed. ## Examples ### Loading a Local SSN File === "C#" ``` csharp const string pathToFile = @"C:\Path\To\Session\File.ssn"; var fileSessionManager = FileSessionManager.CreateFileSessionManager(); var clientSession = fileSessionManager.Load(pathToFile); ``` === "Python" ``` python path_to_file = r"C:\Path\To\Session\File.ssn" file_session_manager = FileSessionManager.CreateFileSessionManager() client_session = file_session_manager.Load(path_to_file) ``` === "MATLAB" ``` matlab pathToFile = sprintf("C:\\Path\\To\\Session\\File.ssn); fileSessionManager = FileSessionManager.CreateFileSessionManager(); clientSession = fileSessionManager.Load(pathToFile); ``` ### Loading a Local SSN2 File === "C#" ``` csharp const string pathToFile = @"C:\Path\To\Session\File.ssn2"; var connectionString = $@"DbEngine=SQLite;Data Source={pathToFile}"; var sessionManager = SessionManager.CreateSessionManager(); var sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); var sessionKey = sessionSummaries.get_Item(0).Key.ToString(); var clientSession = sessionManager.Load(sessionKey, connectionString); ``` === "Python" ``` python path_to_file = r"C:\Path\To\File.ssn2" connection_string = f"DbEngine=SQLite;Data Source={path_to_file}" session_manager = SessionManager.CreateSessionManager() session_summaries = session_manager.FindBySessionState(SessionState.Historical, connection_string) session_key = session_summaries.get_Item(0).Key.ToString() client_session = session_manager.Load(session_key, connection_string) ``` === "MATLAB" ``` matlab pathToFile = sprintf("C:\\Path\\To\\Session\\File.ssn2"); connectionString = sprintf("DbEngine=SQLite;Data Source=%s", pathToFile); sessionManager = SessionManager.CreateSessionManager(); sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); sessionKey = sessionSummaries.Item(0).Key; clientSession = sessionManager.Load(sessionKey, connectionString); ``` ### Loading a Session From an SSNDB === "C#" ``` csharp const string pathToFile = @"C:\Path\To\Session\File.ssndb"; var connectionString = $@"DbEngine=SQLite;Data Source={pathToFile}"; var sessionManager = SessionManager.CreateSessionManager(); var sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); var sessionKey = sessionSummaries.get_Item(0).Key.ToString(); var clientSession = sessionManager.Load(sessionKey, connectionString); ``` === "Python" ``` python path_to_file = r"C:\Path\To\File.ssndb" connection_string = f"DbEngine=SQLite;Data Source={path_to_file}" session_manager = SessionManager.CreateSessionManager() session_summaries = session_manager.FindBySessionState(SessionState.Historical, connection_string) session_key = session_summaries.get_Item(0).Key.ToString() client_session = session_manager.Load(session_key, connection_string) ``` === "MATLAB" ``` matlab pathToFile = sprintf("C:\\Path\\To\\Session\\File.ssndb"); connectionString = sprintf("DbEngine=SQLite;Data Source=%s", pathToFile); sessionManager = SessionManager.CreateSessionManager(); sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); sessionKey = sessionSummaries.Item(0).Key; clientSession = sessionManager.Load(sessionKey, connectionString); ``` ### Loading a Session From an SQLRaceDB === "C#" ``` csharp var serverName = @"SQLServer\InstanceName"; var databaseName = "DatabaseName"; var connectionString = $@"server={serverName};Initial Catalog={databaseName};Trusted_Connection=True"; var sessionManager = SessionManager.CreateSessionManager(); var sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); var sessionKey = sessionSummaries.get_Item(0).Key.ToString(); var clientSession = sessionManager.Load(sessionKey, connectionString); ``` === "Python" ``` python server_name = r"SQLServer\InstanceName" database_name = "DatabaseName" connection_string = f"server={server_name};Initial Catalog={database_name};Trusted_Connection=True" session_manager = SessionManager.CreateSessionManager() session_summaries = session_manager.FindBySessionState(SessionState.Historical, connection_string) session_key = session_summaries.get_Item(0).Key.ToString() client_session = session_manager.Load(session_key, connection_string) ``` === "MATLAB" ``` matlab serverName = "SQLServer\InstanceName"; databaseName = "DatabaseName"; connectionString = sprintf("server=%s;Initial Catalog=%s;Trusted_Connection=True", serverName, databaseName); sessionManager = SessionManager.CreateSessionManager(); sessionSummaries = sessionManager.FindBySessionState(SessionState.Historical, connectionString); sessionKey = sessionSummaries.Item(0).Key; clientSession = sessionManager.Load(sessionKey, connectionString); ``` ### Loading a Session via GUID === "C#" ``` csharp const string serverName = @"SQLServer\InstanceName"; const string dbName = "DatabaseName"; const string connectionString = $@"server={serverName};Initial Catalog={dbName};Trusted_Connection=True"; string sessionKeyGUID = "c5668a15-a996-444b-8cd4-0895e0bd11b3"; var sessionManager = SessionManager.CreateSessionManager(); clientSession = sessionManager.Load(SessionKey.Parse(sessionKeyGUID), connectionString); ``` === "Python" ``` python server_name = r"SQLServer\InstanceName" db_name = "DatabaseName" connection_string = rf"server={server_name};Initial Catalog={db_name};Trusted_Connection=True" session_key_guid = "c5668a15-a996-444b-8cd4-0895e0bd11b3" session_manager = SessionManager.CreateSessionManager() client_session = session_manager.Load(SessionKey.Parse(session_key_guid), connection_string) ``` === "MATLAB" ``` matlab serverName = "SQLServer\InstanceName"; databaseName = "DatabaseName"; connectionString = sprintf("server=%s;Initial Catalog=%s;Trusted_Connection=True", serverName, databaseName); sessionKeyGUID = "c5668a15-a996-444b-8cd4-0895e0bd11b3"; sessionManager = SessionManager.CreateSessionManager(); clientSession = sessionManager.Load(SessionKey.Parse(sessionKeyGUID), connectionString); ``` ### Filter Sessions with QueryManager and Load === "C#" ``` csharp const string serverName = @"SQLServer\InstanceName"; const string dbName = "DatabaseName"; const string connectionString = $@"server={serverName};Initial Catalog={dbName};Trusted_Connection=True"; var filter = new ScalarFilter("Car", MatchingRule.EqualTo, "1", true); var compositeFilter = new CompositeFilter(CombineType.AND); compositeFilter.Add(new ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.GreaterThanOrEquals, "2025-07-14 00:00:00.000", true)); compositeFilter.Add(new ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.LessThan, "2025-07-18 23:00:00.000", true)); var qm = QueryManager.CreateQueryManager(connectionString); qm.Filter = filter; var queryResult = qm.ExecuteQuery(); var sessionManager = SessionManager.CreateSessionManager(); foreach (var sessionSummary in queryResult) { var clientSession = sessionManager.Load(sessionSummary.Key, connectionString); } ``` === "Python" ``` python server_name = r"SQLServer\InstanceName" db_name = "DatabaseName" connection_string = rf"server={server_name};Initial Catalog={db_name};Trusted_Connection=True" # filter by session data item item_filter = ScalarFilter("Car", MatchingRule.EqualTo, "1", True) # Using a composite filter composite_filter = CompositeFilter(CombineType.AND) composite_filter.Add(ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.GreaterThanOrEquals, "2025-07-14 00:00:00.000", True)) composite_filter.Add(ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.LessThan, "2025-07-18 23:00:00.000", True)) qm = QueryManager.CreateQueryManager(connection_string) qm.Filter = item_filter queryResult = qm.ExecuteQuery() # Returns a list of Session Summaries session_manager = SessionManager.CreateSessionManager() for session_summary in queryResult: client_session = session_manager.Load(session_summary.Key, connection_string) ``` === "MATLAB" ``` matlab serverName = "SQLServer\InstanceName"; dbName = "DatabaseName"; connectionString = sprintf('server=%s;Initial Catalog=%s;Trusted_Connection=True;', serverName, dbName); filter = ScalarFilter("Car", MatchingRule.EqualTo, "1", true); compositeFilter = CompositeFilter(CombineType.AND); compositeFilter.Add(ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.GreaterThanOrEquals, "2025-07-14 00:00:00.000", true)); compositeFilter.Add(ScalarFilter(SessionFieldIdentifiers.SessionStartDateTime, MatchingRule.LessThan, "2025-07-18 23:00:00.000", true)); qm = QueryManager.CreateQueryManager(connectionString); qm.Filter = filter; queryResult = qm.ExecuteQuery(); % Get IEnumerable.GetEnumerator() via reflection and invoke it t = queryResult.GetType(); ienumIface = t.GetInterface('System.Collections.IEnumerable'); % the non-generic interface assert(~isempty(ienumIface), 'Object does not implement System.Collections.IEnumerable'); miGetEnum = ienumIface.GetMethod('GetEnumerator'); % MethodInfo emptyArgs = NET.createArray('System.Object', 0); en = miGetEnum.Invoke(queryResult, emptyArgs); % returns an IEnumerator (explicit impl) % Prepare reflection handles for IEnumerator.MoveNext and IEnumerator.Current ienumeratorT = System.Type.GetType('System.Collections.IEnumerator'); miMoveNext = ienumeratorT.GetMethod('MoveNext'); % MethodInfo for bool MoveNext() propCurrent = ienumeratorT.GetProperty('Current'); % PropertyInfo for object Current {get;} miGetCurrent = propCurrent.GetGetMethod(); % MethodInfo for getter % Walk the sequence using interface methods summaries = {}; % MATLAB cell array to collect MESL.SqlRace.Domain.SessionSummary objects while miMoveNext.Invoke(en, emptyArgs) % calls IEnumerator.MoveNext() curr = miGetCurrent.Invoke(en, emptyArgs); % calls IEnumerator.get_Current() summaries{end+1} = curr; end sessionManager = SessionManager.CreateSessionManager(); for j = 1:numel(summaries) ss = summaries{j}; % MESL.SqlRace.Domain.SessionSummary clientSession = sessionManager.Load(ss.Key, connectionString); end ``` --- # Introduction ATLAS ships with a range of standard displays such as: Waveform, Scatterplot and Numeric to name a few. In addition to standard displays, ATLAS is fully customizable, allowing custom displays to be added via plugins created using the ATLAS Display API. Using the Display API, custom displays can be made to look, behave and act just like the standard displays with full access to session and parameter data for custom visualizations tailored to specific third party needs. ## Plugins ATLAS is a modular platform, the ATLAS application executable provides a user interface shell whereby the majority of functionality is provided by plugins. There are essentially three types of plugin - Display Plugins, such as: Waveform, Scatterplot, Numeric - Functionality Plugins, such as: Session Browser, Parameter Browser, Alarms - Recorder Plugins, such as: DTS, File, Advanced Streams A plugin is a standard C# library DLL that exposes an entry point class derived from a suitable base class provided by the ATLAS APIs. At startup, ATLAS will dynamically load compatible plugins that reside within the ATLAS program files folder. ## Architecture Overview ![Architecture](devguide/assets/images/architecture.png) All ATLAS functionality is built on top of the ATLAS Platform and Presentation APIs. These APIs should be seen as "internal/private APIs" that are actively developed and changed between each ATLAS release. The Display API is a thin wrapper over these internal APIs, providing a stable and versioned API for third parties to develop custom display plugins. Changes to the Display API is purely additive, meaning custom display plugins remain compatible with future releases of ATLAS. !!! note All data access is through the SQL Race API. ### Direct use of Platform and Presentation APIs !!! attention The Display API is by its very nature a subset of the internal APIs. When coming across a feature that is not present in the Display API, but is available in the internal APIs, it may be tempting to just reference the internal APIs and use them directly. However, as ATLAS continues to be developed, we reserve the right to change the internal APIs as we see fit without regard for backwards compatibility, There is a high chance that future releases of ATLAS will become incompatible with custom display plugins that use internal APIs, with the most likely consequence being crashes on startup. Before calling internal APIs, please contact Motion Applied for advice, whereby alternative APIs may be recommended, or future enhancements to the Display API will be considered. ## Display API Features Here are the main features of the Display API that can be used to create custom displays. ### Native look and feel Custom displays look, behave and act like standard displays. ### Fully integrated into ATLAS - Appear on main Toolbar - Docking support - Access to Parameter Browser - Properties Window support - Saved to Workbook ### Track common actions via Events - Session loaded/unloaded/associated - Parameters added/removed - Timebase updated - Cursor position changed - Page activated ### Fetch Parameter Samples and Data - Just for primary session or composite sessions - Actual or resampled data ## Display API deployment The Display API is deployed as a NuGet package hosted on GitHub, see [https://github.com/mat-docs/packages](https://github.com/mat-docs/packages) The NuGet packages contains the following ![NuGet Contents](devguide/assets/images/package.png) --- # Getting Started This section is a detailed guide to the [Display API](https://mat-docs.github.io/Atlas.DisplayAPI.Documentation/index.html). !!! tip Have you read the [Introduction](index.md)? Start there to review the architecture and key concepts. The Display API is used by third parties to create custom display visualizations. Custom displays are written in _C#_ using _WPF_ with _MVVM_, see [Required Knowledge](prerequisites.md#required-knowledge) The Display API is published as a [NuGet](https://github.com/mat-docs/packages) package on GitHub. ## Tutorials A number of tutorials are provided to help explain and demonstrate the [features](index.md#display-api-features) of the Display API - Creating a plugin for a custom display from scratch, see [Tutorial 1](devguide/createfromscratch.md) - Diagnostic display to explore Display API features, see [Tutorial 2](devguide/diagnostic.md) - Numeric display that supports compare mode, see [Tutorial 3](devguide/numericcompare.md) - Graphical display that demonstrates richer, more real world visualizations, see [Tutorial 4](devguide/graphical.md) - Quick to create display that visualizes parameter value changes, see [Tutorial 5](devguide/gauge.md) ### Tutorials Setup Before completing the tutorials, you'll need to set up a few things. #### GitHub Account These tutorials use packages and code hosted on [GitHub](https://github.com/mat-docs). !!! note You'll need a free account to access them. #### Development Environment - Visual Studioinstalled - Access to Display API NuGet package - ATLAS installed #### Code Samples Clone our [Atlas.DisplayAPI.Examples](https://github.com/mat-docs/Atlas.DisplayAPI.Examples) git repo for pre-built versions of the tutorials. #### Toolbar Icon Create or find an icon for the custom display. !!! tip A 16x16 pixel PNG file (preferably in non dark colours) ## Detailed explanations of common topics The following common topics are explained separately to avoid repetition and to allow for easier reference - [Factories and Services](factoriesandservices.md) - [Initialization, state and notifications](initialization.md) - [Working with Display Properties](displayprops.md) - [Working with Sessions](sessions.md) - [Working with Timebase and Cursor](timebasecursor.md) - [Working with Display Parameters](parameters.md) - [Working with Data](data.md) - [Recommendations for creating a well behaved Custom Display](goodcitizenship.md) --- # Create Custom Display Plugin from Scratch The following steps demonstrate how to create a custom display plugin from scratch. !!! note The code for this tutorial can be reviewed at [Tutorials/SampleDisplayPlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/SampleDisplayPlugin) A custom display plugin is a .Net DLL that when deployed to the ATLAS program files, is automatically loaded and registered on startup. !!! note Currently only one custom display type is allowed per custom display plugin project. However, any number of instances of a custom display type may be added to a workbook. ## Create Project Create a new _C# WPF User Control Library (.NET Framework)_ project with Visual Studio. !!! tip Enter _wpf_ into the Search box ![Create Project](assets/images/createproject.png) !!! caution Create a _.NET Framework WPF_ project and not a _.Net Core WPF_ project ![Name Project](assets/images/nameproject.png) !!! attention Ensure there is _Plugin_ somewhere within the name, otherwise ATLAS will not load the plugin !!! note This project type includes a template for the _View_ ## Update Assembly Information Edit project settings ![Project Settings](assets/images/editprojectsettings.png) Click _Assembly Information..._ ![Project Settings](assets/images/projectsettings.png) Modify the Title, Description and GUID (if not already set) properties ![Project Settings](assets/images/assemblyinformation.png) The _Title_ property corresponds to the custom display window title ![Window Title](assets/images/windowtitle.png) The _Description_ property corresponds to the custom display icon tooltip ![Toolbar Icon Tooltip](assets/images/icontooltip.png) !!! attention A GUID must be specified otherwise ATLAS will fail to start. Use Tools->Create GUID as necessary ![Tools Create Guid](assets/images/toolscreateguid.png) Copy Registry Format ![Tools Create Guid](assets/images/createguid.png) Remember to remove '{' and '}' from beginning and end of the string ## Add reference to _Atlas.DisplayAPI_ NuGet package Manage NuGet packages of project ![Manage NuGet Packages](assets/images/managenugetpackages.png) Browse to _Atlas.DisplayAPI_ NuGet package and install ![Add Nuget Package](assets/images/nugetpackage.png) Once installed the references should look similar to this ![References](assets/images/references.png) ## Add an icon for the toolbar Add a _Resources_ folder to the project ![Add Resources Folder](assets/images/addresourcesfolder.png) Which should look like ![Resources Folder](assets/images/resourcesfolder.png) Add an existing item to the Resources folder ![Resources Folder](assets/images/addiconasexistingitem.png) !!! note The icon should be a 16 x 16 pixel PNG file compatible with ATLAS dark theme Select and _Add_ icon ![Resources Folder](assets/images/selectandaddicon.png) Which should look like ![Resources Folder](assets/images/icon.png) Ensure _Build Action_ is set to _Resource_ ![Resources Folder](assets/images/iconproperties.png) ## Configure _View_ class The _View_ class presents the custom visualization. - Rename `UserControl1` to `SampleDisplayView` - Rename `UserControl1.xaml` to `SampleDisplayView.xaml` - Update `x:Class` to `SampleDisplayPlugin.SampleDisplayView` - Rename `UserControl1.xaml.cs` to `SampleDisplayView.xaml.cs` - Rename `UserControl1` class to `SampleDisplayView` ```c# hl_lines="3 5" namespace SampleDisplayPlugin { public partial class SampleDisplayView { public SampleDisplayView() { InitializeComponent(); } } } ``` - Add to the _XAML_ a simple `` to display white text ```xml hl_lines="9-13" ``` !!! note Text is white to be compatible with the ATLAS dark theme ## Add _View Model_ class The _View Model_ class provides the business logic behind the custom visualization. Add a _New Item_ to the project ![Add New Item](assets/images/addnewitem.png) Add a _Class_ named _SampleDisplayViewModel_ to the project ![Add Class](assets/images/addviewmodelclass.png) !!! important Remember to derive the _View Model_ class from the _DisplayPluginViewModel_ base class, otherwise the custom display will not be available to ATLAS ```c# hl_lines="5" using MAT.Atlas.Client.Presentation.Displays; namespace SampleDisplayPlugin { public sealed class SampleDisplayViewModel : DisplayPluginViewModel { } } ``` !!! note Subsequent tutorials will expand the _View Model_ class to interact with the _View_ and retrieve data from ATLAS to visualize. ## Add _Plugin_ class The _Plugin_ class registers the _View_ class, _View Model_ class and _Icon_ location with ATLAS. - Create a class named _Plugin_ as per [View Model class](#add-view-model-class) - Derive from _DisplayPlugin<Plugin>_ - Decorate class with _DisplayPlugin_ attribute - Set appropriate values for: _View_, _ViewModel_ and _IconUrl_ properties ```c# hl_lines="5-9" using MAT.Atlas.Client.Presentation.Plugins; namespace SampleDisplayPlugin { [DisplayPlugin( View = typeof(SampleDisplayView), ViewModel = typeof(SampleDisplayViewModel), IconUri = "Resources/icon.png")] public sealed class Plugin : DisplayPlugin { } } ``` ## Add _PluginModule_ class The _PluginModule_ class provides the entry point for ATLAS to initialise the custom display plugin. - Create a class named _PluginModule_ as per [View Model class](#add-view-model-class) - Derive from _Module_ (provided by _Autofac_) - Decorate class with _Export_ attribute (provided by _MEF_) - Add a reference to _System.ComponentModel.Composition_ if required ![Project References](assets/images/mefreference.png) - Override _Load_ method and register display plugin ```c# hl_lines="10-11 13 15" using System.ComponentModel.Composition; using Autofac; using Autofac.Core; using MAT.Atlas.Client.Presentation.Plugins; namespace SampleDisplayPlugin { [Export(typeof(IModule))] public sealed class PluginModule : Module { protected override void Load(ContainerBuilder builder) { DisplayPlugin.Register(builder); } } } ``` ## Optional: combine _PluginModule_ and _Plugin_ classes To reduce boilerplate code the _plugin_ class may be nested within the _PluginModule_ class ```c# hl_lines="13-23" using System.ComponentModel.Composition; using Autofac; using Autofac.Core; using MAT.Atlas.Client.Presentation.Plugins; namespace SampleDisplayPlugin { [Export(typeof(IModule))] public sealed class PluginModule : Module { protected override void Load(ContainerBuilder builder) { Plugin.Register(builder); } [DisplayPlugin( View = typeof(SampleDisplayView), ViewModel = typeof(SampleDisplayViewModel), IconUri = "Resources/icon.png")] private sealed class Plugin : DisplayPlugin { } } } ``` ## Build, Deploy and Debug - Build _Debug_ or _Release_ solution configuration - Copy plugin DLL to ATLAS 10 program files - Manually copy from bin/Debug or bin/Release, or - Add a post build step to automatically copy, see [Deployment Script](deploymentscript.md) - Debug using ATLAS 10 (updating the project settings to start ATLAS on debugging) ![Debug Project Settings](assets/images/debugprojectsettings.png) - Assuming all went well, there should be a toolbar icon for the custom display in ATLAS ![Icon](assets/images/icontooltip.png) - Clicking the custom display toolbar icon should display a window containing the contents of the _View_ ![Display Window](assets/images/customdisplaywindow.png) --- # Enhanced Diagnostic display A useful way to explore the [features](../index.md#display-api-features) of the Display API is to create a diagnostic display to exercise various aspects of the API and log information when interacting with the display and view the results. !!! note The code for this tutorial can be reviewed at [Tutorials/EnhancedDisplayPlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/EnhancedDisplayPlugin) With the ability to log messages, we can start to explore the API by overriding each of the _View Model_ virtual methods and then experiment with the display in ATLAS to see when and in what situations these overridden methods are called. For state, methods and services we can explore these by adding additional user interface elements to the _View_ to view the results and any side effects of interacting with them. Start the tutorial by [creating a new display from scratch](createfromscratch.md) named `EnhancedDisplayPlugin`. ## Add logging Modify the _XAML_ of the _View_ to contain a `TextBox` to view the diagnostic log ```xml hl_lines="9-13" ``` Add a `string` property named `LogText` to the _View Model_ (see [Display Properties](../displayprops.md) for further information) ```c# [Browsable(false)] public string LogText { get => this.logText; set => this.SetProperty(ref this.logText, value); } ``` Add the following backers - `SynchronizationContext` field named `synchronizationContext` to update the `TextBox` on the UI thread - `List` field named `pendingLogMessages` to hold pending log messages - `object` field named `logLock` to prevent multiple threads accessing the log at the same time ```c# private readonly SynchronizationContext synchronizationContext; private readonly List pendingLogMessages = new List(); private readonly object logLock = new object(); ``` Set the `SynchronizationContext` field in the _View Model_ constructor. ```c# hl_lines="3" public SampleDisplayViewModel() { this.synchronizationContext = SynchronizationContext.Current; } ``` !!! attention Notification methods and signal result handlers may be called concurrently on separate threads, therefore some form of thread synchronization is required when accessing shared resources. User Interface must be updated on the UI thread, the [`SynchronizationContext` class](https://docs.microsoft.com/en-us/dotnet/api/system.threading.synchronizationcontext) is used in this tutorial. Add a helper method named `LogMessage` to update the log ```c# private static void LogMessage( SynchronizationContext synchronizationContext, ICollection pendingMessages, object logLock, Func readLog, Action writeLog, string message) { bool firstNewMessage; lock (logLock) { firstNewMessage = !pendingMessages.Any(); pendingMessages.Add(message); } if (firstNewMessage) { synchronizationContext.Post( _ => { lock (logLock) { var stringBuilder = new StringBuilder(readLog()); foreach (var pendingMessage in pendingMessages) { stringBuilder.AppendLine(pendingMessage); } pendingMessages.Clear(); writeLog(stringBuilder.ToString()); } }, null); } } ``` Finally add a simplified method named `Log` to log a single message ```c# private void Log(string message) { LogMessage( this.synchronizationContext, this.pendingLogMessages, this.logLock, () => this.LogText, v => this.LogText = v, message); } ``` ## Exploring state The API state is the value of any properties exposed by the _View Model_ and any properties obtained from services, e.g. configured parameters or session summary. To view the API state at any time add a `Button` to the _View XAML_ ```xml ``` !!! note Layout the User Interface using an appropriate WPF panel, such as: `Grid`, `StackPanel` or `WrapPanel`. Add a _command handler_ named `LogPropertiesCommand` to the _View Model_ ```c# [Browsable(false)] public ICommand LogPropertiesCommand { get; } ``` Initialise the `LogPropertiesCommand` _command handler_ in the _View Model_ constructor ```c# this.LogPropertiesCommand = new DelegateCommand(this.OnLogProperties); ``` Implement the _command handler_ by logging appropriate property values ```c# private void OnLogProperties() { this.Log(nameof(OnLogProperties)); this.Log($" IsSelected: {this.IsSelected}"); this.Log($" CanRetrieveData: {this.CanRetrieveData}"); this.Log($" ScopeIdentity.Identity: {this.ScopeIdentity.Identity}"); // Log this.ActiveCompositeSessionContainer // Log this.displayParameterService.ParameterContainers ``` !!! note The code for this tutorial [Tutorials/EnhancedDisplayPlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/EnhancedDisplayPlugin) provides a full implementation of `OnLogProperties`. The code also allows you to log the properties periodically, this is useful to demonstrate how the API state changes whilst the display is non-interactive, e.g. covered by another display or a different page is active. ## Exploring notifications Notification methods are virtual methods exposed from the _View Model_ and are called when specific events occur in ATLAS, e.g. when the active page is changed. Override notification methods and add log statements to view the value of any arguments, e.g. ```c# public override void OnActiveDisplayPageChanged(bool isActive) { this.Log(nameof(OnActiveDisplayPageChanged)); this.Log($" IsActive: {isActive}"); } ``` Add overrides for all the notification methods as detailed in the following sections ### [Initialization and state change notifications](../initialization.md) - `OnInitialised()` - `OnActiveDisplayPageChanged()` - `OnCanRenderDisplayChanged()` ### [Session notifications](../sessions.md) - `OnCompositeSessionLoaded()` - `OnCompositeSessionUnLoaded()` - `OnCompositeSessionContainerChanged()` ### [Display Parameter notifications](../parameters.md) - `OnParameterContainerAdded()` - `OnParameterContainerRemoved()` - `OnParameterAdded()` - `OnParameterRemoved()` ### [Timebase and cursor notifications](../timebasecursor.md) - `OnCursorDataPointChanged(ICompositeSession)` - `OnSessionTimeRangeChanged(ICompositeSession)` ## Exploring timebase/cursor Using `ISessionCursorService`, it is possible to set the cursor timestamp by calling `MoveCursor()` To gain access to `ISessionCursorService`, add a _backer_ and inject an instance into the _View Model_ constructor ```c# private readonly ISessionCursorService sessionCursorService; public SampleDisplayViewModel(ISessionCursorService sessionCursorService) { this.sessionCursorService = sessionCursorService; } ``` !!! note See [Factories and Services](../factoriesandservices.md) for an explanation of other services Add a `Button` to the _View XAML_ ```xml ``` Add a _command handler_ named `CentreCursorCommand` ```c# [Browsable(false)] public ICommand CentreCursorCommand { get; } ``` Initialise the `CentreCursorCommand` _command handler_ in the _View Model_ constructor ```c# this.CentreCursorCommand = new DelegateCommand(this.OnCentreCursor); ``` Implement the _command handler_ by calculating the timestamp at the centre of the current timebase and passing to `MoveCursor()` ```c# private void OnCentreCursor() { if (!this.CanRetrieveData || !this.ActiveCompositeSessionContainer.IsPrimaryCompositeSessionAvailable) { return; } var primarySession = this.ActiveCompositeSessionContainer.CompositeSessions.FirstOrDefault(c => c.IsPrimary); if (primarySession != null) { var centreTimestamp = (primarySession.TimebaseRange.Start + primarySession.TimebaseRange.End) / 2; this.sessionCursorService.MoveCursor(primarySession, centreTimestamp); } } ``` ## Exploring data requests [Data Requests](../data.md) can be explored by logging the value of parameters when the cursor is changed. Ensure display parameters are allowed by adding the _DisplayPluginSettings_ attribute to the _View Model_ ```c# hl_lines="1" [DisplayPluginSettings(ParametersMaxCount = 100)] public sealed class SampleDisplayViewModel : DisplayPluginViewModel ``` Add _backers_ for the required services, then inject and initialize ```c# private readonly ISignalBus signalBus; private readonly IDataRequestSignalFactory dataRequestSignalFactory; private IDisplayParameterService displayParameterService; public SampleDisplayViewModel( ISignalBus signalBus, IDataRequestSignalFactory dataRequestSignalFactory) { this.signalBus = signalBus; this.dataRequestSignalFactory = dataRequestSignalFactory; } protected override void OnInitialised() { this.displayParameterService = this.ServiceContext.DisplayParameterService; } ``` Register with the `SignalBus` a _data result handler_ for sample requests to the _View Model_ constructor ```c# this.signalBus.Subscribe( this.HandleSampleResultSignal, r => r.SourceId == this.ScopeIdentity.Guid) ``` !!! note The result from calling `Subscribe` should be properly disposed, the tutorial code demonstrates this. Make a _sample data request_ for each _display parameter_ when the cursor changes ```c# public override void OnCursorDataPointChanged(ICompositeSession compositeSession) { if (!this.CanRetrieveData) { return; } foreach (var primaryParameter in this.displayParameterService.PrimaryParameters) { var signal = this.dataRequestSignalFactory.CreateSampleRequestSignal( this.ScopeIdentity.Guid, compositeSession.Key, primaryParameter, compositeSession.CursorPoint + 1, 1, SampleDirection.Previous); this.signalBus.Send(signal); } } ``` !!! hint Since there may not be an exact sample value at the cursor timestamp, it is best practice to +1 to the timestamp and request the previous sample value. Log the sample value for the the _display parameter_ if valid ```c# private void HandleSampleResultSignal(SampleResultSignal signal) { var request = signal.Data.Request; var result = signal.Data; var parameterValues = result.ParameterValues; if (parameterValues.SampleCount == 1) { this.Log(nameof(HandleSampleResultSignal)); this.Log($" Parameter: {request.Parameter.Identifier}, Value: {parameterValues.Data[0]}"); } } ``` !!! caution Due to reuse of the arrays contained within `ParameterValues`, the `Length` property of those arrays should not be used. - Use the `SampleCount` property to determine the extent of the returned values. - If you wish to retain a copy of `ParameterValues` you must call `Lock()`, and then `Unlock()` to release the reference when no longer required ### Testing the display To view the parameter values - Add a session via the _Session Browser_ to the compare set associated with the display - Add some display parameters to the display via the _Parameter Browser_ - Use a _Waveform_ display to change the cursor ## Other API features The tutorial code also demonstrates the following API features - [Working with Display Properties](../displayprops.md) - Demonstrates adding and handling properties within the the properties window - Tracking timebase changes and logging the minimum and maximum values of configured parameters - Demonstrates issuing and handling data requests between two timestamps --- # Numeric Compare display This tutorial demonstrates the use of [Composite Data Requests](../data.md#data-request-properties) for [Compare Sets](../sessions.md#sessions-object-model). The existing Numeric display within ATLAS only displays values for the primary session, therefore a useful way to explore compare mode is to create a numeric like display to view the value of each display parameter for each session within a compare set. !!! note The code for this tutorial can be reviewed at [Tutorials/NumericCompareDisplayPlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/NumericCompareDisplayPlugin) Start the tutorial by [creating a new display from scratch](createfromscratch.md) named `NumericCompareDisplayPlugin`. The user interface should consist of a grid of display parameters vs sessions, where each cell is the display parameter value for that session. - The first column should be the display parameter name. ## Update the _View_ class Configure the user interface as follows - Use an `ItemsControl` to display a collection of cell values - Bind `ItemsSource` attribute to the _View Model_ `CellValues` property - By default items are displayed as a vertical list, set the `ItemsPanel` element to a `UniformGrid` instead - Bind the `Rows` attribute to the _View Model_ `RowCount` property - Bind the `Columns` attribute to the _View Model_ `ColumnCount` property - Set the `ItemTemplate` element to a `DataTemplate` containing a uniform `ViewBox` containing a white `TextBlock` displaying a cell value - Bind the `Text` attribute to the _Cell View Model_ `Value` property ```xml ``` ## Add _Cell View Model_ class Add a simple _View Model_ class to represent a generic cell value ```c# public sealed class CellViewModel : BindableBase { private object obj; public CellViewModel(object obj) { this.obj = obj; } public object Value { get => this.obj; set => SetProperty(ref this.obj, value); } } ``` !!! note `BindableBase` provides an implementation of [INotifyPropertyChanged](https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.inotifypropertychanged) Since the `Value` property setter calls the `SetProperty()` method, the User Interface is automatically refreshed when a new value is set. ## Update the _View Model_ class Derive from [TemplateDisplayViewModelBase](displaypluginlibrary.md#templatedisplayviewmodelbase-view-model-base-class) and allow display parameters by specifying the [`DisplayPluginSettings`](../parameters.md) attribute ```c# [DisplayPluginSettings(ParametersMaxCount = 100)] public sealed class SampleDisplayViewModel : TemplateDisplayViewModelBase ``` Add the following backers - `object` field named `parameterLock` to prevent multiple threads accessing the display parameters at the same time - `List` field named `compositeSessionKeys` to hold the collection of configured composite session keys - `List` field named `parameterIdentifiers` to hold the collection of configured parameter identifiers - `int` field named `rowCount` as backer for the `RowCount` property - `int` field named `columnCount` as backer for the `ColumnCount` property ```c# private readonly object parameterLock = new object(); private List compositeSessionKeys = new List(); private List parameterIdentifiers = new List(); private int rowCount; private int columnCount; ``` Inject the `ISignalBus`, `IDataRequestSignalFactory` and `ILogger` services into the _View Model_ constructor and pass to the base constructor ```c# public SampleDisplayViewModel( ISignalBus signalBus, IDataRequestSignalFactory dataRequestSignalFactory, ILogger logger) : base(signalBus, dataRequestSignalFactory, logger) ``` Add the following properties - `int` property named `ColumnCount` to specify the number of columns (composite sessions count + 1 for display parameter name) - `int` property named `RowCount` to specify the number of rows (display parameters count) - `ObservableCollection` property named `CellValues` to hold display parameter properties ```c# [Browsable(false)] public int ColumnCount { get => this.columnCount; set => SetProperty(ref this.columnCount, value); } [Browsable(false)] public int RowCount { get => this.rowCount; set => SetProperty(ref this.rowCount, value); } [Browsable(false)] public ObservableCollection CellValues { get; } = new ObservableCollection(); ``` Subscribe to `CompositeSampleResultSignal` in the _View Model_ constructor to handle the result of a _composite sample data request_ ```c# this.Disposables.Add( this.SignalBus.Subscribe this.HandleCompositeSampleResultSignal, r => r.SourceId == this.ScopeIdentity.Guid)); ``` !!! note The _subscription instance_ is added to the `Disposables` property for automatic cleanup on dispose. Override the `OnMakeCursorDataRequestsAsync()` method to issue a _composite sample data request_ per display parameter when the cursor timestamp changes ```c# protected override async Task OnMakeCursorDataRequestsAsync(ICompositeSession compositeSession) { await this.ExecuteOnUiAsync(this.SyncParameters); foreach (var parameterContainer in DisplayParameterService.ParameterContainers) { var signal = this.DataRequestSignalFactory.CreateCompositeSampleRequestSignal( this.ScopeIdentity.Guid, this.ActiveCompositeSessionContainer.Key, parameterContainer, compositeSession.CursorPoint + 1, 1, SampleDirection.Previous); this.SignalBus.Send(signal); } } ``` !!! note The `CellValues` property is first updated by calling the `SyncParameters()` method on the UI thread Add the `SyncParameters()` method to synchronize the `CellValues` property with configured composite sessions and display parameters ```c# private void SyncParameters() { lock (this.parameterLock) { var compositeSessions = this.ActiveCompositeSessionContainer?.CompositeSessions?.ToList(); var parameterContainers = this.DisplayParameterService.ParameterContainers.ToList(); if ((compositeSessions?.Count ?? 0) == 0 || parameterContainers.Count == 0) { this.compositeSessionKeys.Clear(); this.parameterIdentifiers.Clear(); this.RowCount = 0; this.ColumnCount = 0; this.CellValues.Clear(); return; } var newCompositeSessionKeys = compositeSessions.Select(cs => cs.Key).ToList(); var newParameterIdentifiers = new List(); var newCellValues = new List(); foreach (var parameterContainer in parameterContainers) { newParameterIdentifiers.Add(parameterContainer.InstanceIdentifier); newCellValues.Add(new CellViewModel(parameterContainer.Name)); var oldParameterIndex = this.parameterIdentifiers.IndexOf(parameterContainer.InstanceIdentifier); foreach (var newCompositeSessionKey in newCompositeSessionKeys) { var oldCompositeSessionIndex = this.compositeSessionKeys.IndexOf(newCompositeSessionKey); if (oldParameterIndex < 0 || oldCompositeSessionIndex < 0) { newCellValues.Add(new CellViewModel(string.Empty)); continue; } var parameterValueIndex = GetCellIndex(oldParameterIndex, oldCompositeSessionIndex); var oldParameterValue = this.CellValues[parameterValueIndex]; newCellValues.Add(oldParameterValue); } } this.compositeSessionKeys = newCompositeSessionKeys; this.parameterIdentifiers = newParameterIdentifiers; this.RowCount = this.parameterIdentifiers.Count; this.ColumnCount = this.compositeSessionKeys.Count + 1; this.CellValues.Clear(); foreach (var newCellValue in newCellValues) { this.CellValues.Add(newCellValue); } } this.MakeDataRequests(true, false); } ``` Add the `HandleCompositeSampleResultSignal()` method to update the appropriate cell values with the results of the _composite sample data request_ ```c# private void HandleCompositeSampleResultSignal(CompositeSampleResultSignal signal) { lock (parameterLock) { var request = signal.Data.Request; var parameterIndex = this.parameterIdentifiers.IndexOf(request.ParameterContainer.InstanceIdentifier); if (parameterIndex < 0) { return; } var result = signal.Data; foreach (var kvp in result.Results) { var compositeSessionIndex = this.compositeSessionKeys.IndexOf(kvp.Key); if (compositeSessionIndex < 0) { continue; } var parameterValues = kvp.Value.ParameterValues; if (parameterValues.SampleCount == 0) { continue; } parameterValues.Lock(); try { var cellIndex = this.GetCellIndex(parameterIndex, compositeSessionIndex); this.CellValues[cellIndex].Value = parameterValues.Data[0]; } finally { parameterValues.Unlock(); } } } } ``` Add the `GetCellIndex()` method to convert a display parameter and session grid coordinate into a cell index ```c# private int GetCellIndex(int parameterIndex, int compositeSessionIndex) { var parameterValueIndex = parameterIndex * (this.compositeSessionKeys.Count + 1) + 1 + compositeSessionIndex; return parameterValueIndex; } ``` ## Testing the display To view the display parameter values - Add at least two sessions via the _Session Browser_ to the compare set associated with the display - Add some display parameters to the display via the _Parameter Browser_ - Use a _Waveform_ display to change the cursor --- # Graphical display This tutorial demonstrates how to create a display with interactive graphics, as the saying goes _"a picture is worth a thousand words"_. WPF is a graphics rich user interface technology with sophisticated support for high resolution bitmap and vector graphics with complex animations, transitions and effects. WPF provides a multitude of ways to display graphics with competing programming complexity and performance characteristics. This tutorial demonstrates a couple of approaches at each end of the complexity/performance spectrum - Use of [Shapes](https://docs.microsoft.com/en-us/dotnet/desktop/wpf/graphics-multimedia/shapes-and-basic-drawing-in-wpf-overview) that are easy to program but performance will suffer if over used - Use of [DrawingContext](https://docs.microsoft.com/en-us/dotnet/api/system.windows.media.drawingcontext) that is harder to program but more suitable for highly interactive complex vector graphics !!! note The code for this tutorial can be reviewed at [Tutorials/DriverDisplayPlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/DriverDisplayPlugin) ## Common setup Start the tutorial by [creating a new display from scratch](createfromscratch.md) named `DriverDisplayPlugin`. ### Update the _View Model_ class Derive from [TemplateDisplayViewModelBase](displaypluginlibrary.md#templatedisplayviewmodelbase-view-model-base-class) and allow parameters by specifying the [`DisplayPluginSettings`](../parameters.md) attribute ```c# [DisplayPluginSettings(ParametersMaxCount = 2)] public sealed class SampleDisplayViewModel : TemplateDisplayViewModelBase ``` Inject the `ISignalBus`, `IDataRequestSignalFactory` and `ILogger` services into the _View Model_ constructor and pass to the base constructor ```c# public SampleDisplayViewModel( ISignalBus signalBus, IDataRequestSignalFactory dataRequestSignalFactory, ILogger logger) : base(signalBus, dataRequestSignalFactory, logger) ``` ## Using Shapes For the majority of use cases, WPF shapes provides an ideal compromise between programming simplicity and performance. The shape elements can be added to the _View XAML_ as per standard User Interface elements such as `TextBlock`, and standard binding to _View Model_ properties can be used to change attributes such as colour, position or size. WPF animations can also be used to add additional flare as required. A perfect example of using WPF shapes is to represent bit field parameters with appropriate graphics. This example shows how to display a dynamic representation of the PCU8D shift light LEDs. The LEDs are represented by a single parameter (e.g. `NLEDStatusAll`, verify from ATLAS settings or ask your System Administrator) ![PCU8D Settings]( assets/images/pcu8dsettings.png) Bit positions are mapped to the following LEDs ![PCU8D Bit Mapping]( assets/images/pcu8dbitmapping.png) ### Add _LED View Model_ class Add a simple _View Model_ class to represent an LED - `Color` field named `offColor` that is the colour displayed when the LED is off (passed to constructor) - `Color` field named `onColor` that is the colour displayed when the LED is on (passed to constructor) - `int` field named `bitIndex` that is bit position of the LED (passed to constructor) - `Color` field named `color` as backer for the `Color` property - `double` field named `shiftY` as backer for the `ShiftY` property (passed to constructor) - `Color` property named `Color` that is the current color of the LED - `double` property named `ShiftY` that is the optional distance to shift LED in the Y direction - `UpdateColor()` method that takes a `BitArray` and sets the LED color based on the state of the bit at `bitIndex` ```c# public sealed class LedViewModel : BindableBase { private readonly Color offColor; private readonly Color onColor; private readonly int bitIndex; private Color color; private double shiftY; public LedViewModel(Color offColor, Color onColor, int bitIndex, double shiftY) { this.offColor = offColor; this.onColor = onColor; this.bitIndex = bitIndex; this.shiftY = shiftY; this.color = offColor; } public Color Color { get => this.color; set => SetProperty(ref this.color, value); } public double ShiftY { get => this.shiftY; set => SetProperty(ref this.shiftY, value); } public void UpdateColor(BitArray bits) => this.Color = bits[bitIndex] ? this.onColor : this.offColor; } ``` ### Update the _View_ class Configure the user interface as follows - Add [`ColorToSolidColorBrushValueConverter`](displaypluginlibrary.md#converters) to the `Resources` - Wrap a fixed size `Grid` of `Height=1000` within a `ViewBox` and `DockPanel` - Configure two rows of `Height=100` and `Height=*` - Configure three columns of `Width=100`, `Width=*` and `Width=100` - Use an `ItemsControl` to display the top shift lights at `Row=0` and `Column=1` - Bind `ItemsSource` attribute to the _View Model_ `TopShiftLights` property - Set the `ItemsPanel` element to a `UniformGrid` with `Rows=1` and `Columns=15` - Set the `ItemTemplate` element to a `DataTemplate` containing a uniform `ViewBox` - Set the `RenderTransform` element to a `TranslateTransform` - Set `X=0` - Bind the `Y` attribute to the _LED View Model_ `ShiftY` property - Set the `ViewBox` contents to an `Ellipse` shape of `Width=100` and `Height=100` - Bind the `Fill` attribute to the _LED View Model_ `Color` property with `Converter=ColorToSolidColorBrushValueConverter` - Use an `ItemsControl` to display the left shift lights at `Row=1` and `Column=0` - Bind `ItemsSource` attribute to the _View Model_ `LeftShiftLights` property - Set the `ItemsPanel` element to a `StackPanel` with `Orientation=Vertical` - Set the `ItemTemplate` element to a `DataTemplate` containing a uniform `ViewBox` containing an `Ellipse` shape of `Width=100` and `Height=100` - Bind the `Fill` attribute to the _LED View Model_ `Color` property with `Converter=ColorToSolidColorBrushValueConverter` - Use an `ItemsControl` to display the right shift lights at `Row=1` and `Column=2` - Bind `ItemsSource` attribute to the _View Model_ `RightShiftLights` property - Set the `ItemsPanel` element to a `StackPanel` with `Orientation=Vertical` - Set the `ItemTemplate` element to a `DataTemplate` containing a uniform `ViewBox` containing `Ellipse` shape of `Width=100` and `Height=100` - Bind the `Fill` attribute to the _LED View Model_ `Color` property with `Converter=ColorToSolidColorBrushValueConverter` ```xml ``` ### Update the _View Model_ class Subscribe to `SampleResultSignal` in the _View Model_ constructor to handle the result of a _sample data request_ ```c# this.Disposables.Add( this.SignalBus.Subscribe this.HandleSampleResultSignal, r => r.SourceId == this.ScopeIdentity.Guid)); ``` Add an [OperationTracker<SampleRequestSignal>](displaypluginlibrary.md#tracking-operations) field named `sampleRequestTracker` to throttle sample data requests ```c# private readonly OperationTracker sampleRequestTracker; ``` Instantiate `sampleRequestTracker` in the _View Model_ constructor ```c# this.sampleRequestTracker = new OperationTracker(ThrottleInterval, signalBus.Send); ``` Add static fields for the various LED on and off colors ```c# private static readonly Color BlueOff = Color.FromArgb(255, 0, 0, 40); private static readonly Color BlueOn = Color.FromArgb(255, 0, 0, 255); private static readonly Color GreenOff = Color.FromArgb(255, 0, 40, 0); private static readonly Color GreenOn = Color.FromArgb(255, 0, 255, 0); private static readonly Color RedOff = Color.FromArgb(255, 40, 0, 0); private static readonly Color RedOn = Color.FromArgb(255, 255, 0, 0); private static readonly Color YellowOff = Color.FromArgb(255, 40, 40, 0); private static readonly Color YellowOn = Color.FromArgb(255, 255, 255, 0); ``` Add properties that represent the left, top and right LED strips ```c# [Browsable(false)] public List TopShiftLights { get; } = new List() { new LedViewModel(GreenOff, GreenOn, 14, 15), new LedViewModel(GreenOff, GreenOn, 13, 10), new LedViewModel(GreenOff, GreenOn, 12, 5), new LedViewModel(GreenOff, GreenOn, 11, 0), new LedViewModel(GreenOff, GreenOn, 10, -5), new LedViewModel(RedOff, RedOn, 9, -10), new LedViewModel(RedOff, RedOn, 8, -10), new LedViewModel(RedOff, RedOn, 7, -10), new LedViewModel(RedOff, RedOn, 6, -10), new LedViewModel(RedOff, RedOn, 5, -10), new LedViewModel(BlueOff, BlueOn, 4, -5), new LedViewModel(BlueOff, BlueOn, 3, 0), new LedViewModel(BlueOff, BlueOn, 2, 5), new LedViewModel(BlueOff, BlueOn, 1, 10), new LedViewModel(BlueOff, BlueOn, 0, 15) }; [Browsable(false)] public List LeftShiftLights { get; } = new List() { new LedViewModel(YellowOff, YellowOn, 20, 0), new LedViewModel(RedOff, RedOn, 18, 0), new LedViewModel(BlueOff, BlueOn, 16, 0) }; [Browsable(false)] public List RightShiftLights { get; } = new List() { new LedViewModel(YellowOff, YellowOn, 19, 0), new LedViewModel(RedOff, RedOn, 17, 0), new LedViewModel(BlueOff, BlueOn, 15, 0) }; ``` Override the `OnMakeCursorDataRequestsAsync()` method to issue a _sample data request_ for the first display parameter when the cursor timestamp changes ```c# protected override Task OnMakeCursorDataRequestsAsync(ICompositeSession compositeSession) { if (this.DisplayParameterService.PrimaryParameters.Count < 1) { return Task.CompletedTask; } var signal = this.DataRequestSignalFactory.CreateSampleRequestSignal( this.ScopeIdentity.Guid, compositeSession.Key, this.DisplayParameterService.PrimaryParameters.FirstOrDefault(), compositeSession.CursorPoint + 1, 1, SampleDirection.Previous); this.sampleRequestTracker.Add(signal); return Task.CompletedTask; } ``` !!! note Rather than send the signal directly to the `SignalBus`, add it to `sampleRequestTracker` Add the `HandleSampleResultSignal()` method to update the LED colours ```c# private void HandleSampleResultSignal(SampleResultSignal signal) { var result = signal.Data; var parameterValues = result.ParameterValues; parameterValues.Lock(); try { if (parameterValues.SampleCount == 1) { var shiftLightsValue = (uint) parameterValues.Data[0]; var bits = new BitArray(BitConverter.GetBytes(shiftLightsValue)); this.TopShiftLights.ForEach(sl => sl.UpdateColor(bits)); this.LeftShiftLights.ForEach(sl => sl.UpdateColor(bits)); this.RightShiftLights.ForEach(sl => sl.UpdateColor(bits)); } } finally { parameterValues.Unlock(); this.sampleRequestTracker.Complete(); } } ``` !!! note In this instance the _View Model_ properties are being modified on the Task Pool. WPF will handle this special case, more sophisticated updates need to be done by calling the `ExecuteOnUiAsync` method. !!! attention It is important to call `Complete()` on `OperationTracker` to allow subsequent operations to execute ### Testing the display To view the LED graphics - Add a session via the _Session Browser_ to the compare set associated with the display - Add the PCU8D LED display parameter (e.g. `NLEDStatusAll`) via the _Parameter Browser_ - Use a _Waveform_ display to change the cursor ## Interactive vector graphics For complex graphics that change regularly, WPF provides the low-level and efficient `DrawingContext` class. The [Display Plugin Library `VisualLayer` class](displaypluginlibrary.md#drawing-graphics) provides a simple way to add `DrawingContext` graphics to a display. This example shows how to display a dynamic trace of a parameter and simple cursor. The simplest way to display a trace is to fetch all the samples of a parameter and draw lines between each. The trouble is, when viewing multiple laps of a high frequency parameter there could be 100,000+ samples, however even at 4K resolution we couldn't possibly resolve that many samples. Therefore we should re-sample the parameter to a more manageable resolution, e.g. 1000 data points. Therefore by making a [_data request_](../data.md#data-request-properties) instead of a _sample data request_ it doesn't matter how many actual samples there are within the time range - The time range is divided into an equal time interval per data point, the sample mode determines how the actual samples that fall within each interval are summarized. The [`Trace` class](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/blob/master/DriverDisplayPlugin/Trace.cs) converts the result of a _data request_ into a collection of lines. The [`Cursor` class](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/blob/master/DriverDisplayPlugin/Cursor.cs) converts the cursor timestamp into a vertical line. !!! note Since the purpose of this example is to demonstrate graphics, `Trace` and `Cursor` classes are provided as is and are not guaranteed to cover all edge cases. ### Update the _View_ class Starting with the XAML from the above example - Add two overlapping `VisualLayer` XAML elements within the `Grid` at `Row=1`and `Column=1` ```xml ``` !!! note Having separate visual layers for trace and cursor allows them to be drawn independently, improving performance when only one is changing. `DrawingContext` is a retained mode graphics system, therefore the previous content persists until refreshed. In the code behind, associate each `VisualLayer` with an associated _View Model_ property ```c# public override void OnApplyTemplate() { base.OnApplyTemplate(); if (this.DataContext is SampleDisplayViewModel vm) { vm.TraceVisual = this.TraceVisualLayer.Visual; vm.CursorVisual = this.CursorVisualLayer.Visual; } } ``` ### Update the _View Model_ class Add the following backers - `Pen` field named `cursorPen` that is the graphics pen used to draw the cursor - [OperationTracker<(DataRequestSignal, IDisplayParameterContainer)>](displaypluginlibrary.md#tracking-operations) field named `dataRequestTracker` to throttle data requests - [OperationTracker<Trace>](displaypluginlibrary.md#tracking-operations) field named `redrawTraceRequestTracker` to throttle trace redraw requests - [OperationTracker<Cursor>](displaypluginlibrary.md#tracking-operations) field named `redrawCursorRequestTracker` to throttle cursor redraw requests - `int` field named `dataRequestSampleCount` as backer for the `DataRequestSampleCount` property ```c# private readonly Pen cursorPen = new Pen(Brushes.White, 1); private readonly OperationTracker<(DataRequestSignal Signal, IDisplayParameterContainer ParameterContainer)> dataRequestTracker; private readonly OperationTracker redrawTraceRequestTracker; private readonly OperationTracker redrawCursorRequestTracker; private int dataRequestSampleCount; ``` Instantiate the operation trackers in the _View Model_ constructor ```c# this.dataRequestTracker = new OperationTracker<(DataRequestSignal Signal, IDisplayParameterContainer)>( ThrottleInterval, operation => signalBus.Send(operation.Signal)); this.redrawTraceRequestTracker = new OperationTracker( ThrottleInterval, async trace => await this.ExecuteOnUiAsync(() => this.Redraw(trace))); this.redrawCursorRequestTracker = new OperationTracker( ThrottleInterval, async cursor => await this.ExecuteOnUiAsync(() => this.Redraw(cursor))); ``` !!! note Trace and Cursor graphics are drawn on the UI thread by calling via the `ExecuteOnUiAsync()` method Add the following properties - `int` property named `DataRequestSampleCount` that determines the number of data points to fetch and draw (defaults to 1000) - `IVisual` property named `CursorVisual` used to draw the cursor (set in the _View_ code behind) - `IVisual` property named `TraceVisual` used to draw the trace (set in the _View_ code behind) ```c# public int DataRequestSampleCount { get => this.dataRequestSampleCount = this.ReadProperty(1000); set { if (this.SetProperty(ref this.dataRequestSampleCount, value)) { this.SaveProperty(value); this.MakeDataRequests(false, true); } } } [Browsable(false)] public IVisual CursorVisual { get; set; } [Browsable(false)] public IVisual TraceVisual { get; set; } ``` Extend `OnMakeCursorDataRequestsAsync()` method to instantiate a `Cursor` instance and add to `redrawCursorRequestTracker` to initiate a redraw ```c# var cursor = new Cursor(compositeSession.TimebaseRange, compositeSession.CursorPoint); this.redrawCursorRequestTracker.Add(cursor); ``` Subscribe to `DataResultSignal` in the _View Model_ constructor to handle the result of a _data request_ ```c# this.Disposables.Add( signalBus.Subscribe( this.HandleDataResultSignal, r => r.SourceId == this.ScopeIdentity.Guid)); ``` Override the `OnMakeTimebaseDataRequestsAsync()` method to issue a _data request_ for the second parameter when the timebase timestamp range changes ```c# protected override Task OnMakeTimebaseDataRequestsAsync(ICompositeSession compositeSession) { if (this.DisplayParameterService.PrimaryParameters.Count < 2) { return Task.CompletedTask; } // Second parameter is the trace line var signal = this.DataRequestSignalFactory.CreateDataRequestSignal( this.ScopeIdentity.Guid, this.DisplayParameterService.PrimaryParameters.Skip(1).FirstOrDefault(), compositeSession.TimebaseRange, this.DataRequestSampleCount, SampleMode.MaximumToMinimum); this.dataRequestTracker.Add((signal, this.DisplayParameterService.ParameterContainers.Skip(1).FirstOrDefault())); return Task.CompletedTask; } ``` !!! note The `Trace` class needs a `SampleMode` of `MaximumToMinimum` in order to render correctly. !!! note Rather than send the signal directly to the `SignalBus`, add it to `dataRequestTracker` Add the `HandleDataResultSignal()` method to instantiate a `Trace` instance and add to `redrawTraceRequestTracker` to initiate a redraw ```c# private void HandleDataResultSignal(DataResultSignal signal) { var request = signal.Data.Request; var result = signal.Data; var parameterValues = result.ParameterValues; parameterValues.Lock(); try { if (parameterValues.SampleCount > 0 && this.dataRequestTracker.GetCurrent(out var currentOperation)) { var trace = new Trace(currentOperation.ParameterContainer, request.Parameter, parameterValues); this.redrawTraceRequestTracker.Add(trace); } } finally { parameterValues.Unlock(); this.dataRequestTracker.Complete(); } } ``` Add the `Redraw` methods for `Trace` and `Cursor` ```c# private void Redraw(Trace trace) { try { var extents = this.TraceVisual.Extents; if (!this.IsDisplayVisible || extents.Width == 0 || extents.Height == 0) { this.TraceVisual.Draw(delegate { }); return; } this.TraceVisual.Draw( dc => { dc.DrawRectangle( Brushes.Transparent, new Pen(Brushes.White, 1), new Rect(new Point(0, 0), extents)); var tracePen = new Pen(new SolidColorBrush(trace.TraceColor), extents.Width / trace.PointCount); foreach (var (start, end) in trace.GetLines(extents)) { dc.DrawLine(tracePen, start, end); } }); } finally { redrawTraceRequestTracker.Complete(); } } private void Redraw(Cursor cursor) { try { var extents = this.CursorVisual.Extents; if (!this.IsDisplayVisible || extents.Width == 0 || extents.Height == 0 || !cursor.GetCursorLine(extents, out var cursorLine)) { this.CursorVisual.Draw(delegate { }); return; } this.CursorVisual.Draw( dc => { dc.DrawLine(this.cursorPen, cursorLine.Item1, cursorLine.Item2); }); } finally { redrawCursorRequestTracker.Complete(); } } ``` !!! note `DrawingContext` is a retained mode graphics system, therefore to clear it, call `Draw()` but draw nothing ### Testing the display To view the trace and cursor graphics - Add a session via the _Session Browser_ to the compare set associated with the display - Add a display parameter to visualize (e.g. `vCar`) via the _Parameter Browser_ - Ensure there is already one display parameter set for the PCU8D LED visualization - Use a _Waveform_ display to change the cursor and timebase --- # Gauge display Many displays just need to visualize the current value (at the cursor) of one or more display parameters. As it is such a common use case, [ParameterSampleDisplayViewModelBase](displaypluginlibrary.md#parametersampledisplayviewmodelbase-view-model-base-class) has been provided to make creating such a display quick and simple. !!! note The code for this tutorial can be reviewed at [Tutorials/SimpleGaugePlugin](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/SimpleGaugePlugin) Start the tutorial by [creating a new display from scratch](createfromscratch.md) named `SimpleGaugePlugin`. ## Add a reference to `CircularGauge` Add a NuGet reference to [`CircularGauge`](https://www.nuget.org/packages/CircularGauge/) ## Update the _View_ class Configure the user interface as follows - Wrap an `ItemsControl` to display a collection of display parameter values within a `ScrollViewer` with `HorizontalScrollBarVisibility=Auto` and `VerticalScrollBarVisibility=Auto` - Bind `ItemsSource` attribute to the _View Model_ `Parameters` property - Set the `ItemsPanel` element to a `UniformGrid` - Set the `ItemTemplate` element to a `DataTemplate` containing a `CircularGaugeControl` - Bind the `MinValue` attribute to the _Parameter View Model_ `DisplayMinimum` property - Bind the `MaxValue` attribute to the _Parameter View Model_ `DisplayMaximum` property - Bind the `CurrentValue` attribute to the _Parameter View Model_ `Value` property - Bind the `DialText` attribute to the _Parameter View Model_ `Description` property ```xml ``` ## Add _Parameter View Model_ class Add a simple _View Model_ class to represent a display parameter value (derived from `ParameterSampleViewModelBase`). Add the following backers, properties and method overrides - `string` field named `description` as backer for the `Description` property - `double` field named `displayMaximum` as backer for the `DisplayMaximum` property - `double` field named `displayMinimum` as backer for the `DisplayMinimum` property - `string` property named `Description` to display within the gauge - `double` property named `DisplayMaximum` to define the scale of the gauge - `double` property named `DisplayMinimum` to define the scale of the gauge - override `OnUpdate()` to set the `DisplayMinimum` and `DisplayMaximum` properties - override `OnValueChanged()` to update the `Description` property ```c# public sealed class ParameterViewModel : ParameterSampleViewModelBase { private string description; private double displayMaximum; private double displayMinimum; public string Description { get => this.description; set => this.SetProperty(ref description, value); } public double DisplayMaximum { get => this.displayMaximum; set => this.SetProperty(ref displayMaximum, value); } public double DisplayMinimum { get => this.displayMinimum; set => this.SetProperty(ref displayMinimum, value); } protected override void OnUpdate() { this.DisplayMinimum = this.DisplayParameter.SessionParameter.Minimum; this.DisplayMaximum = this.DisplayParameter.SessionParameter.Maximum; } protected override bool OnValueChanged(double? oldValue, double newValue) { this.OnUpdate(); if (newValue < this.DisplayMinimum || newValue > this.DisplayMaximum) { return false; } this.Description = $"{this.Name}\r{this.Value}"; return true; } } ``` ## Update the _View Model_ class Derive from [ParameterSampleDisplayViewModelBase](displaypluginlibrary.md#parametersampledisplayviewmodelbase-view-model-base-class) and allow parameters by specifying the [`DisplayPluginSettings`](../parameters.md) attribute ```c# [DisplayPluginSettings(ParametersMaxCount = 100)] public sealed class SampleDisplayViewModel : ParameterSampleDisplayViewModelBase ``` Inject the `ISignalBus`, `IDataRequestSignalFactory` and `ILogger` services into the _View Model_ constructor and pass to the base constructor ```c# public SampleDisplayViewModel( ISignalBus signalBus, IDataRequestSignalFactory dataRequestSignalFactory, ILogger logger) : base(signalBus, dataRequestSignalFactory, logger) ``` Override `OnCreateParameterViewModel` and return an instance of _Parameter View Model_ ```c# protected override ParameterViewModel OnCreateParameterViewModel() => new ParameterViewModel(); ``` ## Copy CircularGauge.dll to ATLAS program files Add a post build step via project settings ``` call "$(SolutionDir)scripts\deploy.bat" "$(TargetDir)CircularGauge.dll“ ``` ![Post Build Step]( assets/images/circulargaugecontrolpostbuildstep.png) ## Testing the display To view a gauge for each display parameter - Add a session via the _Session Browser_ to the compare set associated with the display - Add some display parameters to the display via the _Parameter Browser_ - Use a _Waveform_ display to change the cursor --- # Deployment Script To aid development efficiency, automatically copy the custom display plugin DLL to ATLAS program files after each successful build. - Create a deployment batch file - Add a _scripts_ folder under _Solution_ - Create _deploy.bat_ file with the following contents ``` NET SESSION >nul 2>&1 IF %ERRORLEVEL% EQU 0 ( IF EXIST "C:\Program Files\McLaren Applied Technologies\ATLAS 10\" ( xcopy /r /y %1 "C:\Program Files\McLaren Applied Technologies\ATLAS 10" ) ELSE ( echo "No Atlas installation detected - skip deploying" ) ) ELSE ( echo "This script requires admin privileges - skip deploying" exit /b 0 ) ``` - Add _deploy.bat_ to _scripts_ folder ![Scripts Folder]( assets/images/scriptsfolder.png) - Add a post build step via project settings ``` call "$(SolutionDir)scripts\deploy.bat" "$(TargetDir)$(ProjectName).dll“ ``` ![Post Build Step]( assets/images/postbuildstep.png) !!! attention Special permissions are needed to write to program files, therefore you will need to run Visual Studio as Administrator ![Run Visual Studio]( assets/images/runvisualstudio.png) --- # Display Plugin Library The display plugin library provides base and utility classes that reduce the overall effort needed to create a custom display. !!! note The code for this library can be reviewed at [Tutorials/DisplayPluginLibrary](https://github.com/mat-docs/Atlas.DisplayAPI.Examples/tree/master/DisplayPluginLibrary) ## `TemplateDisplayViewModelBase` _View Model_ base class `TemplateDisplayViewModelBase` (a substitute for `DisplayPluginViewModel`) provides a template to help follow the [Data request guidelines](../data.md#data-request-guidelines) properly - Data requests are throttled to 5Hz by default - Pass a different throttle interval to the constructor if the default is not suitable - Properties are provided for the following common services - `IDataRequestSignalFactory` `DataRequestSignalFactory` - `IDisplayParameterService` `DisplayParameterService` - `ILogger` `Logger` - `ISignalBus` `SignalBus` - Append additional disposables to the `Disposables` property for automatic cleanup - Override `OnMakeTimebaseDataRequestsAsync()` to issue data requests in response to changes to the timebase - Override `OnMakeCursorDataRequestsAsync()` to issue data requests in response to changes to the cursor - Call `MakeDataRequests()` to initiate data requests for any other reason - Call the `ExecuteOnUiAsync()` method to execute code on the UI thread - Observe `IsDisplayVisible` property to determine when appropriate to display UI (by convention display nothing when no set is associated with the display) !!! attention When overriding `DisplayPluginViewModel` notification methods, do not forget to call the base class implementation (i.e. those of `TemplateDisplayViewModelBase`) ## `ParameterSampleDisplayViewModelBase` _View Model_ base class `ParameterSampleDisplayViewModelBase` (a substitute for `TemplateDisplayViewModelBase`) provides automatic retrieval of display parameter sample values - `TParameterViewModel` must be a class derived from `ParameterSampleViewModelBase` - `ParameterSampleViewModelBase` base class wraps a display parameter - Properties of the display parameter such as `Name` and current `Value` - `OnUpdate()` and `OnValueChanged()` notification methods to override - An implementation of the `OnCreateParameterViewModel()` factory method must be provided to supply a `TParameterViewModel` instance for a display parameter on demand - When display parameters are added/removed from the display, the `Parameters` collection is updated - `OnCreateParameterViewModel()` is called for newly added display parameters - `OnUpdateParameters()` is called whenever the `Parameters` collection is modified, and `OnUpdate()` is called on each `TParameterViewModel` instance - When the cursor timestamp changes, a data request is issued for each member of `Parameters` and the `Value` property of each is updated and `OnValueChanged()` called ## Tracking Operations The `OperationTracker` class provides support for flow control and throttling of operations such as data or redraw requests - Pass to the _Constructor_ a 'throttle interval' and 'action' to execute an operation - The 'throttle interval' specifies the minimum time (as a `TimeSpan`) between operations being executed - The 'action' is passed an instance of `TOperation` and when finished must call `Complete()` - Up to two operations can be tracked at any one time - The current (or active) operation (will be the first one added) - Operations added when there is already one active will become pending - A pending operation will always be the last one added (replacing any previous pending operation) - The current and pending operation will not overlap - Call `Add()` to add an operation of type `TOperation` - Call `Complete()` to indicate that the current operation has finished executing - If there is a pending operation it will automatically become active - Call `GetCurrent()` to get a copy of the current operation (if there is one, the method returns true) - Call `Abort()` to clear any active/pending operations ## Drawing Graphics WPF provides a number of ways to draw [graphics](https://docs.microsoft.com/en-us/dotnet/desktop/wpf/graphics-multimedia). The `VisualLayer` class provides a simple way to add graphics drawn using the low-level and efficient [DrawingContext](https://docs.microsoft.com/en-us/dotnet/api/system.windows.media.drawingcontext) class. - Add an `IVisual` instance to the _View Model_ class ```C# [Browsable(false)] public IVisual ExampleVisual { get; set; } ``` - Add a `VisualLayer` to the XAML ```xml hl_lines="10" ``` - In the code behind, associate the `ExampleVisualLayer` _View_ property to the `ExampleVisual` _View Model_ property ```c# hl_lines="7" public override void OnApplyTemplate() { base.OnApplyTemplate(); if (this.DataContext is SampleDisplayViewModel vm) { vm.ExampleVisual = this.ExampleVisualLayer.Visual; } } ``` - Draw graphics as required in the _View Model_ ```c# hl_lines="4" this.ExampleVisual.Draw( dc => { dc.DrawLine(examplePen, exampleLine.Start, exampleLine.End); }); ``` !!! attention Drawing must be done on the UI thread. Make use of [`SynchronizationContext` class](https://docs.microsoft.com/en-us/dotnet/api/system.threading.synchronizationcontext) or call the `TemplateDisplayViewModelBase.ExecuteOnUiAsync()` method. !!! tip To clear the visual layer of all graphics ```c# this.TraceVisual.Draw(delegate { }); ``` ## Converters `ColorToSolidColorBrushValueConverter` is a basic WPF converter that converts a simple colour into a solid colour brush. --- # Factories and Services The following are the factories and services provided by the Display API - `ISignalBus` - Defines a signal bus for application-wide communication - `IDataRequestSignalFactory` - Used for creating a data request signal for posting onto the signal bus - `ISessionService` - Enumerate parameters within a session - `ISessionSummaryService` - Obtain summary information for a session - `IDisplayParameterService` - Service to manipulate display parameters - `ISessionCursorService` - Programmatic setting of the cursor timestamp for the active session !!! note These factories and services (except `IDisplayParameterService`) are registered with _Autofac_ (_IoC container_) To obtain an instance, inject as a parameter into the _View Model_ constructor `IDisplayParameterService` is a display specific service (accessible via `DisplayPluginViewModel.ServiceContext` property) The following sections provide further information regarding usage of each. --- # Initialization, state and notifications The following methods and properties of _DisplayPluginViewModel_ provide display initialization, state and notifications - Display initialization - `OnInitialised()` - Called when display is first initialized (programmatic parameters should ideally be added within this override) - Display state properties - `ActiveCompositeSessionContainer` - Access to compare set associated with display (can be `null` if no association) - `ServiceContext` - Access to per display services - `IsSelected` - True when page is active and display is shown, false when display is overlaid by another display - `CanRetrieveData` - False when display is inactive or no session is associated and therefore display should not retrieve data - `ScopeIdentity` - Unique identity of display (mainly used to associate results of a data request to a display) - Display change notifications - `OnActiveDisplayPageChanged(bool isActive)` - Called whenever page is switched (`isActive` is true when the active page contains the display) - `OnCanRenderDisplayChanged(bool canRender)` - Called whenever display visibility changes (`canRender` is true when display is visible) !!! important `CanRetrieveData` should be used exclusively for deciding whether it is necessary to make data requests. !!! attention Do NOT access `ActiveCompositeSessionContainer` when `CanRetrieveData` is false, as an unexpected exception may be raised --- # Working with Display Properties Display properties can be edited within the properties window and/or persisted to the workbook. ## Adding a display property to the _View Model_ Add a private backer ```c# private Type propertyName; ``` !!! note For properties persisted to the workbook, `Type` must be serializable to JSON. Add a public Property (with getter and setter) to update property backer ```c# public Type PropertyName { get => this.propertyName; set => this.SetProperty(ref this.propertyName, value); } ``` !!! note If there are additional side effects when a property value has changed, e.g. to indicate that calculated properties may need updating, modify setter accordingly ```c# set { if (this.SetProperty(ref this.propertyName, value)) { this.OnPropertyChanged(nameof(this.OtherPropertyName); } } ``` Use `OnPropertyChanged` to inform WPF that a property may have changed (make sure to use `nameof()` as a safe way to specify a property name). ## Expose _View Model_ properties so they can be edited within the properties window Public properties will by default appear within the properties window. !!! attention It is important to always review the properties window after significant changes to the _View Model_ to ensure only required properties are visible. Decorate properties with `[Browsable(false)]` to exclude unwanted properties from the properties Window, see [below](#optional-display-property-attributes). ## Persist display properties to and from the workbook Update the getter to read the property value from the Workbook (passing an appropriate default for new workbooks) ```c# get => this.propertyName = this.ReadProperty(defaultValue); ``` Update the setter to save the property to the workbook when the property value has changed ```c# set { if (this.SetProperty(ref this.propertyName, value)) { this.SaveProperty(value); } } ``` ## Optional Display Property Attributes Decorate display properties with _Attributes_ to configure how they are displayed within the properties window - `[Category(title)]` - To specify a category to group similar properties together - `[DisplayName(name)]` - To specify the display name of the property (otherwise the C# property name is used) - `[Description(description)]` - To specific the property description that appears in the section at the bottom of the properties window - `[Display(Order=n)]` - To specify the order of properties (lowest value are first) - `[Browsable(state)]` - To specify whether a public property should be included within the properties window (`true` – implicit default) or excluded (`false` – set explicitly) from the properties window !!! note Add a reference to `System.ComponentModel.DataAnnotations` as required --- # Working with Sessions A _Session_ is a collection of data samples obtained from a data source. Multiple _Sessions_ may be loaded at one time and grouped into _Sets_. Custom displays should respond appropriately when _Sessions_ are loaded/unloaded. ## Sessions Object Model Each _Set_ is represented by a _composite session container_ ![Set]( devguide/assets/images/sets.png) A _Set_ is associated with a _display_ ![Associated Set]( devguide/assets/images/associateset.png) A _composite session container_ contains zero or more _composite sessions_. When there is more than one _composite session_ that constitutes a _compare set_ (_overlay_) ![Overlay Set]( devguide/assets/images/overlayset.png) A _composite session_ contains one or more _sessions_. When there is more than one _session_ they can either be appended or overlaid ![Append Set]( devguide/assets/images/appendset.png) !!! note Associate sessions are not part of the this model. ### Session properties, notification overrides and services - `DisplayPluginViewModel` properties - `ActiveCompositeSessionContainer` property - Access to the _Set_ associated with a _display_ (can be `null` if no association) - `DisplayPluginViewModel` notifications - The following overrides are called for sessions - `OnCompositeSessionLoaded()` - Called when a _composite session_ is loaded - `OnCompositeSessionUnLoaded()` - Called when a _composite session_ is unloaded - `OnCompositeSessionContainerChanged()` - Called when a _Set_ associated with a _display_ is changed (i.e. `ActiveCompositeSessionContainer` property has changed) - Session Services - `ISessionService` - `GetParameter()` - Get details about a specific parameter - `GetParametersCount()` - Get the count of all parameters within a _composite session_ - `GetParameters()` - Get details of all parameters within a _composite session_ - `ISessionSummaryService` - `GetSessionSummary()` - Get session summary for _session_ --- # Working with Timebase and Cursor The _timebase_ is the currently visible timestamp range that determines the extent of parameter values displayed by various standard displays such as _Waveform_, _Scatterplot_, _Histogram_ etc. The _cursor_ is the currently selected timestamp and determines the parameter values displayed by various standard displays such as _Numeric_, _Bit_, _Circuit_ etc. Most standard displays are based on either _timebase_ or _cursor_. A notable exception is the _Summary_ display that is based on laps. The _Waveform_ display uses both _timebase_ and _cursor_ and responds as they change. Interacting with the _Waveform_ display changes the value of _timebase_ and/or _cursor_. !!! hint To maintain consistency, custom displays should match the behaviour of standard displays (unless doing something different like _Summary Display_). ## Timebase and Cursor timestamps To access the primary session timebase range and/or cursor - Get the F1 timeline extent for the primary session from `ActiveCompositeSessionContainer.CompositeSessions.First(cs => cs.IsPrimary).TimebaseRange` - Get the cursor timestamp for the primary session from `ActiveCompositeSessionContainer.First(cs => cs.IsPrimary).CursorPoint` !!! tip Ensure you check `ActiveCompositeSessionContainer.IsPrimaryCompositeSessionAvailable` is `true` before looking for the primary session. !!! attention Do not confuse `TimeRange` (extent of session) and `TimebaseRange` (extent of F1 timeline selection) properties. !!! note A timestamp is the number of nanoseconds since midnight. A timestamp can be converted to a `TimeSpan` via ```c# var timespan = TimeSpan.FromTicks(timestamp / 100); ``` The following `DisplayPluginViewModel` overrides are called whenever the timebase or cursor changes for a _composite session_ - `OnCursorDataPointChanged(ICompositeSession)` - New timestamp value is obtained from `ICompositeSession.CursorPoint` - `OnSessionTimeRangeChanged(ICompositeSession)` - New timestamp range is obtained from `ICompositeSession.TimebaseRange` ## Cursor Service Programmatic setting of the the cursor for the active session - `ISessionCursorService` - Use `MoveCursor()` to change the timestamp of the cursor --- # Working with Display Parameters By default, display parameters cannot be added to the display. - Decorate the _View Model_ class with the `[DisplayPluginSettings(ParametersMaxCount=n)]` attribute to set an appropriate maximum number of display parameters allowed - 100 is the recommended maximum Display parameters can be configured by - The _user_ with the _Parameter Browser_ - Programmatically via `IDisplayParameterService` Each display parameter consists of a _parameter container_ with a _parameter_ per composite session. Override the following `DisplayPluginViewModel` notifications as appropriate - `OnParameterContainerAdded()` - Called when parameter is first added. - `OnParameterContainerRemoved()` - Called when parameter is removed - `OnParameterAdded()` - Called for each _composite session_ when a parameter is added - `OnParameterRemoved()` - Called for each _composite session_ when a parameter is removed !!! note `OnParameterContainerAdded` will be called, followed by `OnParameterAdded` for each _composite session_ `OnParameterRemoved` is called for each _composite session_, followed by `OnParameterContainerRemoved` !!! note Use `OnParameterAdded()` and `OnParameterRemoved()` for compare set aware custom displays ## Display Parameter Service The `IDisplayParameterService` is used to access and manipulate display parameters - `ParameterContainers` collection property - One per display parameter - `Parameters` collection property - One per _composite session_ - `PrimaryParameters` collection property - One per display parameter - Parameters from the _primary composite session_ only - `AddParameterContainer()` - Programmatically add a display parameter (usually done in `OnInitialised()`) - `RemoveParameterContainer()` - Programmatically remove a display parameter --- # Working with Data ## Making Data Requests The following services are required to retrieve parameter data - `ISignalBus` - `IDataRequestSignalFactory` !!! hint These services are obtained from _Autofac_ by injecting into the _View Model_ constructor as parameters, e.g. ```c# public ViewModelConstructor(ISignalBus signalBus, IDataRequestSignalFactory dataRequestSignalFactory) { // Use now, or store for use later... } ``` A _data request_ is made via a data request signal (obtained from `IDataRequestSignalFactory`). In order to match _results_ to _requests_ made by the display, a `sourceId` needs to be specified; an appropriate value is `this.ScopeIdentity.Guid` as this uniquely identifies the display. !!! hint An optional `requestId` can be specified to distinguish between multiple requests of the same data request signal type. The _signal bus_ is used to initiate the _data request_ by calling `ISignalBus.Send()`. The result is asynchronously sent back via the _signal bus_ and therefore an appropriate handler needs to be registered in the _View Model_ constructor using `ISignal.Subscribe`passing a suitable handler. !!! note The _result signal type_ much match the _request signal type_, [see list here](#data-request-and-result-signals) !!! hint It is highly recommended to specify a filter checking that `resultSignal.SourceId` matches `this.ScopeIdentity.Guid`, avoiding the need to check in the handler. !!! warning Remember to match the `SourceId` property of result signal, not `SignalId` property. !!! attention The signal handler is ran by default on the Task pool. If modifying the UI, dispatch to the UI thread as appropriate via `SynchronizationContext`. ### Data Request and Results Reply Sequence ## Data Request Properties - _Data_ vs _Samples_ requests - Requests for _Samples_ retrieve the actual stored values - Requests for _Data_ retrieve stored values re-sampled at a specified interval - _Single_ vs _Composite_ session requests - _Single_ session requests return results for a single _composite session_ - _Composite_ session requests return results for all _composite sessions_ within a _composite session container_ (primary and secondary) - Request properties - _Source Id_/_Request Id_ - Used to match a _result_ to a _request_ as explained above. - _Composite Session key_ (or _Container Session Container key_ for _composite_ requests) - Usually obtained from `ActiveCompositeSessionContainer` or from arguments passed to notification methods - _Parameter_ (or _Parameter Container_ for _composite_ requests) - _Extent_ of data request - Requests for _Samples_ - _Start Time_ and _Sample Count_, or - _Time Range_ - Requests for _Data_ - _Time Range_ and _Sample Count_ (_interval_ is calculated) - _Sample Direction_ - Retrieve forwards (_Next_) or backwards (_Previous_) through the stored values - _Sample Mode_ (only applicable for requests for _Data_) - How the stored values are re-sampled (_First Value_, _Minimum_, _Maximum_, _Mean_ etc) - _Sample Count_ (see _Extent_ above) - Maximum number of samples to fetch (requests for _Samples_) - Used to calculate sample interval (requests for _Data_) - _Lap_, _Statistics Level_, _Statistics Option_ - What statistics to calculate and over what time range ## Data request and result signals Handle the matching result signal type for each request signal type used - `DataRequestSignal` - Re-sampled values for a single _composite session_ - Handle `DataResultSignal` to process result - `SampleRequestSignal` - Actual values for a single _composite session_ - Handle `SampleResultSignal` to process result - `CompositeDataRequestSignal` - Re-sampled values for _composite session container_ (_compare set_) - Handle `CompositeDataResultSignal` to process result - `CompositeSampleRequestSignal` - Actual values for _composite session container_ (_compare set_) - Handle `CompositeSampleResultSignal` to process result - `DataStatisticsRequestSignal` - Statistic values - Handle `DataStatisticsResultSignal` to process result ## Data Results Results are accessed via the matching results signal - `IResult` - `ParameterValues` property, [see below](#parametervalues-properties) - Result of data request obtained from SQL Race - Remember to call `Lock()`/`Unlock()` if retaining a reference - `IRequest` property (original request) - `ICompositeResult` - `Results` property (`IDictionary`) - `IResult` for each _composite session_ - `ICompositeRequest` property (original request) - `IStatisticsResult` - No results property (bug, see below) - `IRequest` property (original request) !!! attention Currently data statistics requests are broken because the statistics result does not actually expose the retrieved data ### ParameterValues properties Values retrieved by SQL Race are passed back in a `ParameterValues` instance - `SampleCount` - Actual number of values returned from data request - `TimeStamp` - The timestamp of each value - `DataStatus` - The status of each value (e.g. _Missing_ if no meaningful value) - `Data` - Each value (for _sample_ requests and for _mean data_ requests) - `DataFirst` - Each first value (for first data requests) - `DataMin` - Each minimum value (for minimum data requests) - `DataMax` - Each maximum value (for maximum data requests) !!! attention Due to the use of array pooling, the length of the arrays may be greater then the retrieved data quantity Always refer to _SampleCount_ and not array length ## Data request guidelines - Only attempt a data request when `CanRetrieveData` is `true` - Handle the following situations and make data requests to ensure the custom display is up to date (observing `CanRetrieveData` as per above) - Page switch and display visibility changes (stacked displays) - Override `OnCanRenderDisplayChanged()` - Sessions loaded/unloaded - Override `OnCompositeSessionLoaded()` and `OnCompositeSessionUnLoaded()` - Session association changes - Override `OnCompositeSessionContainerChanged()` - Copy/Paste displays - Override `OnInitialised()` - Parameters added/removed - Override `OnParameterAdded()` and `OnParameterRemoved()` --- # Good Citizenship A custom display should be written to look, behave and act just like the standard displays shipped with ATLAS. !!! attention As custom displays are executed directly within ATLAS, an improperly written custom display has the potential to affect ATLAS reliability and performance In order to be a good citizen within the ATLAS ecosystem, a custom display should - Follow [MVVM](mvvm.md) - Not use unsupported private APIs whenever possible - Not do processing on the UI thread - Use _TaskPool_ for processing and dispatch UI updates via `SynchronizationContext` instead - Throttle data requests - Do not overlap requests - Do not just respond to raised events (choose an appropriate refresh rate) - Obey `CanRetrieveData` - Handle appropriate events to ensure display is kept up to date - As explained in [Data request guidelines](data.md#data-request-guidelines) !!! tip The [Display Plugin Library](devguide/displaypluginlibrary.md) demonstrates how to make a custom display a good citizen Using the provided _View Model_ base classes in your own custom displays will help to ensure you follow these guidelines --- # Core API The display API shares some core functionality with ATLAS internal APIs - `SignalBus` - Diagnostics - `ILogger` - MVVM helpers - `BindableBase` - `DisposableBindableBase` - `DelegateCommand` - IDisposable helpers - `DisposableBase` - Colours - `AtlasColor` - `AtlasColors` --- # API Limitations The Display API currently has the following limitations - Missing events - Not notified when composite session becomes primary - Not notified when compare mode settings change - Not notified when parameter order changes - Not notified when parameter properties change - Missing UI helpers - No access to custom converters (e.g. AtlasColorToSolidBrush) - No access to custom property editors (e.g. AtlasColor property editor) - No access to common context menu (e.g. Parameter Browser, Properties) - No support for drag and drop of parameters - No assistance with flow control - Event driven only - Update loop not supported - No inflight data request handling (nor throttling) !!! tip Some of these limitations (especially around flow control) are overcome within the _View Model_ base classes provided by the [Display Plugin Library](devguide/displaypluginlibrary.md) --- # Model-View-ViewModel (MVVM) [MVVM](https://docs.microsoft.com/en-us/archive/msdn-magazine/2009/february/patterns-wpf-apps-with-the-model-view-viewmodel-design-pattern) is a popular software architecture pattern commonly used with WPF applications ![MVVM](devguide/assets/images/mvvm.png) - _Model-View-ViewModel_ pattern - _2 way binding_ - The _View_ `DataContext` property should be assigned an implementation of the _View Model_ - `“{Binding …}”` _XAML_ element links a _View_ property to a _View Model_ property - _Change notification_ - _View Model_ should implement the `INotifyPropertyChanged` interface - `PropertyChanged` event should be raised whenever a property value changes - _Commands_ - _View Model_ must provide an implementation of the `ICommand` interface per _command_ - Helper classes provided by the _Display API_ to implement the above - `BindableBase` base class - Implements `INotifyPropertyChanged` - Provides `SetProperty` methods to simplify following the pattern - `DisposableBindableBase` base class - As per `BindableBase` but also supports `IDisposable` - `DelegateCommand` class - Provides a generic implementation of `ICommand` for _commands_ --- # Display API prerequisites ## Visual Studio - [Visual Studio 2019](https://visualstudio.microsoft.com/downloads/?utm_medium=microsoft&utm_source=docs.microsoft.com&utm_campaign=inline+link&utm_content=download+vs2019) with the **.NET desktop development** workload installed ![Visual Studio Installer](devguide/assets/images/visualstudioinstaller.png) ## Display API NuGet package - Access to _Atlas.DisplayAPI_ NuGet package, see [https://github.com/mat-docs/packages](https://github.com/mat-docs/packages) ## ATLAS - A recent version ATLAS 10 should be installed for testing custom displays, see [Motion Applied Software Portal](https://www.mclarenelectronics.com/Portal) ## Required Knowledge - C# - ATLAS has been implemented in the [C# programming language](https://docs.microsoft.com/en-us/dotnet/csharp/tour-of-csharp/) and therefore C# is the preferred implementation language for custom displays - WPF - [Windows Presentation Foundation](https://docs.microsoft.com/en-us/visualstudio/designers/getting-started-with-wpf) is the User Interface technology used by ATLAS and therefore custom displays should also be implemented in WPF - MVVM - [MVVM](mvvm.md) has been used extensively in ATLAS and therefore the Display API has been designed to be fully MVVM compliant --- # .NET 6 Upgrade !!! warning From ATLAS (`11.4.4.349-W47`) and SQL RACE (`2.1.25308.1-ci`) please see [.NET 8 Upgrade](./net8-upgrade.md) ATLAS client is now updated to .NET 6. Users will need to consider upgrading any custom plugins or scripts they might have written. Provided the custom ATLAS plugin does not use any .NET Framework libraries that are no longer supported in .NET 6, they should still work with this release of ATLAS, but this is not something we can guarantee. You may find it necessary to recompile against .NET 6. ## C\# You may wish to determine any initial portability concerns using the Platform Compatibility Analyzer and .NET Portability Analyzer. For C# there is also the .NET Upgrade Assistant to upgrade projects, this will: * Upgrade to new project format * Replace old style packages.config with project file-based package references * Update dependencies as required to support at least .NET Standard There might be instances when certain dependencies may have to be updated/replaced, e.g., if it is no longer supported in .NET 6. Useful sites: ### Porting Guides [https://docs.microsoft.com/en-us/dotnet/core/porting/](https://docs.microsoft.com/en-us/dotnet/core/porting/) [https://docs.microsoft.com/en-gb/dotnet/core/porting/net-framework-tech-unavailable](https://docs.microsoft.com/en-gb/dotnet/core/porting/net-framework-tech-unavailable) [https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting](https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting) ### .NET Upgrade Assistants [https://docs.microsoft.com/en-us/dotnet/core/porting/upgrade-assistant-overview](https://docs.microsoft.com/en-us/dotnet/core/porting/upgrade-assistant-overview) ### .NET Portability Analyzer [https://docs.microsoft.com/en-us/dotnet/standard/analyzers/portability-analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/portability-analyzer) ### Platform Compatibility Analyzer [https://docs.microsoft.com/en-us/dotnet/standard/analyzers/platform-compat-analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/platform-compat-analyzer) ## MATLAB ## Targeting a Specific .NET Version ### MATLAB R2025a or newer For MATLAB R2025a or newer, you can target a specific .NET Core version and specify required frameworks directly when initializing the environment. This is especially important if your code or dependencies require assemblies such as Microsoft.Extensions.Caching.Memory, which is included in Microsoft.AspNetCore.App. Example: ```matlab ne = dotnetenv("core", Version="6", Frameworks="Microsoft.AspNetCore.App"); ``` You can also define multiple frameworks if needed: ```matlab ne = dotnetenv("core", Version="6", Frameworks=["Microsoft.AspNetCore.App", "Microsoft.NETCore.App"]); ``` ### MATLAB R2024b and older MATLAB versions earlier than 2022b do not support the .NET core runtime. The default .NET runtime for MATLAB versions 2022b can be configured to .NET Core with the following command ```matlab dotnetenv('core') ``` !!! warning "MATLAB R2024b with .NET 6" MATLAB R2024b has a known issue with .NET6. To patch: From a .NET 6 installation, copy ijwhost.dll from `./packs/Microsoft.NETCore.App.Host.win-x64/6.0.XX/runtimes/win-x64/native/ijwhost.dll` into /bin/win64 (replace existing). The runtime configuration file for the .NET runtime can be found in `matlabroot/bin/win64/dotnetcli_netcore.runtimeconfig.json`. With this file, MATLAB loads the latest appropriate assemblies compatible with SQLRace API. An example configuration for the runtime is provided below. ``` { "runtimeOptions": { "rollForward": "Minor", "tfm": "net6.0", "frameworks": [ { "name": "Microsoft.NETCore.App", "version": "6.0.0" }, { "name": "Microsoft.WindowsDesktop.App", "version": "6.0.0" }, { "name": "Microsoft.AspNetCore.App", "version": "6.0.0" } ], "configProperties": { "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization": true } } } ``` More information about calling .NET from MATLAB can be found in the [MATLAB documentation](https://mathworks.com/help/matlab/call-net-from-matlab.html). ## Python The .NET core runtime can be loaded via calling `pythonnet.load` or configuring the environment variable. The runtime must be configured __before `clr` is imported__, otherwise the default runtime will be initialized and used. Information on the runtime in use can be retrieved using `pythonnet.get_runtime_info()`). Calling `pythonnet.load` ```python from pythonnet import load load("coreclr", runtime_config=r"C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.Atlas.Host.runtimeconfig.json") ``` Via Environment Variables ``` PYTHONNET_RUNTIME=coreclr PYTHONNET_CORECLR_RUNTIME_CONFIG=C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.Atlas.Host.runtimeconfig.json ``` More information about calling .NET from Python can be found in the [pythonnet documentation](https://pythonnet.github.io/pythonnet/python.html#loading-a-runtime). --- # .NET 8 Upgrade The ATLAS client (`11.4.4.349-W47`) and SQL RACE (`2.1.25308.1-ci`) is now updated to .NET 8. Users will need to consider upgrading any custom plugins or scripts they might have written to this framework. Provided the custom ATLAS plugin does not use any .NET libraries that are no longer supported in .NET 8, they should still work with this release of ATLAS, but this is not something we can guarantee. You may find it necessary to recompile against .NET 8. ## C\# You may wish to determine any initial portability concerns using the [Platform Compatibility Analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/platform-compat-analyzer) and [.NET Portability Analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/portability-analyzer). For C# there is also the [.NET Upgrade Assistant](https://docs.microsoft.com/en-us/dotnet/core/porting/upgrade-assistant-overview) to upgrade projects, this will: * Upgrade to new project format * Replace old style packages.config with project file-based package references * Update dependencies as required to support at least .NET Standard There might be instances when certain dependencies may have to be updated/replaced, e.g. if it is no longer supported in .NET 8. Useful sites: ### Porting Guides [https://docs.microsoft.com/en-us/dotnet/core/porting/](https://docs.microsoft.com/en-us/dotnet/core/porting/) [https://docs.microsoft.com/en-gb/dotnet/core/porting/net-framework-tech-unavailable](https://docs.microsoft.com/en-gb/dotnet/core/porting/net-framework-tech-unavailable) [https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting](https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/cross-platform-targeting) [https://learn.microsoft.com/en-us/dotnet/desktop/wpf/migration/](https://learn.microsoft.com/en-us/dotnet/desktop/wpf/migration/) ### .NET Upgrade Assistant [https://docs.microsoft.com/en-us/dotnet/core/porting/upgrade-assistant-overview](https://docs.microsoft.com/en-us/dotnet/core/porting/upgrade-assistant-overview) ### .NET Portability Analyzer [https://docs.microsoft.com/en-us/dotnet/standard/analyzers/portability-analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/portability-analyzer) ### Platform Compatibility Analyzer [https://docs.microsoft.com/en-us/dotnet/standard/analyzers/platform-compat-analyzer](https://docs.microsoft.com/en-us/dotnet/standard/analyzers/platform-compat-analyzer) ## MATLAB For compatibility with .NET 8, you will need to run MATLAB R2023a or higher as this will support .NET 6 or higher. ### Targeting a Specific .NET Version #### MATLAB R2025a or newer For MATLAB R2025a or newer, you can target a specific .NET Core version and specify required frameworks directly when initializing the environment. This is especially important if your code or dependencies require assemblies such as `Microsoft.Extensions.Caching.Memory`, which is included in `Microsoft.AspNetCore.App`. Example: ```matlab ne = dotnetenv("core", Version="8", Frameworks="Microsoft.AspNetCore.App"); ``` You can also define multiple frameworks if needed: ```matlab ne = dotnetenv("core", Version="8", Frameworks=["Microsoft.AspNetCore.App", "Microsoft.NETCore.App"]); ``` #### MATLAB R2024b and older MATLAB versions earlier than R2023a do not support the .NET 8 runtime. The default .NET runtime for MATLAB R2023a+ can be configured to .NET Core with the following command: ```matlab dotnetenv('core') ``` If you have newer .NET runtime environments (> .NET 8) then it is advised to state the version explicitly: ```matlab dotnetenv("core", Version="8"); ``` ### Known Issues and Patches !!! warning "MATLAB R2024b with .NET 8" MATLAB R2024b may have issues loading the .NET 8 runtime. If you experience problems, verify the `ijwhost.dll` in `/bin/win64` is compatible with .NET 8. From a .NET 8 installation, copy `ijwhost.dll` from `./packs/Microsoft.NETCore.App.Host.win-x64/8.0.XX/runtimes/win-x64/native/ijwhost.dll` into `/bin/win64` (replace existing). ### Runtime Configuration The runtime configuration file for the .NET runtime can be found in `matlabroot/bin/win64/dotnetcli_netcore.runtimeconfig.json`. With this file, MATLAB loads the latest appropriate assemblies compatible with SQLRace API. An example configuration for the runtime is provided below. ```json { "runtimeOptions": { "rollForward": "Minor", "tfm": "net8.0", "frameworks": [ { "name": "Microsoft.NETCore.App", "version": "8.0.0" }, { "name": "Microsoft.WindowsDesktop.App", "version": "8.0.0" }, { "name": "Microsoft.AspNetCore.App", "version": "8.0.0" } ], "configProperties": { "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization": true } } } ``` More information about calling .NET from MATLAB can be found in the [MATLAB documentation](https://mathworks.com/help/matlab/call-net-from-matlab.html). !!! hint Information on the runtime in use can be retrieved using: ```matlab e = dotnetenv; disp(e) ``` ## Python In order to run .NET in Python you need to use [pythonnet](https://github.com/pythonnet), currently this only supports python `3.7` - `3.13` with `3.14` and newer not [yet](https://github.com/pythonnet/pythonnet/issues/2610) supported. The runtime must be configured __before `clr` is imported__, otherwise the default runtime will be initialized and used. The .NET Core runtime can be loaded in code by calling `pythonnet.load` or configuring the environment variable. **In Code** ```python from pythonnet import load load("coreclr", runtime_config=r"C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.Atlas.Host.runtimeconfig.json") ``` **Environment Variable** ``` PYTHONNET_RUNTIME=coreclr PYTHONNET_CORECLR_RUNTIME_CONFIG=C:\Program Files\McLaren Applied Technologies\ATLAS 10\MAT.Atlas.Host.runtimeconfig.json ``` !!! warning `MAT.Atlas.Host.runtimeconfig.json` now also references the ASP.NET Core shared framework (`Microsoft.AspNetCore.App`), because the ATLAS host uses it. When this runtime configuration is loaded from Python, loading historic SQL Race sessions that contain virtual parameters can fail with an "unable to load PDA" exception. To work around this, make a copy of `MAT.Atlas.Host.runtimeconfig.json`, remove the `Microsoft.AspNetCore.App` entry from its `frameworks` section (leaving `Microsoft.NETCore.App`), and point `pythonnet` at the copy instead of the file in the ATLAS installation folder. ```json { "runtimeOptions": { "frameworks": [ { "name": "Microsoft.NETCore.App", "version": "8.0.0" } ] } } ``` More information about calling .NET from Python can be found in the [pythonnet documentation](https://pythonnet.github.io/pythonnet/python.html#loading-a-runtime). !!! hint Information on the runtime in use can be retrieved using `pythonnet.get_runtime_info()`. --- # Introduction !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Overview This section is a detailed guide to the [RTA API](../api/index.md), [Toolkit Services](../services/index.md) and their [gRPC](https://grpc.io/) APIs. !!! tip Have you read the [Introduction](../../index.md)? Start there to review the archictecture and key concepts. The [RTA API](../api/index.md) is the external-facing API consumed by ATLAS. This is designed to be an interface on the edge of your data environment — so you can exploit ATLAS capabilities without copying your data or making big changes to your data pipeline. The [Toolkit Services](../services/index.md) provide implementations of the API, in either a [single-process](../../../key-functionality/integrate/rta/introduction/architectures.md#rta-server) or [microservices](../../../key-functionality/integrate/rta/introduction/architectures.md#microservices) architecture. They expose [GPRC](https://grpc.io/) APIs to read, write and manage their data stores — but these are specific to these services and not part of the [RTA API Specification](../api/index.md). This means the [services](../services/index.md) in the implementation toolkit are entirely optional, and a seamless integration should be achievable in any convenient framework by implementing only the necessary parts of the [RTA API](../api/index.md). [Quick-Start](tutorials/quick-start/index.md){ .md-button .md-button--primary } --- # Overview !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Toolkit Services These services provide turn-key implementations for RTA functionality. Typical use cases: * Reduce the work needed to support a custom store, by providing all the session and config management; * Provide everything needed to serve data from popular stores like [InfluxDB](https://www.influxdata.com/products/influxdb/); * Off-the-shelf acquisition for high-rate data with low storage costs; The services are [Kubernetes](https://kubernetes.io/)-ready and include specific support for AWS. They can also be deployed as Windows or Linux binaries on a server or laptop. [RTA Server](rta-server/README.md) : Bundles together functionality from the **Session Service**, **Config Service**, and **Data Service** for development and simple deployments. [RTA Session Service](rta-sessionsvc/README.md) : Manages a PostgreSQL database of sessions and provides all the necessary browsing and query interfaces. [RTA Config Service](rta-configsvc/README.md) : Manages RTA configuration and provides the necessary retrieval interfaces. [RTA Data Service](rta-datasvc/README.md) : Provides compact, high-performance telmetry storage with sample rates from 0.001 Hz to 1 MHz. [RTA Gateway Service](rta-gatewaysvc/README.md) : Reverse proxy, typically used in front of the **Session Service**, **Config Service**, and **Data Services**. [RTA Stream Service](rta-streamsvc/README.md) : WebSocket server for streaming live session updates. [RTA Schema Mapping Service](rta-schemamappingsvc/README.md) : Repository for mappings between a source schema and the RTA domain model. [RTA Influx Data Service](rta-influxdatasvc/README.md) : Data adapter for [InfluxDB](https://www.influxdata.com/products/influxdb/). [Docker Images and Downloads](../downloads.md#docker){ .md-button .md-button--primary } --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Server This service is part of the RTA Toolkit Services suite. It bundles together functionality from the **Session Service**, **Config Service**, **Data Service** and **Stream Service** for development convenience. We recommend against using this service in production except for small private deployments. The service exposes internal gRPC interfaces to manage sessions, configuration and data, and an outward-facing REST interface to be exposed to _ATLAS_ users. It is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker _Storing configuration and data on the file system:_ ``` docker run --rm \ -v /path/to/store/configs:/data/rta-configs \ -v /path/to/store/data:/data/rta-data \ -e "RTA_PostgresConnectionString=Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" \ -e RTA_PostgresSchema=rta_sessions \ -e RTA_Store=File \ -p 8080:8080 -p 8081:8081 -p 8082:8082 \ mclarenapplied/rta-server ``` _Storing configuration and data on AWS:_ ``` docker run --rm \ -e "RTA_PostgresConnectionString=Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" \ -e RTA_PostgresSchema=rta_sessions \ -e RTA_Store=Aws \ -e RTA_AwsS3ConfigsBucket=myconfigsbucket \ -e RTA_AwsS3DataBucket=mydatabucket \ -e RTA_AwsDynamoDataTable=mydatatable \ -p 8080:8080 -p 8081:8081 -p 8082:8082 \ mclarenapplied/rta-server ``` ### Command Line _Storing configuration and data on the file system:_ ``` rta-server --PostgresConnectionString "Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" --PostgresSchema rta_sessions --Store File --FileConfigsPath /configs --FileDataPath /data ``` _Storing configuration and data on AWS:_ ``` rta-server --PostgresConnectionString "Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" --PostgresSchema rta_sessions --Store Aws --AwsS3ConfigsBucket myconfigsbucket --AwsS3DataBucket mydatabucket --AwsDynamoDataTable mydatatable ``` ## Ports | Port | Protocol | Usage | |------|----------|-------------------------------------------------------------------------------| | 8080 | HTTP | Expose to ATLAS via a TLS-enabled ingress unless using for local development | | 8081 | HTTPS | Expose to ATLAS directly or via an ingress unless using for local development | | 8082 | gRPC | Expose within local environment | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and all configured service URIs are accessible at their root path. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `PostgresSchema` is `RTA_POSTGRESSCHEMA` or `RTA_PostgresSchema`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | Default | |-----------------------------|------------------------------------------------------------------------------|-----------------|----------------| | `Postgres​ConnectionString` | Database connection string | yes | | | `Postgres​Schema` | Database schema name | no | `rta_sessions` | | `Postgres​Connection​Timeout` | Time (ms) to wait for a connection | no | `60000` | | `Store` | `File` or `Aws` | no | `file` | | `FileConfigsPath` | Path to config volume | if `Store=​File` | | | `FileDataPath` | Path to data volume | if `Store=​File` | | | `AwsS3​ConfigsBucket` | S3 bucket name for configs | if `Store=​Aws` | | | `AwsS3​DataBucket` | S3 bucket name for data | if `Store=​Aws` | | | `Aws​DynamoDataTable` | DynamoDB table name for data indexes | if `Store=​Aws` | | | `EnableDataWriter` | Enables the service for data _write_ access via gRPC; `true` or `false` | no | `true` | | `EnableGrpc` | Enables gRPC services to manage sessions, config and data; `true` or `false` | no | `true` | Memory Management options are also available. Refer to the **Data Service** README for details. ### Init Store `InitStore=true` will cause the service to initialize the database and stores on startup. This enables the service to be used to setup or upgrade the environment. | Option | Value |Required | Default | |-------------|-------------------|---------|---------| | `InitStore` | `true` or `false` | no | `true` | ### Service Info ATLAS will poll `/rta/v2/info` to check whether authentication is required, and retrieve a label and description to display to the user. These are configurable. This is particularly useful in an environment where users be using multiple services, perhaps hosted at different locations. | Option | Value | Required | Default | |-----------------------------|----------------------------|----------|-----------------------------| | `ServiceInfo:​Label` | Human-readable short label | no | Text including the hostname | | `ServiceInfo:Description` | Human-readable description | no | | ### Authentication and Authorization These options enable OAuth 2.0 support. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:Audience` | OAuth2 audience | if `EnableAuth=​true` | --- # gRPC APIs !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # gRPC APIs `rta-server` packages up three capabilities into a single process: * [Session Service](../rta-sessionsvc/README.md) * [Config Service](../rta-configsvc/README.md) * [Data Service](../rta-datasvc/README.md) gRPC interfaces are enabled by the `EnableGprc` option (default: `true`) and present the same API as the individual toolkit services. === "Session Service API" _See also: [Session Service gRPC API](../rta-sessionsvc/grpc.md)_ ``` protobuf --8<-- ../protos/Toolkit/session_service.proto --8<-- ``` === "Config Service API" _See also: [Config Service gRPC API](../rta-configsvc/grpc.md)_ ``` protobuf --8<-- ../protos/Toolkit/config_service.proto --8<-- ``` === "Data Service API" _See also: [Data Service gRPC API](../rta-datasvc/grpc.md)_ The `DataWriter` service is enabled by the `EnableDataWriter` service option (default: `true`). ``` protobuf --8<-- ../protos/Toolkit/data_service.proto --8<-- ``` --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Session Service This service is part of the RTA Toolkit Services suite. It manages a [PostgreSQL](https://www.postgresql.org/) database of _sessions_, which are data sets — like files — that can be opened in _McLaren ATLAS_. It exposes a gRPC interface to define and manage sessions from your environment, and an outward-facing REST interface to be exposed to users via the **Gateway Service**. This service is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker ``` docker run --rm \ -e "RTA_PostgresConnectionString=Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" \ -e RTA_PostgresSchema=rta_sessions \ -p 2650:2650 -p 2651:2651 -p 2652:2652 \ mclarenapplied/rta-sessionsvc ``` ### Command Line ``` rta-sessionsvc --PostgresConnectionString "Server=mydbhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2" --PostgresSchema rta_sessions ``` ## Setup Unlike RTA Server (`rta-server`), this service will not automatically create or upgrade the database schema. This ensures that any upgrade is applied in controlled circumstances, and with different credentials if desired. To configure the database, launch the service with the `--InitStoreAndExit=true` command line argument in addition to the other arguments or environment variables. The service will run and exit. !!! tip This can also be accomplished from Docker by opening a shell on the container and running: `/app/rta-server --InitStoreAndExit=true` assuming that the other configuration is present in environment variables. Restart any running services after applying the change. ## Ports | Port | Protocol | Usage | |------|------------|-------------------------------------| | 2650 | HTTP | Expose to ATLAS via Gateway Service | | 2651 | HTTPS | Expose to ATLAS via Gateway Service | | 2652 | gRPC | Expose within local environment | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and _PostgreSQL_ is accessible. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `PostgresSchema` is `RTA_POSTGRESSCHEMA` or `RTA_PostgresSchema`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | Default | |-----------------------------|------------------------------------|----------------|----------------| | `PostgresConnectionString` | Database connection string | yes | | | `PostgresSchema` | Database schema name | no | `rta_sessions` | | `PostgresConnectionTimeout` | Time (ms) to wait for a connection | no | `60000` | ### Init Store `InitStoreAndExit=true` will cause the service to initialize PostgreSQL and exit immediately. This enables the service to be used to setup or upgrade the environment. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `InitStoreAndExit` | `true` or `false` | no | See Setup notes, above. ### Authentication and Authorization These options enable OAuth 2.0 support. This service is intended to run within a private network, behind the **Gateway Service**, which will forward the Bearer token if present. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:Audience` | OAuth2 audience | if `EnableAuth=​true` | --- # gRPC API !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # gRPC API The service provides this private gRPC API to create and manage sessions. This API enables you to integrate this service into your private data environment. It's not part of the RTA interface specification, and never called by ATLAS. Typical Uses: * Define sessions as part of a batch process after new data arrives * Hook into your ingest pipeline, exposing new sessions to ATLAS in real time * Integrate with your other data management tools and web applications ## Definition Save this definition to file, and use [gRPC tools](https://grpc.io/) to create a client in your preferred language. ``` protobuf --8<-- ../protos/Toolkit/session_service.proto --8<-- ``` --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Config Service This service is part of the RTA Toolkit Services suite. It provides a repository for _configuration_ describing telemetry parameters, channels, events, etc. These are downloaded by _McLaren ATLAS_ as part of loading a session. It exposes a gRPC interface to define and manage sessions from your environment, and an outward-facing REST interface to be exposed to users via the **Gateway Service**. This service is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker _Storing configuration on the file system:_ ``` docker run --rm \ -v /path/to/store:/configs \ -e RTA_Store=File \ -e RTA_FilePath=/configs \ -p 2660:2660 -p 2661:2661 -p 2662:2662 \ mclarenapplied/rta-configsvc ``` _Storing configuration on AWS S3:_ ``` docker run --rm \ -e RTA_Store=Aws \ -e RTA_AwsS3Bucket=my-config-bucket \ -p 2660:2660 -p 2661:2661 -p 2662:2662 \ mclarenapplied/rta-configsvc ``` ### Command Line _Storing configuration on the file system:_ ``` rta-configsvc --Store File --FilePath /configs ``` _Storing configuration on AWS S3:_ ``` rta-configsvc --Store Aws --AwsS3Bucket my-config-bucket ``` ## Ports | Port | Protocol | Usage | |------|------------|-------------------------------------| | 2660 | HTTP | Expose to ATLAS via Gateway Service | | 2661 | HTTPS | Expose to ATLAS via Gateway Service | | 2662 | gRPC | Expose within local environment | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and the file system path or AWS S3 bucket is accessible. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration The service can be configured to store data either on the file system, or on AWS S3. Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `FilePath` is `RTA_FILEPATH` or `RTA_FilePath`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | |----------------------|-----------------------------|-----------------| | `Store` | `File` or `Aws` | yes | | `FilePath` | Path to config volume | if `Store=File` | | `AwsS3Bucket` | S3 bucket name | if `Store=Aws` | ### Init Store `InitStoreAndExit=true` will cause the service to initialize the store and exit immediately. This enables the service to be used to setup or upgrade the environment. | Option | Value |Required | Default | |----------------------|-----------------------------|----------------------|---------| | `InitStoreAndExit` | `true` or `false` | no | `false` | ### Authentication and Authorization These options enable OAuth 2.0 support. This service is intended to run within a private network, behind the **Gateway Service**, which will forward the Bearer token if present. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:Authority` | OAuth2 authorization server | if `EnableAuth=true` | | `Auth:Audience` | OAuth2 audience | if `EnableAuth=true` | --- # gRPC API !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # gRPC API ``` protobuf --8<-- ../protos/Toolkit/config_service.proto --8<-- ``` --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Data Service This service is part of the RTA Toolkit Services suite. It provides compact, high-performance telemetry storage, backed by either a file system or [Amazon S3](https://aws.amazon.com/s3/) and [DynamoDB](https://aws.amazon.com/dynamodb/). It handles backfill (late samples and channels) and copes with sample rates from 0.001 Hz to 1 MHz with stable memory use and back-pressure control. It exposes a gRPC interface to write or read data from your environment, and an outward-facing REST interface to be exposed to users via the **Gateway Service**. This service is intended to be used alongside the **Session Service**, **Config Service**, **Stream Service** (if live updates are required) and **Gateway Service** as a reverse proxy to bring the capabilties together. It is available as a binary for Windows and Linux, and as a Docker image. ## Performance Expected throughput on an AWS EC2 m5.xlarge Linux instance, per thread: | Operation | Local file system | Amazon S3 + DynamoDB | |-----------------------|------------------------------|-----------------------------| | Write | 30-60 million samples/sec | 15-30 million samples/sec | | Read (slow path[^1]) | 150-180 million samples/sec | 90-110 million samples/sec | | Read (fast path[^2]) | 500-1500 million samples/sec | 270-300 million samples/sec | For comparison, a well-tuned [InfluxDB](https://www.influxdata.com/products/influxdb/) facade can expect to read at 2-3 million samples/sec on equivalent hardware. ## When should you use this service? This service does not offer any search or analytical capabilities — only high-performance sample storage and retrieval, optimized for RTA. Other options to consider first: _Can you access data in its existing format?_ : If you have a significant amount of data in an existing format, it might be better to implement the data service interface to access this data in place. If the format is slow — such as CSV — you could cache data aggressively to compensate. _Can you use a time-series database, such as [InfluxDB](https://www.influxdata.com/products/influxdb/) or [TimescaleDB](https://www.timescale.com/)?_ : Much slower to ingest and query, and much more expensive to run: check your data rates to see what your running costs would be. However, in return you get powerful query and aggregation capabilities which — while not used by _ATLAS_ — can be really useful for dashboarding sofware like [Grafana](https://grafana.com/). _Can you use a columnar file format, like [Parquet](http://parquet.apache.org/)?_ : Slow to ingest and query, and will likely require some specialized ingest tooling to buffer and organize the data before it is written. But if you don't have strict ingest latency or query latency requirements, you can have the flexibility to use the same data storage with _ATLAS_ and Big Data tools like [Apache Spark](http://spark.apache.org/) or [Amazon Athena](http://spark.apache.org/). If Motion Applied don't provide a service for your existing format, perhaps we can help you build one? Refer to our developer documentation for more information. You might prefer this **Data Service** if you need: * High speed and efficiency * Bulk access to subsets of data — without resampling or aggregation * Reasonably compact, inexpensive storage (expect around 1.2 bytes/sample) * To store data as it arrives without a slow finalization stage * To work with minimal supporting infrastructure * Support for more data types than double-precision * Motion Applied ECU-specific channel types, such as slow-row data ## Usage ### Docker _Storing data on the file system:_ ``` docker run --rm \ -v /path/to/store:/data \ -e RTA_Store=File \ -e RTA_FilePath=/data \ -p 2670:2670 -p 2671:2671 -p 2672:2672 \ mclarenapplied/rta-datasvc ``` _Storing data in AWS:_ ``` docker run --rm \ -e RTA_Store=Aws \ -e RTA_AwsS3Bucket=mydatabucket \ -e RTA_AwsDynamoTable=mydatatable \ -p 2670:2670 -p 2671:2671 -p 2672:2672 \ mclarenapplied/rta-datasvc ``` ### Command Line _Storing data on the file system:_ ``` rta-datasvc --Store File --FilePath /data ``` _Storing data in AWS:_ ``` rta-datasvc --Store Aws --AwsS3Bucket mydatabucket --AwsDynamoTable mydatatable ``` ## Ports | Port | Protocol | Usage | |------|------------|-------------------------------------| | 2670 | HTTP | Expose to ATLAS via Gateway Service | | 2671 | HTTPS | Expose to ATLAS via Gateway Service | | 2672 | gRPC | Expose within local environment | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and the file system path or AWS S3 bucket and DynamoDB table are accessible. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration The service can be configured to store data either on the file system, or on AWS using [S3](https://aws.amazon.com/s3/) for the data and [DynamoDB](https://aws.amazon.com/dynamodb/) for indexes. Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `FilePath` is `RTA_FILEPATH` or `RTA_FilePath`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | Default | |------------------|--------------------------------------------------------------------------------------|-----------------|---------| | `EnableRead` | Enables the service for _read_ access via gRPC or REST; `true` or `false` | no | `true` | | `EnableWrite` | Enables the service for _write_ access via gRPC; `true` or `false` | no | `true` | | `Store` | `File` or `Aws` | yes | | | `FilePath` | Path to data volume | if `Store=​File` | | | `AwsS3Bucket` | S3 bucket name where the data is stored | if `Store=​Aws` | | | `AwsDynamoTable` | DynamoDB table name where the indexes are stored | if `Store=​Aws` | | We recommend that separate instances are run in _either_ the _read_ or _write_ roles. This helps balance resource use and allows the _write_ service to run with different permissions and network security rules. ### Init Store `InitStoreAndExit=true` will cause the service to initialize the store and exit immediately. This enables the service to be used to setup or upgrade the environment. | Option | Value |Required | Default | |----------------------|-----------------------------|----------------------|---------| | `InitStoreAndExit` | `true` or `false` | no | `false` | ### Memory Management These options manage the memory pool when writing data — and so are relevant only when `EnableWrite=true`. * _Write buffers_ are used to accumulate incoming data; * _Flush buffers_ are used to collate and hold data while is it being flushed to storage; * The _high threshold_ determines when data is transferred from _write buffers_ to _flush buffers_; * The _low threshold_ determines how much data should remain in the _write buffers_ to be merged with back-fill. The default options are quite aggressive, consuming 256 MiB (write) + 128 MiB (flush), beginning a flush at 50% write buffer usage and flushing down to 10%. You probably don't need to change these settings unless you want to raise or lower the memory requirement. | Option | Value |Required | Default | |--------------------------------------------|------------------------------------------------------------------|---------|---------| | `DataWriteMemory:​WriteBuffers` | Number of write buffers | no | `128` | | `DataWriteMemory:​FlushBuffers` | Number of flush buffer | no | `16` | | `DataWriteMemory:​WriteBufferSize` | Write buffer size (MiB) | no | `2` | | `DataWriteMemory:​FlushBufferSize` | Flush buffer size (MiB) | no | `8` | | `DataWriteMemory:​WriteBufferHighThreshold` | Proportion of write buffers to start flushing at (`0.0` - `1.0`) | no | `0.5` | | `DataWriteMemory:​WriteBufferLowThreshold` | Proportion of write buffers to flush down to (`0.0` - `1.0`) | no | `0.1` | | `DataWriteMemory:​Preallocate` | Pre-allocate memory; `true` or `false` | no | `false` | These are soft limits: additional memory is required to merge late data (back-fill), and run the .NET Core Garbage Collector. Pre-allocation helps ensure that the process is stable when under load in a resource-constrained environment — such as Kubernetes. When tuning for maximum throughput, aim for a write:flush ratio of about 2:1 to balance the buffers after allowing for data compression. Set the flush buffer size so that a full flush will span 4-16 buffers for increased I/O parallelism. ### Authentication and Authorization These options enable OAuth 2.0 support. This service is intended to run within a private network, behind the **Gateway Service**, which will forward the Bearer token if present. | Option | Value | Required | |---------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:​Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:​Audience` | OAuth2 audience | if `EnableAuth=​true` | [^1]: Slow-path access happens when the data needs to be decoded before it can be served to the user — for example, to apply a time-shift. [^2]: Fast-path access happens when no decoding is required and data can be routed directly from storage to _ATLAS_ with minimal intermediate processing. --- # gRPC API !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # gRPC API ``` protobuf --8<-- ../protos/Toolkit/data_service.proto --8<-- ``` --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Gateway Service This service is part of the RTA Toolkit Services suite. It is a reverse proxy, and would typically be deployed in front of: * **Session Service** — handling session management, browsing and filtering * **Config Service** — handling configuration storage and retrieval * **Data Service** — or storage-specific service, handling retrieval of data samples and events This microservices-style deployment is very flexible, allowing you to independently scale components and add in your own implementations. For example, if you want to use multiple data storage technologies, you can deploy multiple data services behind the **Gateway Service**. As an alternative, the **Server** package provides most of this functionality in one convenient executable for development use. The service exposes the outward-facing REST interface to _ATLAS_ users. There is no gRPC interface. It is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker ``` docker run --rm \ -e RTA_SessionServiceUri=http://myhost:2650/ \ -e RTA_ConfigServiceUri=http://myhost:2660/ \ -e RTA_DataServiceUris__mydata=http://myhost:2670/ \ -p 80:80 -p 443:443 \ mclarenapplied/rta-gatewaysvc ``` ### Command Line ``` rta-gatewaysvc --SessionServiceUri "http://myhost:2650/" --ConfigServiceUri "http://myhost:2660/" --DataServiceUris:mydata "http://myhost:2670/" ``` ## Ports | Port | Protocol | Usage | |------|------------|--------------------------------------------| | 80 | HTTP | Expose to ATLAS via a TLS-enabled ingress | | 443 | HTTPS | Expose to ATLAS directly or via an ingress | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and all configured service URIs are accessible at their root path. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration The service must be configured with URIs to the **Session Service**, **Config Service** and typically one or more data services. The RTA Toolkit provides drop-in implementations, but you can substitute your own as long as it implements the service interface. Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `SessionServiceUri` is `RTA_SESSIONSERVICEURI` or `RTA_SessionServiceUri`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | |-----------------------------|--------------------------------------------------------------------------------------------|--------------| | `SessionServiceUri` | URI to **Session Service** or customer implementation | yes | | `ConfigServiceUri` | URI to **Config Service** or customer implementation | yes | | `DataServiceUris:` | URIs to **Data Service** deployments or customer implementations, identified by `` | zero or more | ### Service Info ATLAS will poll `/rta/v2/info` to check whether authentication is required, and retrieve a label and description to display to the user. These are configurable. This is particularly useful in an environment where users be using multiple services, perhaps hosted at different locations. | Option | Value | Required | Default | |-----------------------------|----------------------------|----------|-----------------------------| | `ServiceInfo:​Label` | Human-readable short label | no | Text including the hostname | | `ServiceInfo:Description` | Human-readable description | no | | ### Authentication and Authorization These options enable OAuth 2.0 support. Bearer tokens are forwarded to the configured services. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:Audience` | OAuth2 audience | if `EnableAuth=​true` | --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Stream Service This service is part of the RTA Toolkit Services suite. It provides a WebSocket for streaming session updates so that Motion Applied ATLAS can display live data. It reads data from [Redis](https://redis.io/). There is no gRPC interface, and the outward-facing WebSocket interface should be exposed to users either directly or via the **Gateway Service**. This service is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker ``` docker run --rm \ -e RTA_RedisConnectionString=myredishost \ -p 80:80 -p 443:443 \ mclarenapplied/rta-streamsvc ``` ### Command Line ``` rta-streamsvc --RedisConnectionString myredishost ``` ## Ports | Port | Protocol | Usage | |------|------------|-------------------------------------------------------------------------| | 80 | HTTP | Expose to ATLAS via the Gateway Service or TLS-enabled ingress | | 443 | HTTPS | Expose to ATLAS directly, or via Gateway Service or TLS-enabled ingress | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and _Redis_ is accessible. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `Redis​ConnectionString` is `RTA_REDIS​CONNECTIONSTRING` or `RTA_Redis​ConnectionString`. For Linux and on _Kubernetes_, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | Default | |-------------------------------|-----------------------------------------------|----------------|----------------| | `Redis​ConnectionString` | Redis connection string | yes | | | `RedisDb` | Redis database id | no | `0` | ### Trimmer The _trimmer_ caps the length of _Redis_ streams to limit memory consumption. The `TargetStreamLength` should be set according to the data rate, data source, Redis instance size, and how long it takes for data to be flushed to storage in the deployment. | Option | Value | Required | Default | |------------------------------|-----------------------------------------------|----------------|----------------| | `EnableTrimmer` | `true` or `false` | no | `true` | | `Trimmer:​TargetStreamLength` | Target number of entries in each Redis stream | no | `10000` | | `Trimmer:​Interval` | Interval (sec) between stream trimming | no | `30` | | `Trimmer:​ApproxLength` | Approx trimming (`true` or `false`) | no | `false` | ### Authentication and Authorization These options enable OAuth 2.0 support. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:Audience` | OAuth2 audience | if `EnableAuth=​true` | --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Schema Mapping Service This service is part of the RTA Toolkit Services suite. It provides a repository to describe a mapping between a source schema in your data store, and the RTA domain model. RTA makes data requests using channel numbers using descriptions downloaded from the Config Service. When integrating a new data source, these requests need to be mapped to something meaningful to the source schema, which likely represents data as named fields in files, tables or documents. This service provides a simple, standard way to store that _Schema Mapping_ and perform lookups as required. !!! info The _Config Service_ is similar, but only describes the RTA domain model. The _Schema Mapping Service_ describes the relationship with the source schema, which is specific to the data storage technology. It exposes a gRPC interface to define and query the _Schema Mappings_. Aside from metrics and health checks, there is no outward-facing REST interface, and this service is strictly optional. This service is available as a binary for Windows and Linux, and as a Docker image. ## Usage ### Docker _Storing schema mappings on the file system:_ ``` docker run --rm \ -v /path/to/store:/data/rta-schema-mappings \ -e RTA_Store=File \ -e RTA_FilePath=/data/rta-schema-mappings \ -p 2680:2680 -p 2682:2682 \ mclarenapplied/rta-schemamappingsvc ``` _Storing schema mappings on AWS S3:_ ``` docker run --rm \ -e RTA_Store=Aws \ -e RTA_AwsS3Bucket=my-schema-mappings-bucket \ -p 2680:2680 -p 2682:2682 \ mclarenapplied/rta-schemamappingsvc ``` ### Command Line _Storing schema mappings on the file system:_ ``` rta-schemamappingsvc --Store File --FilePath /data/rta-schema-mappings ``` _Storing schema mappings on AWS S3:_ ``` rta-schemamappingsvc --Store Aws --AwsS3Bucket my-schema-mappings-bucket ``` ## Ports | Port | Protocol | Usage | |------|------------|--------------------------------------------------------------------| | 2680 | HTTP | Expose within local environment for health checks and metrics only | | 2682 | gRPC | Expose within local environment | ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and the file system path or AWS S3 bucket is accessible. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration The service can be configured to store data either on the file system, or on AWS S3. Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `FilePath` is `RTA_FILEPATH` or `RTA_FilePath`. | Option | Value | Required | |----------------------|-----------------------------|-----------------| | `Store` | `File` or `Aws` | yes | | `FilePath` | Path to config volume | if `Store=File` | | `AwsS3Bucket` | S3 bucket name | if `Store=Aws` | ### Init Store `InitStoreAndExit=true` will cause the service to initialize the store and exit immediately. This enables the service to be used to setup or upgrade the environment. | Option | Value |Required | Default | |----------------------|-----------------------------|---------|---------| | `InitStoreAndExit` | `true` or `false` | no | `false` | --- # gRPC API !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # gRPC API ``` protobuf --8<-- ../protos/Toolkit/schema_mapping_service.proto --8<-- ``` --- # README !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # RTA Influx Data Service This service is part of the RTA Toolkit Services suite. It provides a data adapter for _InfluxDB_, to be exposed to users via the **Gateway Service**. This service is intended to be used alongside the **Session Service**, **Config Service**, **Stream Service** (if live updates are required) and **Gateway Service** as a reverse proxy to bring the capabilties together. The **Schema Mapping Service** is also required as a support service, to describe the mapping between the _InfluxDB_ domain model (measurements, fields) and the RTA domain model. It is available as a binary for Windows and Linux, and as a Docker image. !!! info This version only supports InfluxQL, not Flux. ## Schema ### Data Identity and Filter Expressions The data identity is passed to the service in the URI: `http://myhost:2670/rta/v2/data//timestamped/-` For this service, the data identity is an _InfluxDB_ filter expression, in InfluxQL: > if the data identity (filter expression) is: `someData='50835ebe-a7f6-453b-9efd-c956e6da09f4'` > the resulting query will include: `... WHERE (someData='50835ebe-a7f6-453b-9efd-c956e6da09f4') ...` > the URI will look like: `http://myhost:2670/rta/v2/data/streamId='50835ebe-a7f6-453b-9efd-c956e6da09f4'/timestamped/-` This expression is merged into the data retrieval query and must therefore come from a trusted source. Do not compose this filter expression directly from user data. The expression is automatically wrapped in brackets in the query. You can use complex boolean expressions with relational comparisons. ### Publishing Schema Mappings You must publish a _Schema Mapping_ to the **Schema Mapping Service**. You must define: | Path | Value | Required | Default | |-----------------------------------------------|----------------------------------------------------------------------------------------------------------------|----------|-----------------------| | `data_source` | Namespace for Schema Mappings. Must match the `DataSource` service configuration option. | yes | | | `data_identity` | Query filter expression (see above). | yes | | | `schema_mapping.properties​[dialect]` | Query dialect. Only `influxql` is currently supported, but `flux` will be supported in future versions. | no | _from service config_ | | `schema_mapping.properties​[database]` | InfluxQL [database](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#database). | no | _from service config_ | | `schema_mapping.properties​[retention_policy]` | InfluxQL [retention policy](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#retention-policy-rp). | no | _from service config_ | | `schema_mapping.properties​[measurement]` | InfluxQL [measurement](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#measurement). | no | _from service config_ | | `schema_mapping.field_mappings` | Mapping from database fields to RTA channel numbers — see below. | no | | | `schema_mapping.event_mappings` | Mapping from event data to RTA event Ids — see below. | no | | The `field_mappings` enable the service to select fields matching the incoming RTA channel-based request. Each `FieldMapping` looks like this: | Path | Value | Required | |-------------------------------------------------|-----------------------------------------------------|----------| | `source_field` | Database field name. | yes | | `target_channel` | RTA channel number. | yes | | `source_field_data_type` | _currently unused: all fields are double-precision_ | no | | `target_channel_data_type` | _currently unused, but may be used in future_ | no | | `properties` | _currently unused for this service_ | no | #### Sample _In JSON format:_ ``` json { "dataSource": "mydata", "dataIdentity": "someData='50835ebe-a7f6-453b-9efd-c956e6da09f4'", "schemaMapping": { "properties": { "dialect": "influxql", "database": "mydb", "measurement": "mydata" }, "fieldMappings": [ { "sourceField": "aaa", "targetChannel": 0 }, { "sourceField": "bbb", "targetChannel": 1 }, { "sourceField": "ccc", "targetChannel": 2 } ], "eventMappings": [] } } ``` In this example, an RTA request for channels `1,2` would translate to an InfluxQL query like this: `SELECT "bbb", "ccc" FROM "mydb"."autogen"."mydata" WHERE (someData='50835ebe-a7f6-453b-9efd-c956e6da09f4')` ## Usage ### Docker _InfluxDB 1.8+:_ ``` docker run --rm \ -e RTA_DataSource=mydata \ -e RTA_SchemaMappingGrpcUri=http://myhost:2682/ \ -e RTA_InfluxUri=http://myhost:8086/ \ -e RTA_MaxFrequency=100 \ -p 2690:2690 -p 2691:2691 \ mclarenapplied/rta-influxdatasvc ``` _Earlier versions:_ ``` docker run --rm \ -e RTA_DataSource=mydata \ -e RTA_SchemaMappingGrpcUri=http://myhost:2682/ \ -e RTA_InfluxUri=http://myhost:8086/ \ -e RTA_InfluxHealthCheckPath=/ping \ -e RTA_MaxFrequency=100 \ -p 2690:2690 -p 2691:2691 \ mclarenapplied/rta-influxdatasvc ``` ### Command Line ``` rta-influxdatasvc --DataSource mydata --SchemaMappingGrpcUri "http://myhost:2682/" --InfluxUri "http://myhost:8086/" --MaxFrequency 100 ``` ## Ports | Port | Protocol | Usage | |------|------------|-------------------------------------| | 2690 | HTTP | Expose to ATLAS via Gateway Service | | 2691 | HTTPS | Expose to ATLAS via Gateway Service | There is no gRPC interface, and you cannot write data using this service. ## Monitoring HTTP `GET /` : Returns `200 OK` and reports the software version. In _Kubernetes_, use this for _liveness_ probes. HTTP `GET /health` : Returns `200 OK` if the service is up and both _InfluxDB_ and the **Schema Mapping Service** are reachable and healthy. In _Kubernetes_, use this for _readiness_ probes. HTTP `GET /metrics` : [Prometheus](https://prometheus.io/) metrics, including the health check status. ## Configuration The service must be configured with URIs to _InfluxDB_ and the **Schema Mapping Service**. Options can be set using both environment variables and command line arguments, or by the provided `appsettings.config` JSON configuration file. Environment variables are all prefixed with `RTA_`, and can be written in `ALLCAPS` or `MixedCase`. For example, `DataSource` is `RTA_DATASOURCE` or `RTA_DataSource`. For Linux and on Kubernetes, also replace `:` with `__`. For example, `Category:Item` is `RTA_CATEGORY__ITEM`. | Option | Value | Required | Default | |-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------| | `DataSource` | Namespace for lookups to the **Schema Mapping Service**.\ Should match the token used with the **Session Service** and **Gateway Service**. | yes | | | `SchemaMappingGrpcUri` | URI to the **Schema Mapping Service** gRPC interface — e.g. `http://localhost:2682/` | yes | | | `InfluxUri` | URI to _InfluxDB_ — e.g. `http://localhost:8086/` | yes | | | `InfluxHealthCheckPath` | Path to use for health checks against _InfluxDB_. For versions prior than 1.8, use `/ping`. | no | `/health` | | `MaxFrequency` | Approximate maximum frequency of data, for performance optimization. | no | `100` | ### Authentication and Authorization These options enable OAuth 2.0 support. Bearer tokens are forwarded to the configured services. | Option | Value | Required | |----------------------|-----------------------------|----------------------| | `EnableAuth` | `true` or `false` | no | | `Auth:​Authority` | OAuth2 authorization server | if `EnableAuth=​true` | | `Auth:​Audience` | OAuth2 audience | if `EnableAuth=​true` | ### Schema Mapping Defaults These options specify default Schema Mapping properties if not specified via the **Schema Mapping Service**. | Option | Value | Required | Default | |--------------------------------------------------|----------------------------------------------------------------------------------------------------------------|--------------|------------| | `SchemaMappingDefaults:Dialect` | Query dialect to use for lookups. Currently only supports `influxql`, but will also support `flux` in future. | no | `influxql` | | `SchemaMappingDefaults:​InfluxQL:​Database` | InfluxQL [database](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#database). | no | | | `SchemaMappingDefaults:​InfluxQL:​RetentionPolicy` | InfluxQL [retention policy](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#retention-policy-rp). | no | `autogen` | | `SchemaMappingDefaults:​InfluxQL:Measurement` | InfluxQL [measurement](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#measurement). | no | | --- # Prerequisites !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Prerequisites Before completing these tutorials, you'll need a few things. ## ATLAS * ATLAS 2021-R3 or later You'll need an ATLAS Licence with the _RTA Data Service Connections_ feature. [Contact](mailto:sales@motionapplied.com) your sales rep or Motion Applied Field Application Engineer if you don't have this. ## PostgreSQL * [PostgreSQL 9.6 or later](https://www.postgresql.org/) !!! note These demos use minimal PostgreSQL resources. You can install PostgreSQL directly into your development environment, or run from a Docker container. ## Code Samples Clone our [RTA.Examples](https://github.com/mat-docs/RTA.Examples) git repo. ## Development Environment === "Visual Studio Code" * [.NET 5.x SDK](https://dotnet.microsoft.com/download/dotnet/5.0) * [Visual Studio Code](https://code.visualstudio.com/) with the latest [C# extension](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) === "Visual Studio" * [Visual Studio 2019 version 16.9.2 or later](https://visualstudio.microsoft.com/downloads/?utm_medium=microsoft&utm_source=docs.microsoft.com&utm_campaign=inline+link&utm_content=download+vs2019) with the **.NET Core cross-platform development** workload installed. The .NET 5.0 SDK is automatically installed when you select this workload. === "JetBrains Rider" * [.NET 5.x SDK](https://dotnet.microsoft.com/download/dotnet/5.0) * [JetBrains Rider 2020.3 or later](https://www.jetbrains.com/rider/) ## GitHub Account These tutorials use some packages hosted on [GitHub](https://github.com/mat-docs/packages). You'll need a free account to access these. ## Recommended Tools * [Docker](https://www.docker.com/products/docker-desktop) * [grpcui](https://github.com/fullstorydev/grpcui/releases) --- # Walkthrough !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Quick-Start — Walkthrough This tutorial creates a simple data environment for ATLAS to connect to. At the end, you'll have a sample data loader using functionality from Motion Applied's [implementation toolkit](../../../services/index.md). In this tutorial, you: * Setup [RTA Server](../../../services/rta-server/README.md) from the [implementation toolkit](../../../services/index.md) * Generate and publish sample test [data](../../data/index.md), [configuration](../../configuration/index.md) and [sessions](../../sessions/index.md) using the [gRPC Services](../../../services/rta-server/grpc.md) * Test the deployment with ATLAS !!! tip These code samples are in C# for .NET 5.0, and use Motion Applied [NuGet packages](../../../downloads.md/#nuget-packages) to keep things as simple as possible. We recommend you follow this tutorial even if you are planning to do your integration in another language. The concepts should translate very easily once you have a working example. ## Prerequisites Check out the common tutorial [prerequisites](../prerequisites.md) before you start. You're going to need: * [ATLAS](../prerequisites.md#atlas) * [PostgreSQL](../prerequisites.md#postgresql) * [Development Environment](../prerequisites.md#development-environment) and optionally: * [Docker](../prerequisites.md#recommended-tools) * [grpcui](../prerequisites.md#recommended-tools) ## Step 1: Deploy [RTA Server](../../../services/rta-server/README.md) [RTA Server](../../../services/rta-server/README.md) is designed to support early development activity. It provides session management, stores configuration, and can also store and serve data. === "Windows: Docker" Install [Docker Desktop for Windows](https://www.docker.com/products/docker-desktop). Create convenient directories to store data and configuration. For example: ```powershell mkdir C:\rta\configs mkdir C:\rta\data ``` Use these paths in the command below for volume mounts (`-v`). This example assumes you are running PostgreSQL on your host machine, accessed through [`host.docker.internal`](https://docs.docker.com/docker-for-windows/networking/#use-cases-and-workarounds) in the connection string below. Update the command below with the hostname (if not `host.docker.internal`) and user/password. By default, [RTA Server will initialize the database](../../../services/rta-server/README.md#init-store) using these credentials. In PowerShell: ``` powershell hl_lines="4" docker run --rm ` -v "C:\rta\configs:/data/rta-configs" ` -v "C:\rta\data:/data/rta-data" ` -e "RTA_PostgresConnectionString=Server=host.docker.internal;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" ` -e "RTA_Store=File" ` -p 8080:8080 -p 8082:8082 ` mclarenapplied/rta-server ``` === "Windows: rta-server.exe" If Docker is not available in your development environment, you can [download](../../../downloads.md#binaries) and run the service as an executable. Create convenient directories to store data and configuration. For example: ``` powershell mkdir C:\rta\configs mkdir C:\rta\data ``` Use these paths in the command below. In PowerShell: ``` powershell rta-server.exe ` --PostgresConnectionString "Server=localhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" ` --Store File --FileConfigsPath C:\rta\configs --FileDataPath C:\rta\data ``` === "Linux/Mac: Docker" Create convenient directories to store data and configuration. For example: ``` bash mkdir -p /tmp/rta/configs mkdir -p /tmp/rta/data ``` Use these paths in the command below for volume mounts (`-v`). Update the command below with the hostname (if not `localhost`) and user/password. By default, [RTA Server will initialize the database](../../../services/rta-server/README.md/#init-store) using these credentials. In a terminal: ``` bash docker run --rm \ -v "/tmp/rta/configs:/data/rta-configs" \ -v "/tmp/rta/data:/data/rta-data" \ -e "RTA_PostgresConnectionString=Server=localhost;Port=5432;Database=postgres;User Id=postgres;Password=hunter2;" \ -e "RTA_Store=File" \ -p 8080:8080 -p 8082:8082 \ mclarenapplied/rta-server ``` !!! notes "Docker Notes" The `-v` option mounts a directory on the host (left-hand-side) into a path in the container (right-hand-side). The host path needs to be an absolute path. If you have something else running on port 8080 or 8082, you can [use Docker to change the port](https://docs.docker.com/config/containers/container-networking/). ## Step 2: Check the Service ### HTTP (port 8080) In a web browser, go to [http://localhost:8080](http://localhost:8080) You should see some text similar to this: ``` MAT.OCS.RTA.Toolkit.Hosts.Server 2.0.0.0 Provides RTA Toolkit services as a single deployment Grpc: enabled Data write: enabled Data admin: enabled Metrics: /metrics Health: /health ``` ??? notes "Troubleshooting" * Is something else running on port 8080? Also browse to [http://localhost:8080/health](http://localhost:8080/health) This should report: `Healthy`. If the service says it is `Unhealthy`, the console log output should say why. ??? notes "Troubleshooting" Possible reasons why RTA Server would report `Unhealthy`: * Can't communicate with PostgreSQL, or wrong PostgreSQL credentials * Can't find the specified directories for configs and data ### gRPC (port 8082) Grab a copy of [grpcui](https://github.com/fullstorydev/grpcui/releases). From a terminal: ``` grpcui -plaintext localhost:8082 ``` If successful, this will open a web page where you can browse and interact with the gRPC services — which will be useful later. ??? notes "Troubleshooting" * Make sure you ran `grpcui` from a command prompt / terminal * Is something else running on port 8082? ## Step 3: Build the Demo project ### Get the sample source code Clone the sample project from [RTA.Examples on GitHub](https://github.com/mat-docs/RTA.Examples). You will need a free GitHub account. ### Setup the NuGet Package feed The Motion Applied GitHub site has [instructions for setting up the feed](https://github.com/mat-docs/packages) in Visual Studio. Use your GitHub account to create a PAT (access token) with the `read:packages` permission, and [modify the `NuGet.Config` file](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-nuget-registry#authenticating-with-a-personal-access-token) in the sample project to match. The resulting `NuGet.Config` file should look something like this: ``` xml ``` From the terminal (or using Visual Studio), in the source code directory: ``` bash dotnet restore ``` ### Build the projects From the terminal (or using Visual Studio), build the solution: ``` bash dotnet build ``` ??? "Troubleshooting" * Make sure you are in the correct directory * Make sure you have .NET 5.0: `dotnet --version` should return something like `5.0.xxx` ## Step 4: Run the Demo Open the solution in your development environment (e.g. Visual Studio Code). There are several projects. You will be running `RTA.Examples.Loader`. From the terminal, navigate to this sub-directory, and then: ``` bash dotnet run ``` If the service has been setup correctly, this will create an example session. You can create more sessions by running it again. ??? Troubleshooting * Make sure RTA Server is running * Make sure you're running the `RTA.Examples.Loader` project * If you changed the gRPC port, update the demo source code ## Step 5: Setup ATLAS ### Setup the connection Open the ATLAS Session Browser. Right-click on **Sources** and select **Service Connections...** !!! info If you don't see this option, your ATLAS licence does not have the _RTA Data Service Connections_ feature. Please [Contact](mailto:sales@motionapplied.com) your sales rep or Motion Applied Field Application Engineer. From the **Service Connection Manager** dialog, add a new connection like this: * **URI**: `http://localhost:8080` * **WebSocket URI**: _(blank)_ * **Friendly Name**: `localhost` Close the dialogs, and right-click **Sources** and select **Refresh**. The _localhost_ source should appear. Tick the _localhost_ source and press the **Refresh** button on the right-hand-side of the dialog. You should see one or more sessions. === "Open Session Browser" ATLAS Session Browser - no connections === "Right-click to add Service Connection" ATLAS Session Browser - connections menu === "Dialog" ATLAS Session Browser - Service Connections dialog === "Add Service Connection" ATLAS Session Browser - Adding a connection === "Tick Source to list sessions" ATLAS Session Browser - Session Listing ??? notes "Troubleshooting" * If the _localhost_ source is Red, maybe you mistyped the URI or the service isn't running? * Do you have the right version of ATLAS (see [Prerequisites](#prerequisites))? ### Test the session Double-click to load a session. Add a Waveform display, and press `P` to add parameters using the _Parameter Browser_ by double-clicking or dragging. You should see waveform traces. === "Open Parameter Browser" ATLAS Parameter Browser menu === "Add Parameters to Waveform" ATLAS Parameter Browser === "Waveform display" ATLAS Waveform display ## Next Steps [Review the sample project](review.md) to understand how each routine contributes to the demo integration. You can use this project to experiment with RTA features in this documentation. --- # Review !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Quick-Start — Review !!! info Complete the [Walkthrough](index.md) first. This demo project illustrates a data loading pattern, where data is format-shifted for use with ATLAS. This is not the only way to use RTA. You can also connect to existing stores using a [Data Service as an adapter](../../../../../key-functionality/integrate/rta/introduction/data-services.md). ## Importing NuGet Packages The project imports [NuGet Packages](../../../downloads.md#nuget-packages) from the Motion Applied GitHub feed: ```xml ``` This package provides everything needed to write a tool that communicates with the [Toolkit Services](../../../services/index.md). Pre-compiled gRPC clients are included. You can [compile your own](https://developers.google.com/protocol-buffers) from the gRPC API schemas: * [Session Service gRPC API](../../../services/rta-sessionsvc/grpc.md) * [Config Service gRPC API](../../../services/rta-configsvc/grpc.md) * [Data Service gRPC API](../../../services/rta-datasvc/grpc.md) * [Schema Mapping Service gRPC API](../../../services/rta-schemamappingsvc/grpc.md) ## Configuring gRPC Clients A gRPC network connection is called a _Channel_ in most languages: ```c# using var channel = GrpcChannel.ForAddress("http://localhost:8082"); ``` The `using` keyword means the channel will get cleaned up when it goes out of scope. Creating channels involves setting up a new network connection, so they should be reused. The clients can share the same channel if the same endpoint is offering multiple gRPC services (like [RTA Server](../../../services/rta-server/grpc.md)): ```c# var sessionClient = new SessionStore.SessionStoreClient(channel); var configClient = new ConfigStore.ConfigStoreClient(channel); var dataClient = new DataWriter.DataWriterClient(channel); ``` !!! tip Prior to .NET 5.0, there is an extra step to communicate with a gRPC service without TLS: [https://docs.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-5.0#call-insecure-grpc-services-with-net-core-client](https://docs.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-5.0#call-insecure-grpc-services-with-net-core-client) ## Uploading Data This project demonstrates loading data into the toolkit [Data Service](../../../services/rta-datasvc/grpc.md), using a simple Signal Generator as a source. Data is uploaded to the service as a stream of messages. ```c# using var dataStream = dataClient.WriteDataStream(); var requestStream = dataStream.RequestStream; ``` The first message must be the data identity, so the service knows how to store the data: ```c# await requestStream.WriteAsync(new WriteDataStreamMessage { DataIdentity = dataIdentity }); ``` The data is sent channel-by-channel in short bursts. In this example, the data is [represented as periodic samples](../../data/index.md#periodic-data) since the timestamps are evenly spaced. ```c# var burst = new PeriodicData { ChannelId = (uint) f, StartTimestamp = startNanos + offset, Interval = intervalNanos, Samples = burstLength, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(burstSamples.AsSpan())) }; await requestStream.WriteAsync(new WriteDataStreamMessage { PeriodicData = burst }); ``` Finally, complete the stream call: ```c# await requestStream.CompleteAsync(); ``` !!! tip Avoid unneccessary memory allocation by reusing data arrays where possible. This will have the biggest impact on the performance of your tools and services. !!! note A burst of data should contain up 128 samples. Note that a gRPC message cannot exceed 4 MiB unless all clients and services are configured for a higher limit. ## Configuration The configuration tells ATLAS what data is available, and how to render it. === "Sample Code" Configuration is defined here using the _MAT.OCS.Configuration_ library, which is brought in as a [NuGet Package](../../../downloads.md#nuget-packages). The _Builder_ objects all have constructors indicating the mandatory properties, and the overall `ConfigurationBuilder` is consistency-checked when `BuildConfiguration()` is called. The **Equivalent JSON** tab illustrates the configuration you need to create if you are working without this helper library. ``` C# var channels = new ChannelBuilder[Fields.Length]; var parameters = new ParameterBuilder[Fields.Length]; for (var f = 0; f < Fields.Length; f++) { channels[f] = new ChannelBuilder((uint) f, intervalNanos, DataType.Signed16Bit, ChannelDataSource.Periodic); parameters[f] = new ParameterBuilder($"{Fields[f]}:demo", Fields[f], $"{Fields[f]} field") { ChannelIds = {(uint) f}, MinimumValue = -1500, MaximumValue = +1500 }; } var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = parameters, Channels = channels } } }.BuildConfiguration(); ``` === "Equivalent JSON" The outermost element is the app name. Configuration can define multiple apps. * `"tree"` describes a tree of references to `"parameters"` by `"id"` * `"parameters"` is a list of all the parameters in the app, referencing channels by `"id"` * `"channels"` is a list of all the channels in the app ``` json { "demo": { "tree": { "params": [ "alpha:demo", "beta:demo", "gamma:demo" ] }, "parameters": [ { "id": "alpha:demo", "name": "alpha", "desc": "alpha field", "min": -1500.0, "max": 1500.0, "conv": "1to1:demo", "channels": [ 0 ] }, { "id": "beta:demo", "name": "beta", "desc": "beta field", "min": -1500.0, "max": 1500.0, "conv": "1to1:demo", "channels": [ 1 ] }, { "id": "gamma:demo", "name": "gamma", "desc": "gamma field", "min": -1500.0, "max": 1500.0, "conv": "1to1:demo", "channels": [ 2 ] } ], "channels": [ { "id": 0, "source": "periodic", "interval": 10000000, "type": "s16" }, { "id": 1, "source": "periodic", "interval": 10000000, "type": "s16" }, { "id": 2, "source": "periodic", "interval": 10000000, "type": "s16" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` Note that there is also a list of `"conversions"`. A single 1:1 (input = output) conversion is defined and referenced from each parameter. The _MAT.OCS.Configuration_ library does this automatically if you don't specify a Conversion. Each field is mapped onto both a channel and parameter. !!! info _Channels_ represent data capture and storage, and can be mapped onto multiple parameters. _Parameters_ are the data access abstraction, and can merge data from multiple channels. The channel properties need to match the data: * `Interval`: spacing between timestamps, in nanoseconds * `Data Type`: matching the C# samples array (`short[]` in this case) * `Data Source`: indicating the data representation so ATLAS knows what to request (`Periodic` in this case) The parameter must have: * `Identifier` (e.g. `alpha:demo`): * globally unique identifier * includes the app name as a suffix (`:demo` in this example) * `Name` (e.g. `alpha`): * generally unique (non-strict) * displayed to the user in the ATLAS Parameter Browser * used as a variable name in functions — so should not contain spaces or special characters * `Description` (e.g. `Angle of attack`): * displayed to the user as a longer description in the ATLAS Parameter Browser Parameters have some other optional properties. In this example, the _MinimumValue_ and _MaximumValue_ indicate the valid data range so that ATLAS can auto-scale the displays. Configuration is published with a unique identifier and can be reused across multiple sessions **as long as the config is unchanged** (as it will be cached aggressively). ```C# // arbitrary, unique and can be reused as long as the config stays the same const string configIdentifier = "6D711DBC-5F0D-47D2-B127-8C9D20BCEC02"; await configClient.PutConfigAsync(configIdentifier, config); ``` The `PutConfigAsync` method is an extension method which abstracts away the serialization. ## Session The session descriptor is very straightforward. These properties are mandatory: * _Identity_ — a human-readable label * _Timestamp_ (ISO 8601) — including a time-zone where applicable The additional properties reflect the state of the session after adding data: * _TimeRange_ — to reflect the actual range of data in the session, in nanoseconds * _ConfigBindings_ — associate the session with the published configuration identifier If not otherwise specified, the initial state of the session is `closed`. === "C# Sample" ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = dataIdentity, CreateIfNotExists = new() { Identifier = $"PeriodicData Demo {timestamp:f}", Timestamp = timestamp.ToString("O") }, Updates = { new SessionUpdate { SetTimeRange = new() { StartTime = startNanos, EndTime = startNanos + durationNanos } }, new SessionUpdate { SetConfigBindings = new() { ConfigBindings = { new ConfigBinding { Identifier = configIdentifier } } } } } }); ``` In this gRPC call: * `Identity` specifies the session, and is the same as the `dataIdentity` for RTA Server * `CreateIfNotExists` is used only if the session is not already present Subsequent service calls could be made to update the session metadata. === "JSON" ``` json { "sessions": [ { "identity": "f10f4a19-3831-4dd6-a178-faa94931a5b0", "state": "closed", "timestamp": "2021-05-06T18:58:56.20289+01:00", "startTimestamp": "2021-05-06T18:58:56.20289+01:00", "endTimestamp": "2021-05-06T19:08:56.20289+01:00", "timeRange": { "startTime": 1620323936202890000, "endTime": 1620324536202890000 }, "identifier": "PeriodicData Demo 06 May 2021 18:58", "details": {}, "alternates": [], "children": [], "configBindings": [ { "identifier": "6D711DBC-5F0D-47D2-B127-8C9D20BCEC02", "channelOffset": 0 } ], "markers": [] } ] } ``` --- # Walkthrough !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Microservices — Walkthrough This tutorial converts the [Quick-Start Tutorial](../quick-start/index.md) deployment from [RTA Server](../../../services/rta-server/README.md) into microservices. At the end, you'll have a production-ready architecture that can be scaled and modified with data adapter services to suit your environment. In this tutorial, you: * Deploy the [Session](../../../services/rta-sessionsvc/README.md), [Config](../../../services/rta-configsvc/README.md), [Data](../../../services/rta-datasvc/README.md) and [Gateway Service](../../../services/rta-gatewaysvc/README.md) using [Docker Compose](https://docs.docker.com/compose/) * Add a [Data Binding](../../sessions/data-bindings.md) to locate the previous tutorial data in the [Data Service](../../../services/rta-datasvc/README.md) * Modify the data generator to publish new sample sessions * Test the deployment with ATLAS !!! tip These code samples are in C# for .NET 5.0, and use Motion Applied [NuGet packages](../../../downloads.md/#nuget-packages) to keep things as simple as possible. We recommend you follow this tutorial even if you are planning to do your integration in another language. The concepts should translate very easily once you have a working example. ## Prerequisites You'll want to complete the [Quick-Start Walkthrough](../quick-start/index.md) first. This will ensure that your development environment and database are setup and ATLAS is ready to use. Check out the common tutorial [prerequisites](../prerequisites.md) before you start. You're going to need: * [ATLAS](../prerequisites.md#atlas) * [PostgreSQL](../prerequisites.md#postgresql) * [Development Environment](../prerequisites.md#development-environment) * [Docker](../prerequisites.md#recommended-tools) * [grpcui](../prerequisites.md#recommended-tools) ### Why do I need Docker? The microservices architecture is designed for a [container orchestrator](https://www.redhat.com/en/topics/containers/what-is-container-orchestration) or similar automated rollout. Popular choices are: * [Kubernetes](https://kubernetes.io/) (available on [AWS](https://aws.amazon.com/eks), [Azure](https://azure.microsoft.com/en-gb/services/kubernetes-service/), and [Google Cloud](https://cloud.google.com/kubernetes-engine)) * [AWS ECS (Elastic Container Service / Fargate)](https://aws.amazon.com/ecs/) * [Docker Swarm](https://docs.docker.com/engine/swarm/) * [Hashicorp Nomad](https://www.nomadproject.io/) For this tutorial, we'll use [Docker Compose](https://docs.docker.com/compose/) to download and configure the four services running together. !!! info "If you can't use Docker" We also provide Windows and Linux executables. You can start these in separate terminals or using a script. In production, you could deploy these as services using automation tools like [Puppet](https://puppet.com/) and [Ansible](https://www.ansible.com/), or [Nomad](https://www.nomadproject.io/) (without containers). ## Step 1: Stop [RTA Server](../../../services/rta-server/README.md) If you completed the [Quick-Start Tutorial](../quick-start/index.md), you might still have [RTA Server](../../../services/rta-server/README.md) running. This can run side-by-side with the microservices, but it's best to terminate it to avoid confusion. [RTA Server](../../../services/rta-server/README.md) will stop if you press ++ctrl+c++. If you used Docker, use `docker ps` and `docker kill` to make sure the service has stopped. Verify that it is not running by browsing [http://localhost:8080/](http://localhost:8080/). ## Step 2: Deploy the Toolkit Microservices Create a directory to hold the Docker Compose file for the tutorials — e.g. `rta-tutorials` You'll reuse and extend this script in the other tutorials. !!! tip [Visual Studio Code](https://code.visualstudio.com/) is a great YAML editor. * YAML is whitespace-sensitive. * Indentation as seen below creates a nested object structure; hyphens indicate a list. * Strings can be double-quoted, single-quoted or not quoted at all — but it's best to use quotes if you want to make absolutely sure your markup is going to be parsed as a string, not a number or a boolean. [Be aware that `yes` and `no` will get parsed as booleans unless quoted as `"yes"` and `"no"`](https://medium.com/@lefloh/lessons-learned-about-yaml-and-norway-13ba26df680). Save and edit this script into the directory you created, as `docker-compose.yaml`: === "Windows" ```yaml services: rta-sessionsvc-init: image: mclarenapplied/rta-sessionsvc:latest restart: "no" environment: - RTA_PostgresConnectionString=Host=host.docker.internal; Username=postgres; Password=hunter2; entrypoint: ["/app/rta-sessionsvc", "--InitStoreAndExit=true"] rta-sessionsvc: image: mclarenapplied/rta-sessionsvc:latest restart: "always" depends_on: - rta-sessionsvc-init ports: - "2650:2650" - "2652:2652" environment: - RTA_PostgresConnectionString=Host=host.docker.internal; Username=postgres; Password=hunter2; rta-configsvc: image: mclarenapplied/rta-configsvc:latest restart: "always" ports: - "2660:2660" - "2662:2662" volumes: - C:\rta\configs:/data/rta-configs environment: - RTA_Store=File - RTA_FilePath=/data/rta-configs rta-datasvc: image: mclarenapplied/rta-datasvc:latest restart: "always" ports: - "2670:2670" - "2672:2672" volumes: - C:\rta\data:/data/rta-data environment: - RTA_Store=File - RTA_FilePath=/data/rta-data rta-gatewaysvc: image: mclarenapplied/rta-gatewaysvc:latest restart: "always" depends_on: - rta-sessionsvc - rta-configsvc - rta-datasvc ports: - "8080:80" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc=http://rta-datasvc:2670 ``` === "Linux/Mac" ```yaml services: rta-sessionsvc-init: image: mclarenapplied/rta-sessionsvc:latest restart: "always" environment: - RTA_PostgresConnectionString=Host=localhost; Username=postgres; Password=hunter2; entrypoint: ["/app/rta-sessionsvc", "--InitStoreAndExit=true"] restart: "no" rta-sessionsvc: image: mclarenapplied/rta-sessionsvc:latest restart: "always" depends_on: - rta-sessionsvc-init ports: - "2650:2650" - "2652:2652" environment: - RTA_PostgresConnectionString=Host=localhost; Username=postgres; Password=hunter2; rta-configsvc: image: mclarenapplied/rta-configsvc:latest restart: "always" ports: - "2660:2660" - "2662:2662" volumes: - /tmp/rta/configs:/data/rta-configs environment: - RTA_Store=File - RTA_FilePath=/data/rta-configs rta-datasvc: image: mclarenapplied/rta-datasvc:latest restart: "always" ports: - "2670:2670" - "2672:2672" volumes: - /tmp/rta/data:/data/rta-data environment: - RTA_Store=File - RTA_FilePath=/data/rta-data rta-gatewaysvc: image: mclarenapplied/rta-gatewaysvc:latest restart: "always" depends_on: - rta-sessionsvc - rta-configsvc - rta-datasvc ports: - "8080:80" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc=http://rta-datasvc:2670 ``` * Update the database username/password (`postgres`/`hunter2`) to match your deployment: * [X] `rta-sessionsvc-init` * [X] `rta-sessionsvc` * Update the volume paths if you changed them in the [Quick-Start Tutorial](../quick-start/index.md): * [X] `rta-configsvc` * [X] `rta-datasvc` !!! alert "This script expects PostgreSQL to be running on your host machine" If you are running PostgreSQL in Docker, things get a bit more complicated. You can either: * Add an extra section to the compose file to run Postgres, so the containers can see each other * Connect to the database by IP address, [using the bridge network](https://www.tutorialworks.com/container-networking/) Start the services from a terminal in the folder where you saved the `docker-compose.yaml` file: ```bash docker compose up ``` ## Step 3: Check the Services ### HTTP In a web browser, go to [http://localhost:8080](http://localhost:8080) You should see some text similar to this: ``` Provides a Gateway for RTA Toolkit services 0.0.0.0 Provides RTA Toolkit services as a single deployment Grpc: enabled DataWriter: enabled Metrics: /metrics Health: /health ``` ??? notes "Troubleshooting" * Is something else running on port 8080? Also browse to [http://localhost:8080/health](http://localhost:8080/health) This should report: `Healthy`. This will verify that the three other services are reachable. If the service says it is `Unhealthy`, the console log output should say why. Perform the same checks on the other three services: | Service | HTTP Port | |----------------|-----------| | rta-sessionsvc | 2650 | | rta-configsvc | 2660 | | rta-datasvc | 2670 | ??? notes "Troubleshooting" * Can the [Session Service (rta-sessionsvc)](../../../services/rta-sessionsvc/README.md) communicate with PostgreSQL? * Is the host correct? * Is the username/password correct? * Does the [Config Service (rta-configsvc)](../../../services/rta-configsvc/README.md) have the same volume/folder mounted as used for the [Quick-Start Tutorial](../quick-start/index.md)? * Does the [Data Service (rta-datasvc)](../../../services/rta-datasvc/README.md) have the same volume/folder mounted as used for the [Quick-Start Tutorial](../quick-start/index.md)? ### gRPC Using [grpcui](https://github.com/fullstorydev/grpcui/releases) from a terminal: ``` grpcui -plaintext localhost:2652 ``` If successful, this will open a web page where you can browse and interact with the [Session Service gRPC API](../../../services/rta-sessionsvc/grpc.md) — which will be useful later. Do this for all the services except the Gateway Service (which does not have a gRPC API): | Service | gRPC Port | API | |----------------|-----------|------------------------------------------------------| | rta-sessionsvc | 2652 | [gRPC API](../../../services/rta-sessionsvc/grpc.md) | | rta-configsvc | 2662 | [gRPC API](../../../services/rta-configsvc/grpc.md) | | rta-datasvc | 2672 | [gRPC API](../../../services/rta-datasvc/grpc.md) | ??? notes "Troubleshooting" * Make sure you ran `grpcui` from a command prompt / terminal * Is something else running on these ports? ## Step 4: Add Data Bindings to the Existing Sessions Using ATLAS, try to browse and load the session(s) you created in the [Quick-Start Tutorial](../quick-start/index.md). You should find that you can see the session and browse parameters to add to the displays... but no data appears. This is because we need to add [Data Bindings](../../sessions/data-bindings.md) so the [Gateway Service](../../../services/rta-gatewaysvc/README.md) can locate the data — since it's not all managed by a single process anymore. Let's start by doing an example by hand. Use [grpcui](https://github.com/fullstorydev/grpcui/releases) to open the [Session Service](../../../services/rta-sessionsvc/grpc.md): ``` grpcui -plaintext localhost:2652 ``` Use the `ListSessions` method to find a session, and note down an `identity` — which will be a GUID. ??? notes "Troubleshooting" If you don't see any sessions: * Check the [Session Service](../../../services/rta-sessionsvc/grpc.md) REST API by browsing to [http://localhost:2650/rta/v2/sessions/](http://localhost:2650/rta/v2/sessions/) * If you get a valid, but empty response (`#!json {"sessions":[]}`) the database is empty or you are talking to the wrong database * If you get an error or a blank page in your browser, check the `docker compose` logs in the terminal to see any errors * Use PgAdmin (or a similar front-end) to have a look at the database and verify: * Are you connecting to the right database? * Are you using the correct username/password? * Does it have any data in it? If you're sure you have all the details correct, perhaps there is a network problem between Docker and the database: * Are you using `host.docker.internal` if you are on Windows? * If your database is also running on Docker, have you [set up the bridge network](https://www.tutorialworks.com/container-networking/)? * Do you have a firewall on the host that could be blocking access? If you have got database connectivity but lost the sessions, just skip onto Step 5. Select the `CreateOrUpdateSession` method: * Set `identity` as noted above * Click the `[+]` next to `updates`, and select `set_data_bindings` * Click the `[+]` next to `data_bindings`: * Set the `key`: * `source` = `rta-datasvc` (to match the last line of the `docker-compose.yaml`) * `identity` = _as above_ * Scroll to the bottom and click _Invoke_ This should complete successfully: Response Data +----------------- | {} | Now try to load this session in ATLAS (make sure it's the right one). You should see the data you previously created. ??? notes "Troubleshooting" If you still don't see any data: * Make sure the volume for the [Config Service (rta-configsvc)](../../../services/rta-configsvc/README.md) is correct: On Windows it should look something like: ```yaml volumes: - C:\rta\configs:/data/rta-configs ``` The path on the left hand side must be an absolute path to a folder that already contains a file/files from the [Quick-Start Tutorial](../quick-start/index.md). * Make sure the volume for the [Data Service (rta-datasvc)](../../../services/rta-datasvc/README.md) is correct: On Windows it should look something like: ```yaml volumes: - C:\rta\data:/data/rta-data ``` The path on the left hand side must be an absolute path to a folder that already contains a file/files from the [Quick-Start Tutorial](../quick-start/index.md). If the volumes weren't setup correctly in the earlier tutorial, you might have written data into the Docker container itself instead of into a folder. You could fix this by running [Quick-Start Tutorial](../quick-start/index.md) Step 4 again after fixing the configuration, or just skip onto Step 5 in this tutorial. ## Step 5: Modify the Demo ### Update the gRPC channels In the `RTA.Examples.Loader` project, change this code: ```c# using var channel = GrpcChannel.ForAddress("http://localhost:8082"); var sessionClient = new SessionStore.SessionStoreClient(channel); var configClient = new ConfigStore.ConfigStoreClient(channel); var dataClient = new DataWriter.DataWriterClient(channel); ``` to this: ```c# using var sessionChannel = GrpcChannel.ForAddress("http://localhost:2652"); using var configChannel = GrpcChannel.ForAddress("http://localhost:2662"); using var dataChannel = GrpcChannel.ForAddress("http://localhost:2672"); var sessionClient = new SessionStore.SessionStoreClient(sessionChannel); var configClient = new ConfigStore.ConfigStoreClient(configChannel); var dataClient = new DataWriter.DataWriterClient(dataChannel); ``` ### Add Data Bindings As in Step 4, we need to add data bindings. Add a new `sessionIdentity` variable to make it clearly different to _data identity_: ```c# hl_lines="2 8 9" var dataIdentity = Guid.NewGuid().ToString(); var sessionIdentity = Guid.NewGuid().ToString(); //... await WriteSessionAsync( sessionClient, sessionIdentity, dataIdentity, timestamp, startNanos, durationNanos, configIdentifier); ``` Add some additional logic to `WriteSessionAsync` to create the data binding along with the session: ```c# hl_lines="3 4 12 41-57" private static async Task WriteSessionAsync( SessionStore.SessionStoreClient sessionClient, string sessionIdentity, string dataIdentity, DateTimeOffset timestamp, long startNanos, long durationNanos, string configIdentifier) { await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = sessionIdentity, CreateIfNotExists = new() { Identifier = $"PeriodicData Demo {timestamp:f}", Timestamp = timestamp.ToString("O") }, Updates = { new SessionUpdate { SetTimeRange = new() { StartTime = startNanos, EndTime = startNanos + durationNanos } }, new SessionUpdate { SetConfigBindings = new() { ConfigBindings = { new ConfigBinding { Identifier = configIdentifier } } } }, new SessionUpdate { SetDataBindings = new() { DataBindings = { new DataBinding { Key = new() { Source = "rta-datasvc", Identity = dataIdentity } } } } } } }); } ``` Run the project again from your IDE or the terminal: ``` bash dotnet run ``` This should create a new example session, but this time the data is stored in the [Data Service (rta-datasvc)](../../../services/rta-datasvc/README.md). ## Step 6: Test the Session Browse and load the session in ATLAS to make sure it's working. You should be able to browse and add parameters to displays, and see the data download. In the `docker compose` terminal, you will see activity as requests to the [Gateway Service (rta-gatewaysvc)](../../../services/rta-gatewaysvc/README.md) are proxied through to the other services. ## Next Steps [Review the changes](review.md) to understand how the modifications changed the demo. --- # Review !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Microservices — Review !!! info Complete the [Walkthrough](index.md) first. Like the [Quick-Start Tutorial](../quick-start/review.md), this demo project is based around a data loading pattern, where data is format-shifted for use with ATLAS. But in this example, we've split the services up, and we're nearly ready to connect to existing stores using a [Data Service as an adapter](../../../../../key-functionality/integrate/rta/introduction/data-services.md). ## Docker Compose and Container Orchestration Compared to the [Quick-Start Tutorial](../quick-start/review.md), you might be thinking: > There are too many processes! > Instead of one `rta-server`, there are four separate services to do the same thing! 😠 In this architecture, each service is doing one small job. This means each interface is simpler and the services are easier to replicate. This is particularly true for the config and data services, and is essentially an alternative to an in-process plugin architecture. For example, you could replace the toolkit [Data Service](../../../services/rta-datasvc/README.md) with an in-house data service built in Java to read a custom file format. Or use both services together — maybe even presenting the data as a single session. To make this manageable, you really need automation to configure, deploy, scale and restart the services if they fail or need upgrading. [Docker Compose](https://docs.docker.com/compose/) is one of the simpler ways to do this for a developer, but you'll want something more powerful in production. The difference is cluster management: a full [container orchestrator](https://www.redhat.com/en/topics/containers/what-is-container-orchestration) like [Kubernetes](https://kubernetes.io/) will handle all aspects of networking, scaling and fault management across a cluster of machines — though this comes with some complexity. If you are running on-premises, consider something simpler like [Docker Swarm](https://docs.docker.com/engine/swarm/) or [Nomad](https://www.nomadproject.io/). For small deployments, you can still use Docker Compose or run the executables directly — for example, as a set of Windows or Linux services. ## Deploying the Toolkit Microservices ### Initializing the PostgreSQL database This service is a simple database initializer: ```yaml hl_lines="5" rta-sessionsvc-init: image: mclarenapplied/rta-sessionsvc:latest environment: - RTA_PostgresConnectionString=Host=host.docker.internal; Username=postgres; Password=hunter2; entrypoint: ["/app/rta-sessionsvc", "--InitStoreAndExit=true"] restart: "no" ``` The `--InitStoreAndExit=true` instruction tells the service to initialize or upgrade the database and exit. !!! warning This pattern is not quite robust enough for production. Docker starts `rta-sessionsvc` immediately after `rta-sessionsvc-init`, so it is likely that the database will not be fully-initialized by the time this happens. In production, it would be better to have this happen as a scripted step that completes before rolling out the rest of the services. In this example, it _should_ be unneccessary (but harmless) if the [Quick-Start Tutorial](../quick-start/index.md) has been completed, since [RTA Server](../../../services/rta-server/README.md) will initialize the database on startup by default, as a convenience for developers. ### Mounting Volumes Mounting volumes ensures that the data is saved outside the container, so it's still there if you restart the stack. You should be able to see the files in the folders. There are two parts to a volume mount: ```yaml volumes: - C:\rta\data:/data/rta-data ``` * The left-hand side is the path to a folder on the host machine. It must be an absolute path. * The right-hand side is the path _inside_ the container. In this example, we also specify the path inside the container explicitly: ```yaml environment: - RTA_Store=File - RTA_FilePath=/data/rta-data ``` ### Gateway Service URIs The [Gateway Service](../../../services/rta-gatewaysvc/README.md) is configured to reference the other three services on the Docker network: ```yaml environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc="http://rta-datasvc:2670" ``` Communication between the services is REST-based. Each service essentially implements a section of the overall [REST API](../../../api/index.md) sections[^1]. The [Gateway Service](../../../services/rta-gatewaysvc/README.md) does not make any gRPC calls. The network name — e.g. `rta-sessionsvc` — is derived from the name of the service in the YAML file. Visualize `RTA_DataServiceUris` as having this structure: ```json { "RTA_DataServiceUris": { "rta-datasvc": "http://rta-datasvc:2670" } } ``` The nested property `rta-datasvc` is part of the [Data Binding](../../sessions/data-bindings.md) key. The value is the URI for the service hosting the data. !!! info `__` indicates the nested configuration: `parent__child` `:` can also be used on Windows only: `parent:child` This token can have any format — it doesn't need to be a hostname. To add another service, with source `mycustomsource`, located at `http://example.org:8030`: ```json hl_lines="4" { "RTA_DataServiceUris": { "rta-datasvc": "http://rta-datasvc:2670", "mycustomsource": "http://example.org:8030" } } ``` or in terms of environment variables: ```yaml hl_lines="5" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc="http://rta-datasvc:2670" - RTA_DataServiceUris__mycustomsource="http://example.org:8030" ``` The [Gateway Service](../../../services/rta-gatewaysvc/README.md) will retrieve the [Data Bindings](../../sessions/data-bindings.md) from the [Session Service](../../../services/rta-sessionsvc/README.md) to determine which service to talk to. ### Data Bindings [RTA Server](../../../services/rta-server/README.md) does not need [Data Bindings](../../sessions/data-bindings.md), because it handles reading and writing the data in a single process, and simply stores the data using the session identity. This tutorial adds a simple binding to tell the [Gateway Service](../../../services/rta-gatewaysvc/README.md) where the data is, but bindings offer other capabilities, including: * Selecting time-ranges to include in a session * Mixing channels/parameters from different sources * Applying a time-shift to the data [^1]: This is a slight over-simplification. For example, the data service interface is similar, but uses different URI routing paths. --- # Walkthrough !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # InfluxDB Data Adapter — Walkthrough This tutorial stores and serves data directly from [InfluxDB](https://www.influxdata.com/products/influxdb/), using the [microservices architecture](../../../../../key-functionality/integrate/rta/introduction/architectures.md#microservices) introduced in [Tutorial 02](../microservices/index.md). At the end, you'll have ATLAS connected to a popular off-the-shelf timeseries database. In this tutorial you: * Update the [Microservices demo](../microservices/index.md) to add [InfluxDB](https://www.influxdata.com/products/influxdb/) with the RTA [Influx Data Service](../../../services/rta-influxdatasvc/README.md) and supporting [Schema Mapping Service](../../../services/rta-schemamappingsvc/README.md) * Ingest some sample data into [InfluxDB](https://www.influxdata.com/products/influxdb/) * Call the other services using gRPC to describe sessions * Test the deployment with ATLAS !!! tip These code samples are in C# for .NET 5.0, and use Motion Applied [NuGet packages](../../../downloads.md#nuget-packages) to keep things as simple as possible. We recommend you follow this tutorial even if you are planning to do your integration in another language. The concepts should translate very easily once you have a working example. ## Prerequisites You'll want to complete the [Quick-Start](../quick-start/index.md) and [Microservices](../microservices/index.md) tutorials first. This should get you setup with everything you need. ## Step 1: Stop your existing Docker stack Hit ++ctrl+c++ to stop your Docker Compose stack from the [previous tutorial](../microservices/index.md). ## Step 2: Edit the Docker Compose deployment !!! info If you already have InfluxDB running in your development environment, you could skip this step and use the existing deployment when you configure the RTA [Influx Data Service](../../../services/rta-influxdatasvc/README.md). Watch out for port clashes if you run them side-by-side. Remember that you can [remap ports in Docker](https://docs.docker.com/config/containers/container-networking/). ### InfluxDB Modify the `docker-compose.yaml` to add [InfluxDB](https://hub.docker.com/_/influxdb) as an extra service: === "Windows" Create a directory for the `InfluxDB` volume. For example: ```powershell mkdir C:\rta\influxdb ``` Add [InfluxDB](https://hub.docker.com/_/influxdb) with the volume mount: ```yaml hl_lines="2-12" services: influxdb: image: influxdb:1.8 restart: always ports: - "8086:8086" volumes: - C:\rta\influxdb:/var/lib/influxdb environment: INFLUXDB_DB: "rtademo" INFLUXDB_ADMIN_USER: "admin" INFLUXDB_ADMIN_PASSWORD: "admin" # ... ``` === "Linux/Mac" Create a directory for the `InfluxDB` volume. For example: ```bash mkdir -p /tmp/rta/influxdb ``` Add [InfluxDB](https://hub.docker.com/_/influxdb) with the volume mount: ```yaml hl_lines="2-12" services: influxdb: image: influxdb:1.8 restart: always ports: - "8086:8086" volumes: - /tmp/rta/influxdb:/var/lib/influxdb environment: INFLUXDB_DB: "rtademo" INFLUXDB_ADMIN_USER: "admin" INFLUXDB_ADMIN_PASSWORD: "admin" # ... ``` ### Schema Mapping Service and Influx Data Service Modify the `docker-compose.yaml` to add these two extra services: === "Windows" Create a directory to hold files for the [Schema Mapping Service (rta-schemamappingsvc)](../../../services/rta-schemamappingsvc/README.md). For example: ```powershell mkdir C:\rta\schema-mappings ``` Add the [Schema Mapping Service (rta-schemamappingsvc)](https://hub.docker.com/r/mclarenapplied/rta-schemamappingsvc) and [Influx Data Service (rta-influxdatasvc)](https://hub.docker.com/r/mclarenapplied/rta-influxdatasvc) — which will be the data adapter for InfluxDB: ```yaml hl_lines="14-24 26-37" services: influxdb: image: influxdb:1.8 restart: always ports: - "8086:8086" volumes: - C:\rta\influxdb:/var/lib/influxdb environment: INFLUXDB_DB: "rtademo" INFLUXDB_ADMIN_USER: "admin" INFLUXDB_ADMIN_PASSWORD: "admin" rta-schemamappingsvc: image: mclarenapplied/rta-schemamappingsvc:latest restart: always ports: - "2680:2680" - "2682:2682" volumes: - C:\rta\schema-mappings:/data/rta-schema-mappings environment: - RTA_Store=File - RTA_FilePath=/data/rta-schema-mappings rta-influxdatasvc: image: mclarenapplied/rta-influxdatasvc:latest restart: always depends_on: - rta-schemamappingsvc - influxdb ports: - "2690:2690" environment: - RTA_DataSource=rta-influxdatasvc - RTA_SchemaMappingGrpcUri=http://rta-schemamappingsvc:2682/ - RTA_InfluxUri=http://influxdb:8086/ # ... ``` === "Linux/Mac" Create a directory to hold files for the [Schema Mapping Service (rta-schemamappingsvc)](../../../services/rta-schemamappingsvc/README.md). For example: ```bash mkdir -p /tmp/rta/schema-mappings ``` Add the [Schema Mapping Service (rta-schemamappingsvc)](https://hub.docker.com/r/mclarenapplied/rta-schemamappingsvc) and [Influx Data Service (rta-influxdatasvc)](https://hub.docker.com/r/mclarenapplied/rta-influxdatasvc) — which will be the data adapter for InfluxDB: ```yaml hl_lines="14-24 26-37" services: influxdb: image: influxdb:1.8 restart: always ports: - "8086:8086" volumes: - /tmp/rta/influxdb:/var/lib/influxdb environment: INFLUXDB_DB: "rtademo" INFLUXDB_ADMIN_USER: "admin" INFLUXDB_ADMIN_PASSWORD: "admin" rta-schemamappingsvc: image: mclarenapplied/rta-schemamappingsvc:latest restart: always ports: - "2680:2680" - "2682:2682" volumes: - /tmp/rta/schema-mappings:/data/rta-schema-mappings environment: - RTA_Store=File - RTA_FilePath=/data/rta-schema-mappings rta-influxdatasvc: image: mclarenapplied/rta-influxdatasvc:latest restart: always depends_on: - rta-schemamappingsvc - influxdb ports: - "2690:2690" environment: - RTA_DataSource=rta-influxdatasvc - RTA_SchemaMappingGrpcUri=http://rta-schemamappingsvc:2682/ - RTA_InfluxUri=http://influxdb:8086/ # ... ``` !!! note You should still have the other services: * `rta-sessionsvc` * `rta-configsvc` * `rta-datasvc` (optional for this tutorial) * `rta-gatewaysvc` ### Update Gateway Service Modify the [Gateway Service](../../../services/rta-gatewaysvc/README.md) environment to include the URI for the [Influx Data Service](../../../services/rta-influxdatasvc/README.md). This sets the data adapter up as a data source, which we will name `rta-influxdatasvc` to keep the naming consistent: ```yaml hl_lines="8 15" rta-gatewaysvc: image: mclarenapplied/rta-gatewaysvc:latest restart: always depends_on: - rta-sessionsvc - rta-configsvc - rta-datasvc - rta-influxdatasvc ports: - "8080:80" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc=http://rta-datasvc:2670 - RTA_DataServiceUris__rta-influxdatasvc=http://rta-influxdatasvc:2690 ``` !!! note The [Gateway Service](../../../services/rta-gatewaysvc/README.md) is configured to talk to the data adapter service (RTA [Influx Data Service](../../../services/rta-influxdatasvc/README.md)), not directly to [InfluxDB](https://www.influxdata.com/products/influxdb/). ### Start the Stack Start the services from a terminal in the folder where you saved the `docker-compose.yaml` file: docker compose up ## Step 3: Check the Services ### HTTP Using a web browser, make sure all these health endpoints are reporting `Healthy`: | Service | /health link | |------------------------------------------------|--------------------------------------------------------------| | [rta-gatewaysvc](http://localhost:8080/) | [http://localhost:8080/health](http://localhost:8080/health) | | [rta-sessionsvc](http://localhost:2650/) | [http://localhost:2650/health](http://localhost:2650/health) | | [rta-configsvc](http://localhost:2660/) | [http://localhost:2660/health](http://localhost:2660/health) | | [rta-datasvc](http://localhost:2670/) | [http://localhost:2670/health](http://localhost:2670/health) | | [rta-schemamappingsvc](http://localhost:2680/) | [http://localhost:2680/health](http://localhost:2680/health) | | [rta-influxdatasvc](http://localhost:2690/) | [http://localhost:2690/health](http://localhost:2690/health) | If a service says it is `Unhealthy`, the console log output should say why. ### gRPC Using [grpcui](https://github.com/fullstorydev/grpcui/releases) from a terminal: ``` grpcui -plaintext localhost:2682 ``` This will check that the [Schema Mapping Service gRPC API](../../../services/rta-schemamappingsvc/grpc.md) is available. ## Step 4: Run the Demo Open the tutorial solution in your development environment (e.g. Visual Studio Code). This time, you will be running `RTA.Examples.Influx`. From the terminal, navigate to this sub-directory, and then: ``` bash dotnet run ``` If the service has been setup correctly, this will load some sample data into InfluxDB and create an example session. You can create more sessions by running it again. ## Step 5: Test the Session Browse the sessions in ATLAS and identify the InfluxDB session. !!! info If you followed the previous tutorials, you should have some other sessions. The [Data Service](../../../services/rta-datasvc/README.md) is still running, so these other sessions should still load and display data. Load an InfluxDB session in ATLAS to make sure it's working. You should be able to browse and add parameters to displays, and see the data download. As before, in the `docker compose` terminal, you will see activity as requests to the [Gateway Service (rta-gatewaysvc)](../../../services/rta-gatewaysvc/README.md) are proxied through to the other services. ## Next Steps [Review the changes and the sample project](review.md) to understand how the services are working together. --- # Review !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # InfluxDB Data Adapter — Review !!! info Complete the [Walkthrough](index.md) first. This demo project illustrates the data adapter pattern, where data is served to ATLAS without copying and format-shifting. In this tutorial, the store is [InfluxDB](https://www.influxdata.com/products/influxdb/) — but since the data retrieval mechanics are quite simple, it could be any of a wide range of technologies. ## Sample Code Structure The sample project has four key steps: ```c# linenums="1" var dataIdentity = await WriteDataAsync(influxUri, sessionTag, startNanos, durationNanos, intervalNanos); await WriteSchemaMappingAsync(schemaMappingClient, dataIdentity); var configIdentifier = await WriteConfigAsync(configClient); await WriteSessionAsync(sessionClient, sessionIdentity, dataIdentity, timestamp, startNanos, durationNanos, configIdentifier); ``` These steps are: 1. Write data to [InfluxDB](https://www.influxdata.com/products/influxdb/); _This step is just providing example data, and really has nothing RTA-specific at all._ 2. Describe how the database fields will map onto [Timestamped Data requests](../../../api/index.md#operation/get-timestamped-data); _The data adapter ([Influx Data Service](../../../services/rta-influxdatasvc/README.md)) will use this to handle the requests._ 3. Describe the fields as [Configuration](../../configuration/index.md); _ATLAS will use this to show the user can see what parameters (fields) are available._ 4. Publish the [Session](../../sessions/index.md); _ATLAS will use this to provide data browsing and search._ ## Writing Data to InfluxDB The first stage is just creating sample data. The data adapter pattern only cares about _reading_ data: it does not consider how the data arrives in storage. In this demo, the sample project writes data using the [InfluxDB line protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/), but it could just as easily be [batch-loaded from a CSV file](https://www.influxdata.com/blog/how-to-write-points-from-csv-to-influxdb/), or [streamed from a broker](https://www.influxdata.com/blog/influxdb-and-kafka-how-companies-are-integrating-the-two/). ## Defining the Schema Mapping ATLAS does not know anything about how the data is stored. [Configuration](../../configuration/index.md) provides the client with a catalog of parameters and all the information it needs to make requests to the [REST API](../../../api/index.md#operation/get-timestamped-data) — but it doesn't contain any information about database tables and fields, or file paths and offsets. When the data adapter receives a request, it is expressed in terms of: * data identity * channels * time range (optional) The adapter must map this onto the data source schema to retrieve the data. [Schema Mappings](../../data/schema-mappings.md) provide a generalizable way to do this. === "C# Sample" In this demo, the sample code defines and publishes a schema mapping like this: ```c# var schemaMapping = new SchemaMapping { Properties = { ["dialect"] = "influxql", ["database"] = Database, ["measurement"] = Measurement }, FieldMappings = { Fields.Select((field, f) => new FieldMapping { SourceField = field, TargetChannel = (uint) f }) } }; await schemaMappingClient.PutSchemaMappingAsync(new PutSchemaMappingRequest { DataSource = DataBindingSource, DataIdentity = dataIdentity, SchemaMapping = schemaMapping }); ``` === "C# Sample — Unrolled" Here is the demo schema mapping with the `Fields`, `DataSource` and `DataIdentity` unrolled: ```c# var schemaMapping = new SchemaMapping { Properties = { ["dialect"] = "influxql", ["database"] = Database, ["measurement"] = Measurement }, FieldMappings = { new FieldMapping {SourceField = "alpha", TargetChannel = 0u}, new FieldMapping {SourceField = "beta", TargetChannel = 1u}, new FieldMapping {SourceField = "gamma", TargetChannel = 2u} } }; await schemaMappingClient.PutSchemaMappingAsync(new PutSchemaMappingRequest { DataSource = "rta-influxdatasvc", DataIdentity = "session=", SchemaMapping = schemaMapping }); ``` The _data source_ needs to match the [Gateway Service configuration](../../../services/rta-gatewaysvc/README.md#configuration) and the [Data Binding](../../sessions/data-bindings.md) written to the [Session Service](../../../services/rta-sessionsvc/README.md). The _data identity_ and schema mapping properties are specific to the data adapter. The [Influx Data Service README](../../../services/rta-influxdatasvc/README.md#publishing-schema-mappings) describes its requirements: * The _data identity_ must be a filter expression to select data that belongs to the session e.g. `car_reg='EA04 LFG' AND trip_id='cd15a409'` * The properties indicate: * `dialect` — because a future version will support [flux](https://docs.influxdata.com/influxdb/v2.0/query-data/) as well as [influxql](https://docs.influxdata.com/influxdb/v2.0/query-data/influxql/) * [`database`](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#database), [`retention_policy`](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#retention-policy-rp) (default "autogen") and [`measurement`](https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#measurement) (equivalent to a table) !!! warning "Security Note" Take care to avoid query injection vulnerabilities with this data identity. Un-escaped user-supplied data could allow attackers to manipulate the data retrieval query. ## Configuration and Session The sample project also publishes the [Configuration](../../configuration/index.md) and [Session](../../sessions/index.md) — which will look familiar if you completed the [Quick-Start Tutorial](../quick-start/index.md). ## Data Pipeline Integration Now we have an example where ATLAS can connect to existing data, it's worth thinking about data pipeline integration. The schema mapping, configuration and session could be published at any stage, including: * As the data is being written * At the end of each ingest job * From a script, to index existing data These are all very similar — but the first option enables live-streaming to ATLAS. To see a live-stream version of this InfluxDB setup, check out the [next tutorial](../live/index.md). --- # Walkthrough !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Live Streaming — Walkthrough This tutorial demonstrates live streaming to ATLAS, using [InfluxDB](https://www.influxdata.com/products/influxdb/), and building on the architecture from [Tutorial 03](../influx/index.md). At the end, you'll have ATLAS connected to a popular off-the-shelf timeseries database, showing live updates. In this tutorial you: * Update the [Influx Data Adapter demo](../influx/index.md) to add [Redis](https://redis.io/) and the RTA [Stream Service](../../../services/rta-streamsvc/README.md) * Generate and ingest data into [InfluxDB](https://www.influxdata.com/products/influxdb/) in real-time, while also publishing live updates [Redis](https://redis.io/) * Test the deployment with ATLAS !!! tip These code samples are in C# for .NET 5.0, and use Motion Applied [NuGet packages](../../../downloads.md#nuget-packages) to keep things as simple as possible. We recommend you follow this tutorial even if you are planning to do your integration in another language. The concepts should translate very easily once you have a working example. ## Prerequisites You'll want to complete the [Influx Data Adapter Tutorial](../influx/index.md) first. This should get you setup with everything you need. ## Step 1: Stop your existing Docker stack Hit ++ctrl+c++ to stop your Docker Compose stack from the [previous tutorial](../influx/index.md). ## Step 2: Edit the Docker Compose deployment Make sure you're working from the deployment for the [Influx Tutorial](../influx/index.md#edit-the-docker-compose-deployment). ### Redis Modify the `docker-compose.yaml` to add [Redis](https://hub.docker.com/_/redis) as an extra service: ```yaml hl_lines="2-6" services: redis: image: redis:6.2 restart: always ports: - "6379:6379" # ... ``` ### Stream Service Modify the `docker-compose.yaml` to add the [Stream Service](https://hub.docker.com/r/mclarenapplied/rta-streamsvc): ```yaml hl_lines="8-14" services: redis: image: redis:6.2 restart: always ports: - "6379:6379" rta-streamsvc: image: mclarenapplied/rta-streamsvc:latest restart: "always" ports: - "8180:80" environment: - RTA_RedisConnectionString=redis # ... ``` ### Start the Stack Start the services from a terminal in the folder where you saved the `docker-compose.yaml` file: ```bash docker compose up ``` ## Step 3: Check the Services ### HTTP Using a web browser, make sure all these health endpoints are reporting `Healthy`: | Service | /health link | |------------------------------------------------|------------------------------------------------------------------| | [rta-gatewaysvc](http://localhost:8080/) | [http://localhost:8080/health](http://localhost:8080/health) | | ==[rta-streamsvc](http://localhost:8180/)== | ==[http://localhost:8180/health](http://localhost:8180/health)== | | [rta-sessionsvc](http://localhost:2650/) | [http://localhost:2650/health](http://localhost:2650/health) | | [rta-configsvc](http://localhost:2660/) | [http://localhost:2660/health](http://localhost:2660/health) | | [rta-datasvc](http://localhost:2670/) | [http://localhost:2670/health](http://localhost:2670/health) | | [rta-schemamappingsvc](http://localhost:2680/) | [http://localhost:2680/health](http://localhost:2680/health) | | [rta-influxdatasvc](http://localhost:2690/) | [http://localhost:2690/health](http://localhost:2690/health) | If a service says it is `Unhealthy`, the console log output should say why. ### gRPC The [Stream Service](../../../services/rta-streamsvc/README.md) does not introduce any new gRPC interfaces to check. ## Step 4: Run the Demo Open the tutorial solution in your development environment (e.g. Visual Studio Code). This time, you will be running `RTA.Examples.InfluxLive`. From the terminal, navigate to this sub-directory, and then: ``` bash dotnet run ``` If the service has been setup correctly, this will live-stream data into InfluxDB and Redis, where it can be served to ATLAS. By default, the demo will create a series of five-minute sessions. ## Step 5: Setup ATLAS Live Connections ### Setup the live connection Open the ATLAS Session Browser. Right-click on **Sources** and select **Service Connections...** Select the service connection you previously defined, and click **Properties** to open a dialog. Fill in the **Web Socket URI**: `ws://localhost:8180/` Close the dialogs, and right-click **Sources** and select **Refresh**. Un-tick and re-tick the _localhost_ source to make sure it picks up the new settings, and press the **Refresh** button on the right-hand-side of the dialog. You should see your existing sessions, and a live session at the top of the list. ??? notes "Troubleshooting" If you don't see a live session: * Double-check the **Web Socket URI** is correct * Open the URI in a web browser the the 'http://' scheme and make sure you are communicating with the [Stream Service](../../../services/rta-streamsvc/README.md) * Restart ATLAS and try again * Check the ATLAS log file (**Tools** > **View Log Folder**), which may explain what the problem is If you _only_ see live data, with no back-fill to the start of the live session, this means that data is not loading from InfluxDB. ### Test the session Double-click to load a live session. Add a Waveform display, and press `P` to add parameters using the _Parameter Browser_. You should see waveform traces updating in real time. If you unload and reload the session, you should see that all the data is still visible. !!! info You may notice a small discontinuity where the data from the [Gateway Service](../../../services/rta-gatewaysvc/README.md) joins the data from the [Stream Service](../../../services/rta-streamsvc/README.md). This is a temporary limitation in the Stream Service implementation that will be fixed in a future release. Redis should contain enough streamed data to seamlessly cover this gap. You may also notice that the time-range in the displays extends back beyond the start of the session. This appears to be a time-zone issue, which will be looked at and fixed in a future development release. ## Next Steps [Review the changes and the sample project](review.md) to understand how the live stream is fed from the demo process to ATLAS. --- # Review !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Live Streaming — Review !!! info Complete the [Walkthrough](index.md) first. The demo project illustrates a pattern where live data is fed to ATLAS around the side of the main data store. The live stream is buffered through [Redis](https://redis.io/), and then selectively distributed to clients using [WebSockets](https://en.wikipedia.org/wiki/WebSocket). This means that clients are never directly interacting with the ingest process, and the write-latency of the data store becomes less important. ## Service Calls ### Call Ordering Users can't browse and load the session until it is published to the [Session Service](../../../services/rta-sessionsvc/README.md), so the service call ordering is different: === "This tutorial" 1. Describe the fields as [Configuration](../../configuration/index.md); 2. Describe how the database fields will map onto [Timestamped Data requests](../../../api/index.md#operation/get-timestamped-data); 3. Publish the [Session](../../sessions/index.md) in the `open` state; 4. Write data to [InfluxDB](https://www.influxdata.com/products/influxdb/); 5. Update the [Session](../../sessions/index.md) state to `closed`; _Other orderings are allowed — e.g. adding [Configuration](../../configuration/index.md) after the session has started._ === "Previous tutorial" 1. Write data to [InfluxDB](https://www.influxdata.com/products/influxdb/); 2. Describe how the database fields will map onto [Timestamped Data requests](../../../api/index.md#operation/get-timestamped-data); 3. Describe the fields as [Configuration](../../configuration/index.md); 4. Publish the [Session](../../sessions/index.md); ### Session State Transitions Publishing the session before the data is written requires greater attention to session state: * Start in the `open` state so the client is aware that the session is live; * Transition to: * `closed` when all data is written * `truncated` if the session was cut short (e.g. ++ctrl+c++) * `failed` if the session was cut short due to an exception !!! info Sessions can also start in the `waiting` state if there will be a preparation period before data is available. Always transition to `open` before data starts streaming. ## Streaming to InfluxDB Compared to the [previous tutorial](../influx/index.md), the writes to [InfluxDB](https://www.influxdata.com/products/influxdb/) need to be broken up. This ensures that data is flushed and visible in the database. In this example, this is done at the same time as [Session Model Updates](#session-model-updates). ## Streaming to Redis Live data is both written to [InfluxDB](https://www.influxdata.com/products/influxdb/) and buffered to [Redis Streams](https://redis.io/topics/streams-intro) — similar in design to [Apache Kafka](https://kafka.apache.org/). The [Redis](http://redis.io/) data flow follows a [simple protocol](../../live/redis.md), where the stream is a copy of all session model updates, and data: * Beginning with a [Session JSON](../../sessions/model.md#serializing-to-json) update * Buffering data so bursts are sent at around 10 Hz * Updating the session time-range often, and the session model if any metadata changes * Ending with a transition to the `closed`, `truncated` or `failed` state Each message is `StreamData`, but sent in bursts as `StreamDataBurst` to improve network efficiency. The streams are typically buffered in Redis long enough to cover the time taken for data to flush to the main store — which could be seconds or hours, depending on the storage technology. The Redis instance needs to be sized appropriately. The [Stream Service](../../../services/rta-streamsvc/README.md) includes an automatic trimmer to limit retention. ### Library Support [The protocol](../../live/redis.md) is implemented by the _MAT.OCS.RTA.StreamBuffer_ [NuGet Package](../../../downloads.md#nuget-packages). To connect to Redis: ```c# using var redisBuffer = new RedisStreamBuffer(hostname, db); var streamBuffer = await redisBuffer.InitStreamSessionAsync(sessionIdentity, streamName); ``` * `hostname` can include the port — e.g. `localhost:8389` — if the default Redis port (`6379`) is not used * `db` number should be `0` unless the Redis instance has multiple databases (essentially namespaces) * `streamName` collates messages into a named [Redis Streams](https://redis.io/topics/streams-intro) — a similar concept to a Kafka topic, which aids message trimming There is a single method to send the bursts: ```c# hl_lines="9" var burst = new StreamDataBurst { Data = { // ... } } await streamBuffer.WriteAsync(burst); ``` Serializing and sending the [Session Model](../../sessions/model.md#serializing-to-json) as JSON is a bit long-winded, so an extension method is provided: === "With extension method" ```c# await streamBuffer.WriteSessionJsonAsync(session); ``` === "Without extension method" ```c# using Newtonsoft.Json; using Newtonsoft.Json.Serialization; // setup Session Model conventions private static readonly JsonSerializerSettings JsonSettings = new JsonSerializerSettings { ContractResolver = new DefaultContractResolver { NamingStrategy = new CamelCaseNamingStrategy(false, false) }, DateParseHandling = DateParseHandling.DateTimeOffset }; // ... await streamBuffer.WriteAsync(new StreamDataBurst { Data = { new StreamData { SessionJson = JsonConvert.SerializeObject(session, JsonSettings) } } }); ``` ### Session Model Updates The [Session](../../sessions/index.md) needs to be available from both the REST API and the WebSocket. !!! important These session models _should_ be the same, but sometimes this is difficult: * The REST API might expose metadata that isn't available in the ETL pipeline * The pipeline might have separate processes writing to Redis and writing to storage, making coordination difficult The client compensates by using the session metadata that appears to be most recent based on time-range, and by copying any collections (e.g. [Configuration Bindings](../../sessions/model.md#configuration-bindings)) that are available from one source but undefined in another. The first and last messages streamed to Redis should be a Session JSON update. Further updates can be sent if any other metadata changes — e.g. adding configuration, or updating the identifier or details. The session model does _not_ need to be streamed to update the session time-range, as there is a dedicated operation for this high-frequency operation, as seen in #4 below. ### Buffering and Streaming Data Data arriving sample-by-sample needs to be buffered for efficient transfer. In this code sample, the data is arriving/generated in rows, so the code structure looks like this (excluding the InfluxDB writes): 1. Set up a timestamps buffer, and a values buffer for each field/column: ```c# var sampleCount = 0; var timestampsBuffer = new long[SamplesPerBurst]; var valuesBuffers = new double[Fields.Length][]; ``` 2. Push timestamps and values into the buffers: ```c# for (var f = 0; f < Fields.Length; f++) { valuesBuffers[f][sampleCount] = values[f]; } timestampsBuffer[sampleCount] = timestamp; sampleCount++; ``` 3. When the buffers are full, create a `StreamDataBurst`: ```c# if (sampleCount == SamplesPerBurst) { var burst = new StreamDataBurst(); for (var f = 0; f < valuesBuffers.Length; f++) { var tData = new TimestampedData { ChannelId = (uint)f, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(valuesBuffers[f].AsSpan(0, sampleCount))) }; tData.SetTimestamps(timestampsBuffer); burst.Data.Add(new StreamData { TimestampedData = tData }); } // ... } ``` 4. Include a time range update — typically in every burst: ```c# burst.Data.Add(new StreamData { TimeRange = new StreamTimeRange { StartTime = startTime, EndTime = endTime } }); ``` 5. Send the burst to Redis — ideally with overlapped I/O to minimise performance overhead: ```c# await lastBurst; lastBurst = streamBuffer.WriteAsync(burst); ``` --- # Walkthrough !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Creating a Custom Data Adapter — Walkthrough This tutorial demonstrates creating a Data Adapter service, to support a custom data source. At the end, you'll have ATLAS connected to the custom data adapter, via the [Gateway Service](../../../services/rta-gatewaysvc/README.md). In this tutorial you: * Run an "indexing" script to create [Sessions](../../sessions/index.md) and [Configuration](../../configuration/index.md) * Run a sample custom adapter service * Configure the [Gateway Service](../../../services/rta-gatewaysvc/README.md) to use this service * Test the sample data adapter with ATLAS !!! tip These code samples are in C# for .NET 5.0, and use Motion Applied [NuGet packages](../../../downloads.md#nuget-packages) to keep things as simple as possible. We recommend you follow this tutorial even if you are planning to do your integration in another language. The concepts should translate very easily once you have a working example. ## Prerequisites You can do this tutorial directly after the [Influx Data Adapter](../influx/index.md) or [Live Streaming](../live/index.md) tutorials. This will ensure that you have a Docker Compose project with all the required services. ## Step 1: Check your Docker Compose stack Make sure Docker Compose is running from the previous tutorial. Using a web browser, make sure all these health endpoints are reporting `Healthy`: | Service | /health link | |------------------------------------------------|------------------------------------------------------------------| | [rta-gatewaysvc](http://localhost:8080/) | [http://localhost:8080/health](http://localhost:8080/health) | | [rta-sessionsvc](http://localhost:2650/) | [http://localhost:2650/health](http://localhost:2650/health) | | [rta-configsvc](http://localhost:2660/) | [http://localhost:2660/health](http://localhost:2660/health) | | [rta-schemamappingsvc](http://localhost:2680/) | [http://localhost:2680/health](http://localhost:2680/health) | If a service says it is `Unhealthy`, the console log output should say why. ## Step 2: Generate and Index Custom Data Files Open the tutorial solution in your development environment (e.g. Visual Studio Code). This time, you will be running `RTA.Examples.DataAdapter.GeneratorIndexer`. From the terminal, navigate to this sub-directory, and then: ``` bash dotnet run ``` ``` Generating data (8 files) 000.sqlite 001.sqlite 002.sqlite 003.sqlite 004.sqlite 005.sqlite 006.sqlite 007.sqlite Indexing files 000.sqlite 001.sqlite 002.sqlite 003.sqlite 004.sqlite 005.sqlite 006.sqlite 007.sqlite ``` As the output suggests, this: 1. Generates sample [SQLite](https://www.sqlite.org/) files, containing: * 3x 100Hz samples (1 hour) * 3x 50Hz samples (1 hour) * a small amount of metadata 2. Indexes the sample data by: * publishing [Configuration](../../configuration/index.md) * publishing [Schema Mappings](../../data/schema-mappings.md) * publishing [Sessions](../../sessions/index.md) The files are written to the working directory: * When running from Visual Studio, this will be `RTA.Examples.DataAdapter.GeneratorIndexer\bin\Debug\net5.0\` * When running from the terminal, this will be the current directory !!! tip You may find it helpful to inspect the sample data using a [SQLite](https://www.sqlite.org/) database browser, such as: * [DB Browser for SQLite](https://sqlitebrowser.org/) * [SQLite Expert](http://www.sqliteexpert.com/) The code structure is very similar to the [InfluxDB Data Adapter](../influx/review.md#sample-code-structure). ## Step 3: Run the Demo Data Adapter Service Switch to running the `RTA.Examples.DataAdapter.Service`. From the terminal, navigate to this sub-directory, and then: ``` bash dotnet run -c Release ``` !!! tip Release mode can be significantly faster in some projects. When tuning a Data Adapter, make sure you are running with the Release configuration and without a debugger attached. !!! note The service will run on port 5000 by default. If you need to change this port, you'll also need to modify the port in the next step. ## Step 4: Edit the Docker Compose deployment ### Stop your existing Docker stack Hit ++ctrl+c++ to stop your Docker Compose stack. ### Update Gateway Service Modify the [Gateway Service](../../../services/rta-gatewaysvc/README.md) environment to include the URI for the demo Data Adapter Service. This sets the data adapter up as a data source on the host machine, named simply `sqlite` to match the data binding setup by the indexing process: === "Windows" ```yaml hl_lines="16" rta-gatewaysvc: image: mclarenapplied/rta-gatewaysvc:latest restart: always depends_on: - rta-sessionsvc - rta-configsvc - rta-datasvc - rta-influxdatasvc ports: - "8080:80" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc=http://rta-datasvc:2670 - RTA_DataServiceUris__rta-influxdatasvc=http://rta-influxdatasvc:2690 - RTA_DataServiceUris__sqlite=http://host.docker.internal:5000 ``` === "Linux/Mac" ```yaml hl_lines="16" rta-gatewaysvc: image: mclarenapplied/rta-gatewaysvc:latest restart: always depends_on: - rta-sessionsvc - rta-configsvc - rta-datasvc - rta-influxdatasvc ports: - "8080:80" environment: - RTA_SessionServiceUri=http://rta-sessionsvc:2650 - RTA_ConfigServiceUri=http://rta-configsvc:2660 - RTA_DataServiceUris__rta-datasvc=http://rta-datasvc:2670 - RTA_DataServiceUris__rta-influxdatasvc=http://rta-influxdatasvc:2690 - RTA_DataServiceUris__sqlite=http://localhost:5000 ``` ### Start the Stack Start the services from a terminal in the folder where you saved the `docker-compose.yaml` file: docker compose up ## Step 5: Check the Services ### HTTP Using a web browser, make sure all these health endpoints are reporting `Healthy`: | Service | /health link | |------------------------------------------------|------------------------------------------------------------------| | [rta-gatewaysvc](http://localhost:8080/) | [http://localhost:8080/health](http://localhost:8080/health) | | [rta-sessionsvc](http://localhost:2650/) | [http://localhost:2650/health](http://localhost:2650/health) | | [rta-configsvc](http://localhost:2660/) | [http://localhost:2660/health](http://localhost:2660/health) | | [rta-schemamappingsvc](http://localhost:2680/) | [http://localhost:2680/health](http://localhost:2680/health) | If a service says it is `Unhealthy`, the console log output should say why. !!! tip The [Gateway Service](http://localhost:8080/health) will be `Healthy` only if all the configured URIs are responding with `HTTP 200 OK`. This is a quick and easy sanity check that all your services — including the demo Data Adapter Service — are running and reachable. ## Step 6: Test the Sessions Browse the sessions in ATLAS and identify the SQLite sessions. Load a SQLite session in ATLAS to make sure it's working. You should be able to browse and add parameters to displays, and see the data download. As before, in the `docker compose` terminal, you will see activity as requests to the [Gateway Service (rta-gatewaysvc)](../../../services/rta-gatewaysvc/README.md) are proxied through to the other services. You should also see activity in the demo Data Adapter Service terminal. ## Next Steps [Review the sample projects](review.md) to understand how the Indexer and Data Adapter Service work. --- # Review !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Creating a Custom Data Adapter — Review !!! info Complete the [Walkthrough](index.md) first. This demo project is another example of the data adapter pattern, where data is served to ATLAS without copying and format-shiting. The initial [indexing step](index.md#step-2-generate-and-index-custom-data-files) is essentially the same for any data adapter. ## Generating Sample SQLite Files The data generator writes files to the working directory: * When running from Visual Studio, this will be the bin directory * When running from the terminal, this will be the current directory Each file contains three tables: * `properties` has the session timestamp, identifier, and details * `samples_100` has three channels of 100Hz data, with Unix timestamps * `samples_50` has three channels of 50Hz data, with Unix timestamps The timestamps are all at millisecond precision, with perfect spacing — so they can easily be represented with [Periodic Data](../../data/index.md#periodic-data). !!! tip You can open [SQLite](https://www.sqlite.org/) files using a database browser, such as: * [DB Browser for SQLite](https://sqlitebrowser.org/) * [SQLite Expert](http://www.sqliteexpert.com/) Each file contains 1 hour of data: | Field | Frequency | Samples | Running Total | |-----------|----------:|--------:|--------------:| | alpha | 100Hz | 360,000 | 360,000 | | beta | 100Hz | 360,000 | 720,000 | | gamma | 100Hz | 360,000 | 1,080,000 | | delta | 50Hz | 180,000 | 1,260,000 | | epsilon | 50Hz | 180,000 | 1,440,000 | | zeta | 50Hz | 180,000 | 1,620,000 | ## Indexing the SQLite Files The indexing process is completely independent of data generation. In this example, it could be taking place after an ETL pipeline run, or after detecting new files, or integrating ATLAS with existing data for the first time. ### Configuration The [Configuration](../../configuration/index.md) should look familiar from the other tutorials. Channels are described as Periodic, with an interval of 10ms or 20ms (100Hz and 50Hz). ### Schema Mappings [Schema Mappings](../../data/schema-mappings.md) contain the information needed by a Data Adapter to handle requests. In this case, each field has an extra property to map it onto one of the two tables. ??? example ```json { "properties": {}, "field_mappings": [ { "source_field": "alpha", "target_channel": 0, "properties": { "table": "samples_100" } }, { "source_field": "beta", "target_channel": 1, "properties": { "table": "samples_100" } }, { "source_field": "gamma", "target_channel": 2, "properties": { "table": "samples_100" } }, { "source_field": "delta", "target_channel": 3, "properties": { "table": "samples_50" } }, { "source_field": "epsilon", "target_channel": 4, "properties": { "table": "samples_50" } }, { "source_field": "zeta", "target_channel": 5, "properties": { "table": "samples_50" } } ], "event_mappings": [] } ``` ### Sessions Sessions are published as follows: * [`identity`](../../sessions/model.md#required-properties) is just the filename (e.g. `001.sqlite`) * [`identifier`](../../sessions/model.md#required-properties), [`timestamp`](../../sessions/model.md#required-properties) and [`details`](../../sessions/model.md#details-and-extended-details) are taken from the `properties` table in each SQLite file * [configuration binding](../../sessions/model.md#configuration-bindings) to the shared configuration * [data binding](../../sessions/data-bindings.md): * data source is `sqlite` to match the [Gateway Service](../../../services/rta-gatewaysvc/README.md) configuration * data identity is the full path to the file !!! important It's bad practise to expose internal details like file paths through the public API. The [Data Bindings](../../sessions/data-bindings.md) are not available via the [Gateway Service](../../../services/rta-gatewaysvc/README.md), and are treated as both trusted and private. ## Data Adapter Service This service is based on the **ASP.NET Core Web Application** template in Visual Studio 2019, using the McLaren _MAT.OCS.RTA.Services.AspNetCore_ [NuGet Package](../../../downloads.md#nuget-packages) to provide common functionality. This library is also used in all the [Toolkit Services](../../../services/index.md). The key modifications are: * [`Controllers\DemoDataController.cs`](#controller) — handles the incoming API requests * [`DemoSampleDataStore.cs`](#sample-data-store) — reads data from SQLite and returns it in Chunks * [`Startup.cs`](#startup) — ties it all together ### Controller The _MAT.OCS.RTA.Services.AspNetCore_ package provides a controller base class: `BaseDataServiceControllerV2`. This just needs sub-classing and annotating as shown: ```c# [Route("rta/v2")] [ApiController] public class DemoDataController : BaseDataServiceControllerV2 { public DemoDataController(IEventStore eventStore, ISampleDataStore sampleDataStore) : base(eventStore, sampleDataStore) { } } ``` !!! info There are two sets of Controller base classes. * `MAT.OCS.RTA.Services.AspNetCore.Controllers.API` provides controllers for the [RTA public API](../../../api/index.md), like the [Gateway Service](../../../services/rta-gatewaysvc/README.md). * `MAT.OCS.RTA.Services.AspNetCore.Controllers.Services` provides controllers for microservices like this data adapter. There are small differences. For example the request route for data requests: * RTA Public API: `/rta/v2/sessions/{sessionIdentity}/data/{type}/{channels}` * Data microservice: `/rta/v2/data/{dataIdentity}/{type}/{channels}` This emphasizes the difference between the _session identity_ — which is public — and the _data identity_ — which is private and might be bound to multiple sessions. The base controller class depends on services implementing `IEventStore` and `ISampleDataStore` (defined in the _MAT.OCS.RTA.Services_ package). ### Sample Data Store #### Interface The `ISampleDataStore` interface works with the _MAT.OCS.RTA.Services.AspNetCore_ to handle the [REST API](../../../api/index.md) responses. The library also provides a `DefaultSampleDataStore` with a no-op implementation of this interface, so the demo extends this class and adds just one method, handling [Periodic Data](../../data/index.md#periodic-data): ```c# public class DemoSampleDataStore : DefaultSampleDataStore { public override async Task GetPeriodicDataChunksAsync( string dataIdentity, long? startTime, long? endTime, ChannelRangeSet channels, RequestContext context, CancellationToken cancellationToken) { // ... return new ChunkedResult(/*...*/); } } ``` `ChunkedResult` wraps an `IEnumerable` or `IAsyncEnumerable` on more recent versions of .NET. The NuGet package provides formatters which recognise and handle this response automatically. #### Calling the Schema Mapping Service We'll need an instance of `SchemaMappingStoreClient` to retrieve the Schema Mapping. In ASP.NET Core, this is done by "injecting" the dependency in the constructor. The framework will instantiate `DemoSampleDataStore` and automatically pass in the client. ```c# hl_lines="4 6-9" public class DemoSampleDataStore : DefaultSampleDataStore { private readonly SchemaMappingStore.SchemaMappingStoreClient schemaMappingClient; public DemoSampleDataStore(SchemaMappingStore.SchemaMappingStoreClient schemaMappingClient) { this.schemaMappingClient = schemaMappingClient; } // ... } ``` Once injected, the first step in handling a request is to _query_ the [Schema Mapping Service](../../../services/rta-schemamappingsvc/grpc.md): ```c# SchemaMapping sm; try { sm = await schemaMappingClient.QuerySchemaMappingAsync( new QuerySchemaMappingRequest { DataSource = DataSource, DataIdentity = dataIdentity, SelectChannels = channels.ToString() }, new CallOptions(cancellationToken: cancellationToken)); } catch (RpcException ex) { if (ex.StatusCode == StatusCode.NotFound) return ChunkedResult.Empty(); throw; } ``` A query request (vs. _get_ request) filters the `SchemaMapping` using the specified [Channels](../../configuration/channels-parameters.md#channels), so the result is ready for use. In this demo, the data is split between two tables: * `samples_100` — 100Hz * `samples_50` — 50Hz Each `FieldMapping` has a `table` property, so we can apply a little more business logic to query each of the tables using only the relevant fields: ```c# schemaMapping.FieldMappings.Where(field => field.Properties["table"] == table) ``` These fields are then put together into a simple SQL query. #### Encoding Chunks Encoding [Chunks](../../data/chunks.md) involves: 1. Widening the time bounds to achieve [stable chunk boundaries](../../data/chunks.md#stable-chunks): ```c# var (startChunkTime, endChunkTime) = ChunkTime.FromSessionTimeRange(startTime, endTime); ``` 2. Buffering the [row-oriented database records into column-oriented chunk data](../../data/chunks.md#column-oriented-data): This is done in the `ChunkingBuffer` class, following a [time-based chunking strategy](../../data/chunks.md#time-based-chunking), in two steps. First, collating the samples into [`PeriodicData`](../../data/index.md#periodic-data) results: ```c# for (var i = 0; i < buffers.Length; i++) { var result = new PeriodicData { ChannelId = fields[i].TargetChannel, Samples = sampleCount, Interval = intervalNanos, StartTimestamp = bufferStartTime, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(buffers[i].AsSpan(0, sampleCount))) }; results[i].Add(result); } ``` Second, [encoding the results](../../data/chunks.md#using-the-api) into chunks: ```c# var chunks = new Chunk[fields.Count]; for (var i = 0; i < chunks.Length; i++) { // each Chunk contains a compressed PeriodicDataList var data = new PeriodicDataList { PeriodicData = { results[i] } }; // one periodic channel per chunk var chId = fields[i].TargetChannel; var chunkData = ChunkData.EncodePooled(ChunkDataMemoryPool.Shared, data, new[] {chId}); var chunk = new Chunk(unflushedStartTime, unflushedEndTime, chunkData); chunks[i] = chunk; results[i].Clear(); } ``` The steps are triggered based on the chunk [size limits](../../data/chunks.md#size-limits) and known maximum data frequency. ## Startup The `Startup.cs` script is where ASP.NET services handle configuration, and setup "services" which handle internal functionality. ### Schema Mapping Service Setting up the connection to the [Schema Mapping Service](../../../services/rta-schemamappingsvc/README.md): ```c# services.AddSingleton(_ => GrpcChannel.ForAddress("http://localhost:2682")); services.AddSingleton(); ``` ### Registering Services Registering [Sample Data Store](#sample-data-store): ```c# services.AddTransient(); services.AddTransient(); ``` Boiler-plate to enable RTA library support: ```c# services.AddRTA(); ``` ### Respond to `GET /` Add a response for a top-level GET request, so that the [Gateway Service](../../../services/rta-gatewaysvc/README.md) can see that the Data Adapter is reachable: ```c# hl_lines="4-9" app.UseEndpoints(endpoints => { endpoints.MapControllers(); endpoints.MapGet("/", async context => { context.Response.ContentType = "text/plain"; await context.Response.WriteAsync("RTA.Examples.DataAdapter.Service"); }); }); ``` ### Improvements Writing Data Adapter services in C# and ASP.NET Core shortens development using the RTA [NuGet packages](../../../downloads.md#nuget-packages) — and even without this advantage, ASP.NET Core is a mature, powerful, cross-platform framework, with a rich ecosystem. The package ecosystem offers some really useful packages to bring the service up to production standard: * Configure the service with the [ASP.NET Configuration](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-5.0) framework * Add `/health` endpoint with the [_AspNetCore.HealthChecks_](https://www.nuget.org/packages/AspNetCore.HealthChecks/) NuGet package * Add `/metrics` endpoint with the [_prometheus-net.AspNetCore_](https://www.nuget.org/packages/prometheus-net.AspNetCore/) NuGet package * Add structured logging with the [_Serilog.AspNetCore_](https://www.nuget.org/packages/Serilog.AspNetCore) NuGet package --- # Session Basics !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Session Basics _Sessions_ are a logical way to organise telemetry — like a file system. An RTA Session does not have to be a 1:1 match for a telemetry recording, and there is no prescribed file or database layout. An RTA service can present any bounded subset of telemetry as a session, with associated metadata and descriptors. === "JSON" The RTA API represents sessions in JSON, like this: ``` json { "sessions": [ { "identity": "f10f4a19-3831-4dd6-a178-faa94931a5b0", "state": "closed", "timestamp": "2021-05-06T18:58:56.20289+01:00", "startTimestamp": "2021-05-06T18:58:56.20289+01:00", "endTimestamp": "2021-05-06T19:08:56.20289+01:00", "timeRange": { "startTime": 1620323936202890000, "endTime": 1620324536202890000 }, "identifier": "Session Demo 06 May 2021 18:58", "details": { "Lab Tech": "Bob Jones", "Run": 17 }, "configBindings": [ { "identifier": "6d711dbc-5f0d-47d2-b127-8c9d20bcec01", "channelOffset": 0 } ] } ] } ``` Review the [Session Model](model.md) for more detail. === "C# Sample" The Toolkit [Session Service](../../services/rta-sessionsvc/grpc.md) provides an API to create and modify sessions using [gRPC](https://grpc.io/). This C# sample — using the _MAT.OCS.RTA.Toolkit.API.GrpcClients_ [NuGet Package](../../downloads.md#nuget-packages) — produces an equivalent result: ```c# // re-use the channel across service calls using var sessionChannel = GrpcChannel.ForAddress("http://localhost:2652"); var sessionClient = new SessionStore.SessionStoreClient(sessionChannel); await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "f10f4a19-3831-4dd6-a178-faa94931a5b0", CreateIfNotExists = new() { Identifier = "Session Demo 06 May 2021 18:58", Timestamp = "2021-05-06T18:58:56.20289+01:00", State = (int)SessionState.Closed }, Updates = { new SessionUpdate { SetTimeRange = new() { StartTime = 1620323936202890000, EndTime = 1620324536202890000 } }, new SessionUpdate { SetDetailsJson = new() { Json = new JObject { ["Lab Tech"] = "Bob Jones", ["Run"] = 17 }.ToString() } }, new SessionUpdate { SetConfigBindings = new() { ConfigBindings = { new ConfigBinding { Identifier = "6d711dbc-5f0d-47d2-b127-8c9d20bcec01" } } } } } }); ``` --- # Session Model !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Session Model This page describes all the properties on the RTA JSON Session model. This is the representation in the [OpenAPI specification](../../api/index.md) and consumed by ATLAS. ## Conventions * Dates are ISO 8601 with a timezone, and at the highest available precision * Session times are expressed as 64 bit signed integers, as an offset from the Unix Epoch, in nanoseconds * Properties can be omitted if they are `#!json null` or empty arrays/maps * Properties are _camelCase_ — e.g. `"timeRange"` instead of `"TimeRange"` * Enumerations are expressed as _camelCase_ string constants — e.g. `#!json "closed"` instead of `#!json "Closed"` ??? warning ".NET Pascal Case issues" Watch out for the _camelCase_ naming scheme when working with .NET — libraries tend to default to _PascalCase_. For the _Newtonsoft.Json_ library, set the serializer up like this: ``` C# new JsonSerializerSettings { ContractResolver = new DefaultContractResolver { // avoids mangling casing of dictionaries NamingStrategy = new CamelCaseNamingStrategy(false, false) }, DateFormatHandling = DateFormatHandling.IsoDateFormat, DateParseHandling = DateParseHandling.DateTimeOffset } ``` This also ensures that properties in `#!json "details"` and other dictionaries are not changed. ## Required Properties `#!json "identity": "f10f4a19-3831-4dd6-a178-faa94931a5b0"` : Unique identity — which can be any format that is convenient for the server. Good security practise for public identity strings suggests it should be something unpredictable, rather than a file path or database sequence number. `#!json "state": "closed"` : Indicates whether the session is expected to receive more data. ??? info "Valid States" | State | Numeric | Description | |----------------------|---------|--------------------------------------------------------------------------------------------------| | `#!json "unknown"` | 0 | Should be ignored by clients | | `#!json "waiting"` | 1 | Acquisition is starting, or temporarily paused | | `#!json "open"` | 2 | Receiving data, and there may be more data to follow even if the session appears idle | | `#!json "closed"` | 4 | No more data should be received, but metadata may be changed | | `#!json "truncated"` | 12 | _Closed_ without receiving all data | | `#!json "failed"` | 28 | _Closed_ without receiving all data as the result of a fault | | `#!json "abandoned"` | 44 | _Closed_ manually or by a watchdog after being left idle too long (e.g. due to an upstream crash)| The numeric values are part of the RTA specification. They are not used in JSON but are seen in the gRPC API. Values are chosen so that all closed states are >= `#!json "closed"` or >= 4 — and bitwise comparison is also supported. `#!json "timestamp": "2021-05-06T18:58:56.20289+01:00"` : Official session timestamp in ISO 8601, with time-zone offset if required — which ATLAS uses to cue session time rendering. This timestamp does _not_ have to match the actual data in the session: it could represent a time-tabled service, for example. `#!json "identifier": "Turbo Encabulator experiment 5 - Joe Bloggs"` : Human-readable name. This is quite similar to a filename — but is not required to be unique. Can be modified while the session is open and long after it has been closed. It is common practise to include some of the session details into the identifier for ease of reading. ## Time Range `#!json "startTimestamp": "2021-05-06T18:58:56.20289+01:00"` : Start of the telemetry (inclusive) in ISO 8601, or `null` if the session contains no data. This property (and `"endTimestamp"`) are present primarily to make sure that the session descriptor is fully usable from web applications, since the 64-bit nanosecond timestamps fall outside the numeric range reliably handled by double-precision variables. `#!json "endTimestamp": "2021-05-06T19:08:56.20289+01:00"` : End of the telemetry (inclusive) in ISO 8601, or `null` if the session contains no data. Must be present if `"startTimestamp"` is present. `#!json "timeRange": {"startTime": 1620323936202890000, "endTime": 1620324536202890000}` : Describes the valid time range (inclusive) in the session at nanosecond precision, measured from the Unix Epoch (1970-01-01T00:00:00Z) — and therefore always in UTC. Must be present when there is data in the session and be approximately the same as `"startTimestamp"` and `"endTimestamp"` after timezone correction. ## Details and Extended Details `#!json "details": {"Lab Tech": "Bob Jones", "Run": 17}` : Session metadata, which the client may use for filtering and display. Like the `"identifier"`, these details can be modified while a session is open and long after it has been closed. Can be a _string_, _number_, _integer_, _boolean_ or _dateTime_ — the latter being parsed from ISO 8601. Cannot contain any nesting or `null` values. `#!json "extDetails": {"Car Setup": {"rideHeightFront": 32, "rideHeightRear": 78}}` : Extended session metadata, which the client may use for filtering and display. Behaves like groups of `"details"`. Could be used to represent technical or large-scale metadata that users may not need to see by default. Extended details are still available for search and can be requested by the client, but are typically not shown to users by default — though this is entirely down to the server implementation. !!! info ATLAS does not currently exploit this feature, but may do so in future. ## Data Management `#!json "type": "DDS"` : Type classifier, indicating the origin or content of the session. `#!json "quality": 0.8` : Quality indicator in the range `0.0` - `1.0` (worst - best). Can indicate the measured or expected completeness of the data, especially when acquired from a lossy source. `#!json "group": "Aero"` : Enables the client to organize the session into groups amongst peers. Applicable when listed amongst `"alternates"` or `"children"`. If the session derived from a parent as a result of model processing, the group conventionally indicates the model (e.g. `#!json "aero"`) that was used, and the `"version"` indicates subsequent iterations of the model. ATLAS uses this grouping for visual clarity and as part of the loading heuristic when selecting `"children"` to load automatically with the parent — where it should select the latest `"version"` within each group by default. `#!json "version": "1.2.0"` : _SemVer_-style version number. The client displays this version number if specified and uses it with `"group"` as part of the loading heuristic for selecting `"children"` to load automatically. When loading `"children"`, ATLAS will automatically load the latest child session in each `"group"` (where specified). The latest session is determined by comparing `"version"` using _SemVer_ semantics, falling back to alphanumerical comparison. If no version number is present, a child session will always be loaded by default. ## Children and Alternates `#!json "children": []` : References the identities of child sessions, which may be loaded together with the parent session. These are usually derived from their parent as a result of model processing, but may also simply be additional optional data sets. ATLAS organizes these children using their `"group"` property as a label, and their `"version"` to indicate loading priority within each group, with the latest version (by _SemVer_) being preferred. The `"group"` is conventionally used to name the model (e.g. `#!json "aero"`) where children are the result of model processing, so that the `"version"` indicates subsequent iterations of the model. Each child session can itself have child sessions, forming a tree. `#!json "alternates": []` : References the identities of alternate sessions, which may be loaded instead of this session. Alternates may be at a different quality, or have different metadata, or even different content. ATLAS organizes these alternates using their `"group"` property as a label. They may have a different `"version"` but this is not required. !!! seealso "See also..." The RTA Session model presents sessions like a tree, with references from parent to child. During data acquisition, relationships are frequently captured the other way around, with a reference from child model to parent telemetry. Refer to [Relationships](relationships.md) to read about setting up relationships using the Toolkit [Session Service](../../services/rta-sessionsvc/README.md). ## Configuration Bindings `#!json "configBindings": [{"identifier": "foo", "channelOffset": 1000}]` : Binds one or more sets of [Configuration](../configuration/index.md) onto a session. Each binding has: `#!json "identifier": "foo"` : References a configuration that can be read from RTA. The same configuration can be published once and referenced across multiple sessions. `#!json "channelOffset": 1000` : Tells the client to apply an offset to every channel in the configuration descriptor (>= 0). For example, if the configuration specifies channel `#!json 50`, and the `"channelOffset"` is `#!json 1000`, then ATLAS will request data from the service using channel `#!json 1050`. This is done so that sessions can contain configuration from multiple independently-compiled "apps", and shared across sessions. This turns out to be important for [scalability](../../../../key-functionality/integrate/rta/introduction/configuration.md#designed-for-scale) when sessions have 10s of thousands of parameters. Many deployments will use a single configuration binding with no offset. !!! important It is important for channels to be numbered contiguously from `#!json 0` or `#!json 1` to enable this offset. If a server assigns channels randomly or using a hash, the channel namespace is effectively consumed and configurations cannot be stacked. Channel offsets are also used by ATLAS when loading child sessions. It is important that configurations are not modified after they are published. ATLAS assumes that configurations may be shared and cached for improved performance reduced database size. If there is any possibility that a configuration might be modified after publication, use a hash or a GUID for naming. ## Serializing to JSON Using the _Newtonsoft_ library and following the [conventions](#conventions) above: ```c# using Newtonsoft.Json; using Newtonsoft.Json.Serialization; // ... private static readonly JsonSerializerSettings JsonSettings = new JsonSerializerSettings { ContractResolver = new DefaultContractResolver { // avoids mangling dictionary keys NamingStrategy = new CamelCaseNamingStrategy(false, false) }, DateParseHandling = DateParseHandling.DateTimeOffset }; // ... var session = new Session("001", SessionState.Closed, DateTimeOffset.Now, "example"); var json = JsonConvert.SerializeObject(session, JsonSettings); ``` --- # Session Service !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Session Service Basics !!! tip Explore the gRPC API using [gRPC UI](../utilities.md#grpc-ui). This provides an intuitive web interface to issue API calls and see the responses. ## Opening a Channel The network connection to a server is called a _channel_ in most gRPC bindings. This should be reused across service calls where possible, for efficiency. The session store service is provided by the Toolkit [Session Service](../../services/rta-sessionsvc/grpc.md) (default port 2652) and [RTA Server](../../services/rta-server/grpc.md) (default port 8082). ```c# using var sessionChannel = GrpcChannel.ForAddress("http://localhost:2652"); var sessionClient = new SessionStore.SessionStoreClient(sessionChannel); ``` ## Creating/Updating a Session In this example, `CreateIfNotExists` defines the initial properties to set if the session does not already exist, along with `Updates` to be applied in the same transaction. If the session _does_ exist, only the `Updates` are applied. === "C# Sample" ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "001", CreateIfNotExists = new() { Identifier = "Demo Session", Timestamp = "2021-03-14T12:15:30Z", State = (int)SessionState.Closed } }); ``` === "JSON" ```json { "identity": "001", "state": "closed", "timestamp": "2021-03-14T12:15:30Z", "identifier": "Demo session" } ``` Each `SessionUpdate` can only do one operation: ???+ bug "Incorrect" In this example, only `SetQuality` is executed: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetType = "type66", SetQuality = 0.25 } } }); ``` ??? success "Correct" In this example, `SetType` and `SetQuality` are executed in a single transaction: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetType = "type66" }, new SessionUpdate { SetQuality = 0.25 } } }); ``` Updates to collections offer a consistent pattern of operations: * `Set...` — overwrites the collection * `Update...` — adds and updates items * `Delete...` — deletes the specified items For example, when defining [relationships](relationships.md), these two samples are equivalent: === "Using `Update` and `Delete` to modify data" ??? example "Starting with `"ref:aaa"`, `"ref:bbb"`, `"ref:ccc"`" ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetRefAnchors = new() { RefUris = { "ref:aaa", "ref:bbb", "ref:ccc" } } } } }); ``` Merge update using `Update` and `Delete`: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { UpdateRefAnchors = new() { RefUris = { "ref:ccc", "ref:ddd" } } }, new SessionUpdate { DeleteRefAnchors = new() { RefUris = { "ref:aaa" } } } } }); ``` Result: * `"ref:bbb"` * `"ref:ccc"` * `"ref:ddd"` !!! note Notice that the overlap with `"ref:ccc"` is allowed. This is similar to JSON patching. === "Using `Set` to overwrite" ??? example "Starting with `"ref:aaa"`, `"ref:bbb"`, `"ref:ccc"`" ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetRefAnchors = new() { RefUris = { "ref:aaa", "ref:bbb", "ref:ccc" } } } } }); ``` Overwriting with `Set`: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetRefAnchors = new() { RefUris = { "ref:bbb", "ref:ccc", "ref:ddd" } } } } }); ``` Result: * `"ref:bbb"` * `"ref:ccc"` * `"ref:ddd"` !!! note Notice that `"ref:aaa"` is removed from the list. ## Fetching and Listing Sessions The [Session Service API](../../services/rta-sessionsvc/grpc.md) provides the same [query capabilities](queries.md) as the REST API, but with an extended data model covering [Relationships](relationships.md), [Folders](folders.md) and [Data Bindings](data-bindings.md). These elements are opt-in through the `SessionElements` mask: ```c# var session = await sessionClient.GetSessionAsync(new GetSessionRequest { Identity = "example", Elements = new() { Json = true, Refs = true, Folders = true, DataBindings = true } }); ``` !!! tip The _MAT.OCS.RTA.Toolkit.API.GrpcClients_ [NuGet package](../../downloads.md#nuget-packages) provides a convenience method to retrieve and parse the JSON Session Model: ```c# var sessionModel = await sessionClient.GetSessionJsonAsync("example"); ``` Listing sessions works the same way, using the same [query](queries.md) dialect as the REST API: ```c# var list = await sessionClient.ListSessionsAsync(new ListSessionsRequest { Query = "{\"identifier\": \"Example\"}", SessionElements = new() { Json = true, DataBindings = true } }); ``` ## Sequenced Updates The `CreateOrUpdateSessionRequest` has an optional `sequence` field, which can help avoid re-applying stale session updates when working with a checkpoint-based system like [Apache Kafka](https://kafka.apache.org/). If populated, the service will apply the update only if the specified sequence number is greater than previous sequence numbers. As the sequence monotonically increases — for example, using a Kafka message offset — it is safe to replay a sequence of updates without causing the session metadata to roll back. ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Sequence = 100, Updates = { new SessionUpdate { SetType = "aaa" } } }); await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Sequence = 120, Updates = { new SessionUpdate { SetType = "bbb" } } }); ``` ## Deleting Sessions Multiple sessions can be deleted in one operation, and deletion completes without error even if a session does not exist: ```c# await sessionClient.DeleteSessionsAsync(new DeleteSessionsRequest { Identities = { "example", "xxx-does-not-exist-xxx" } }); ``` ## An Example of Creating Sample Sessions This section presents an example of how to create few records of RTA sessions in the Postgres DB by utilizing the `OCS.RTA.Toolkit` which write the sample data into the DB and associated 'data chuncks' files. The data in question are RTA sessions and its Laps (called 'markers' here) are linked to each sessions. They are handled by using `SessionStore` service. It also creates a session configuration data, which is stored in a Json file. This is handled by `ConfigStore` service. Finally, it generates sample 'data chuncks' files to simulates the data which represents session parameters. This is handeld by the `DataWriter`. ```c# public static async Task Main(string[] args) { private static readonly string[] Fields = {"alpha", "beta", "gamma"}; using var channel = GrpcChannel.ForAddress("http://localhost:8082"); var sessionClient = new SessionStore.SessionStoreClient(channel); var configClient = new ConfigStore.ConfigStoreClient(channel); var dataClient = new DataWriter.DataWriterClient(channel); var timestamp = DateTimeOffset.Now; var startNanos = (timestamp.ToUniversalTime() - DateTimeOffset.UnixEpoch).Ticks * 100; var durationNanos = TimeSpan.FromMinutes(10).Ticks * 100; var intervalNanos = TimeSpan.FromMilliseconds(10).Ticks * 100; var configIdentifier = await WriteConfigAsync(configClient, intervalNanos); // At this point we create a set of sample sessions: // 1) creates 9 sessions (where '9' is an arbitrary constant, good enough for testing purpose); // 2) creates 'data chuncks' files assiciated with each session; // 3) and links the lap/marker sample data for each session (this is done inside of WriteSessionAsync() call) // const int sessionInTotal = 9; for (int sessionNumber = 1; sessionNumber <= sessionInTotal; sessionNumber++) { var dataIdentity = Guid.NewGuid().ToString(); await WriteDataAsync(dataClient, dataIdentity, startNanos, durationNanos, intervalNanos); await WriteSessionAsync(sessionClient, dataIdentity, timestamp, startNanos, durationNanos, intervalNanos, configIdentifier, sessionNumber); Console.WriteLine(dataIdentity); } } private static async Task WriteDataAsync( DataWriter.DataWriterClient dataClient, string dataIdentity, long startNanos, long durationNanos, long intervalNanos) { var rng = new Random(); var signals = new SignalGenerator[Fields.Length]; for (var f = 0; f < signals.Length; f++) { signals[f] = new SignalGenerator(rng); } const int burstLength = 100; var burstSamples = new short[burstLength]; using var dataStream = dataClient.WriteDataStream(); var requestStream = dataStream.RequestStream; var responseStream = dataStream.ResponseStream; await requestStream.WriteAsync (new WriteDataStreamMessage { DataIdentity = dataIdentity } ); for (var offset = 0L; offset < durationNanos; offset += burstLength * intervalNanos) { for (var f = 0; f < signals.Length; f++) { var signal = signals[f]; for (var t = 0; t < burstLength; t++) { var timeOffsetNanos = offset + t * intervalNanos; burstSamples[t] = (short)signal[timeOffsetNanos]; // short precision for this demo } var burst = new PeriodicData { ChannelId = (uint)f, StartTimestamp = startNanos + offset, Interval = intervalNanos, Samples = burstLength, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(burstSamples.AsSpan())) }; await requestStream.WriteAsync (new WriteDataStreamMessage { PeriodicData = burst } ); } if ((offset % 1_000_000_000L) == 0) { Console.Write("."); } } // 1. complete request stream await requestStream.CompleteAsync(); // 2. drain response stream (can also be done in a background task, or interleaved) // (no responses expected since no flush tokens were sent) await foreach (var _ in responseStream.ReadAllAsync()); Console.WriteLine(); } private static async Task WriteConfigAsync( ConfigStore.ConfigStoreClient configClient, long intervalNanos) { var channels = new ChannelBuilder[Fields.Length]; var parameters = new ParameterBuilder[Fields.Length]; for (var f = 0; f < Fields.Length; f++) { channels[f] = new ChannelBuilder((uint)f, intervalNanos, DataType.Signed16Bit, ChannelDataSource.Periodic); parameters[f] = new ParameterBuilder($"{Fields[f]}:demo", Fields[f], $"{Fields[f]} field") { ChannelIds = {(uint)f}, MinimumValue = -1500, MaximumValue = +1500 }; } var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = parameters, Channels = channels } } }.BuildConfiguration(); // arbitrary, unique and can be reused as long as the config stays the same const string configIdentifier = "6D711DBC-5F0D-47D2-B127-8C9D20BCEC02"; await configClient.PutConfigAsync(configIdentifier, config, MediaTypes.JsonConfig); return configIdentifier; } private static async Task WriteSessionAsync( SessionStore.SessionStoreClient sessionClient, string dataIdentity, DateTimeOffset timestamp, long startNanos, long durationNanos, long intervalNanos, string configIdentifier, int sessionNumber) { var lapChunk = durationNanos / (sessionNumber + 1); // 1e9 * 60 = 60000000000 // This part implements how to add Laps/Markers to each session. // In this change: // -- each session has an increasing number of markers, i.e. 2, 3, 4, 5, ... etc. // -- each lap/marker has different id and label values correspondingly. // -- the StartTime and EndTime of each marker is calculated in the way that the laps are evenly divided for the whole timeline // (again, this is merely for 'facilitating' testing in A10, when a session is loaded) // -- the markers firstly are generated as a list of them, and then the list is added to the session in question. // var newMarkerList = new MarkersList(); for (int i = 1; i <= sessionNumber; i++) { var m1 = new Marker { Id = i.ToString(), Type = "Test", Label = "Test Label" + i.ToString(), StartTime = startNanos + (i - 1) * lapChunk, EndTime = startNanos + i * lapChunk, DetailsJson = "" }; newMarkerList.Markers.Add(m1); } var m2 = new Marker { Id = (sessionNumber + 1).ToString(), Type = "Test", Label = "Test Label" + (sessionNumber + 1).ToString(), StartTime = startNanos + sessionNumber * lapChunk, EndTime = startNanos + (sessionNumber + 1) * lapChunk, DetailsJson = "" }; newMarkerList.Markers.Add(m2); await sessionClient.CreateOrUpdateSessionAsync ( new CreateOrUpdateSessionRequest { Identity = dataIdentity, CreateIfNotExists = new() { Identifier = $"Session {sessionNumber} for Data Service Demo {timestamp:f}", Timestamp = timestamp.ToString("O") }, Updates = { new SessionUpdate { SetType = "rta" }, new SessionUpdate { // inclusive timestamps SetTimeRange = new() { StartTime = startNanos, EndTime = startNanos + durationNanos - intervalNanos } }, new SessionUpdate { SetConfigBindings = new() { ConfigBindings = { new ConfigBinding { Identifier = configIdentifier } } } }, new SessionUpdate { UpdateMarkers = new SessionUpdate.Types.MarkersMap { MarkersByCategory = { ["Laps"] = newMarkerList } } }, new SessionUpdate { UpdateDetailsJson = new DetailsJson { Json = "{\"Number\":" + sessionNumber+"}" } } }, } ); } ``` ## How to Work with PostrgreSQL DB Data When a sample sessions set is created [see an example above](#an-example-of-creating-sample-sessions), it is easy to run a simple query in PostrgreSQL DB to see and check whether the data created are as expected: ```SQL SELECT S.identity, S.state, S.tstamp, S.tz_offset, S.start_time, S.end_time, S.identifier, S.type_attr, S.quality_attr, S.group_attr, S.version_attr, S.details, S.schema_version, S.schema_metadata, S.heartbeat_utc, S.sequence, SM.session_identity, SM.id, SM.category, SM.type, SM.label, SM.description, SM.start_time, SM.end_time, SM.details FROM rta_sessions.session AS S JOIN rta_sessions.session_marker AS SM ON S.identity = SM.session_identity WHERE S.identity = '35811dba-0665-4ce1-a6bd-76337fecf5f7'; ``` The query should return some data as shown below: At this point you can make any chages to the data in DB tables by using Postgres 'Edit' capabilities. --- # Relationships !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Relationships ## Children Child sessions[^1] are additional data sets that can be loaded together with a parent session. A common use-case is where telemetry is processed through a simulation — for example, a vehicle dynamics model — to produce simulated data. The resulting session is a logical child and extension of the parent, so it makes sense to load both at the same time. parent | +-- child If multiple models are used, then there should be multiple child sessions to load. The `"group"` property discriminates between the models: parent | +-- dynamics | | | +-- child | +-- thermal | +-- child But the models may be updated or reconfigured, so it is common to produce more than one `"version"` of child sessions: parent | +-- dynamics | | | +-- child v1.1 * | +-- child v1.0 | +-- thermal | +-- child v2.0 * +-- child v1.1 +-- child v1.0 In this case, ATLAS will load the most recent version in each group by default (`*`). The user can choose to load different versions or skip some of the child sessions entirely. The tree can be arbitrary depth — e.g. bringing together independent sessions from multiple sub-systems, and their models: parent | +-- Sub-system A | | | +-- dynamics | | | | | +-- child v1.1 * | | | +-- thermal | | | +-- child v2.0 * | +-- Sub-system B | +-- dynamics | | | +-- child v1.1 * | +-- thermal | +-- child v2.0 * !!! tip "Tip: Combine sessions using Data Bindings" To simplify a complex scenario like this, consider presenting the sub-systems as a single unified session. You can do this in the [Session Service](../../services/rta-sessionsvc/README.md) with [Data Bindings](data-bindings.md). ## Alternates You can also define alternate versions of sessions, which could vary in: * quality — e.g. RF telemetry vs. download * version — e.g. original vs. post-processed * duration or time-range * parameters * metadata * etc. For example: parent | +-- alternate parent This has no special semantics — it is simply another session offered to the user as an alternate to the parent. Alternates can be used in combination with child sessions: parent | +-- alternate parent | | | +-- child | +-- child ATLAS uses the `"group"` property — if specified — to organize these sessions, but does not load them automatically. ## Creating Relationships using the Session Service Dynamically creating relationships between parents and children can be painful if the sessions are recorded independently. The two common issues are: * Ordering — a child session may arrive before or after the parent; * Keys — it can be unclear how to associate sessions; The [Session Service](../../services/rta-sessionsvc/README.md) handles this by exposing "anchors" — which are essentially natural keys expressed as a URI — and references to those anchors to create the relationships. This means that a session is not dependent on creating relationships using a key generated at the time of recording, and the sessions can arrive in any order. For example: Telemetry Session +- ref_anchor = guid:8319B646-D262-47BF-8BD9-85AECC314C50 +- ref_anchor = acq:turbo-encabulator-5/2021-05-20T08:44:21+0000 Downsampled Session +- alternate_of_ref = guid:8319B646-D262-47BF-8BD9-85AECC314C50 Model Session +- child_of_ref = acq:turbo-encabulator-5/2021-05-20T08:44:21+0000 This results in: Telemetry Session | +-- | | | +-- Downsampled Session | +-- | +-- Model Session To create these relationships using the [gRPC API](../../services/rta-sessionsvc/grpc.md): === "C# Sample" Creating each of the three sessions (in any order): ```c# using var sessionChannel = GrpcChannel.ForAddress("http://localhost:2652"); var sessionClient = new SessionStore.SessionStoreClient(sessionChannel); await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "telemetry_session", CreateIfNotExists = new() { Identifier = "Telemetry Session", Timestamp = DateTimeOffset.Now.ToString("O") }, Updates = { new SessionUpdate { SetRefAnchors = new() { RefUris = { "guid:8319B646-D262-47BF-8BD9-85AECC314C50", "acq:turbo-encabulator-5/2021-05-20T08:44:21+0000" } } } } }); await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "downsampled_session", CreateIfNotExists = new() { Identifier = "Downsampled Session", Timestamp = DateTimeOffset.Now.ToString("O") }, Updates = { new SessionUpdate { SetAlternateOfRefs = new() { RefUris = { "guid:8319B646-D262-47BF-8BD9-85AECC314C50" } } } } }); await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "model_session", CreateIfNotExists = new() { Identifier = "Model Session", Timestamp = DateTimeOffset.Now.ToString("O") }, Updates = { new SessionUpdate { SetChildOfRefs = new() { RefUris = { "acq:turbo-encabulator-5/2021-05-20T08:44:21+0000" } } } } }); ``` Additional anchors and references can be added or removed at any time. === "JSON API Response" Listing sessions from the REST API: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions"` ```json { "sessions": [ { "identity": "telemetry_session", "state": "unknown", "timestamp": "2021-05-20T10:48:10.072433+01:00", "identifier": "Telemetry Session", "alternates": [ "downsampled_session" ], "children": [ "model_session" ] } ] } ``` !!! note Notice that the list only returns the top-level session, with references to alternates and children. This is the correct behaviour and the only option for the REST API. The Session Service gRPC API does offer an `include_sub_sessions` option to list all sessions. The client lazily resolves the relationships by requesting the child or alternate identities: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions/downsampled_session"` ```json { "identity": "downsampled_session", "state": "unknown", "timestamp": "2021-05-20T10:48:10.562559+01:00", "identifier": "Downsampled Session", "alternates": [], "children": [] } ``` [^1]: Sometimes called "associated sessions" --- # Listing and Queries !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Listing and Queries The RTA API provides paginated session listing, with sorting and filtering. This functionality is combined with the [Session Metamodel](metamodel.md) to drive the ATLAS session browser and provide rich functionality for web applications and integrations in the environment. To request a basic sessions list: === "GET" ```bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` === "POST" ```bash curl -X POST -d "" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` ```json { "sessions": [ { "identity": "example", "state": "closed", "timestamp": "2021-05-22T14:30:08.244956+01:00", "identifier": "Example", "details": {} } // ... ] } ``` This returns a page of sessions in descending order by `"timestamp"` (i.e. most-recent first) by default. ???+ info "GET or POST?" POST requests are required for `query` and `prop` to avoid issues with excessive URI lengths. | Argument | GET | POST | |-------------|-----|------| | `pageSize` | ✔️ | ✔️ | | `pageIndex` | ✔️ | ✔️ | | `folder` | ✔️ | ✔️ | | `sort` | ✔️ | ✔️ | | `query` | ❌ | ✔️ | | `prop` | ❌ | ✔️ | ## Pagination `pageSize` specifies the page size, in sessions. The default size is implementation-dependent. `pageIndex` specifies the 0-based page number. === "GET" ```bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?pageSize=50&pageIndex=0" ``` === "POST" ```bash curl -X POST -d "pageSize=50&pageIndex=0" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` ## Folders To limit results to sessions in a [virtual folder](folders.md) subtree (e.g. `#!json "animals/felines"`): === "GET" ```bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?folder=animals%2Ffelines" ``` === "POST" ```bash curl -X POST -d "folder=animals%2Ffelines" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` ## Sorting The `sort` argument requests a sort, using a property key with `:asc` or `:desc` suffix: * `timestamp:asc` * `quality:desc` * `details.Engineer:asc` * `extDetails.CarSetup.TyrePressure:asc` For example: === "GET" ```bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?sort=timestamp:asc&quality:desc" ``` === "POST" ```bash curl -X POST -d "sort=timestamp:asc&quality:desc" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` Multiple sorts can be specified as shown above, from highest priority to lowest. The Toolkit [Session Service](../../services/rta-sessionsvc/README.md) supports multi-property sorts, but other implementations may not. ??? important "Specification Guidance" * Implementations MUST sort in descending order by timestamp, by default * Implementations SHOULD support user-specified sorting — it is expected by users * Implementations MAY support sorting on multiple properties, from highest- to lowest-priority * Sorting SHOULD use the appropriate data type — i.e. not simply string comparison * Sorting on date-time fields (specifically including `"timestamp"`, `"startTimestamp"` and `"endTimestamp"`) SHOULD handle the time-zone correctly — which in practice tends to require storing the timezone offset separately * Sorting support MUST match the metamodel, if one is provided * Sort requests for fields that do not exist or are declared as non-sortable in the metamodel MUST be accepted gracefully — do not assume the client has checked the metamodel ## Query The ATLAS 10 session browser can construct a query from the free-text search field (top), and from structured filters (lower-left panel) — which are populated using the [metamodel](metamodel.md). A typical query using these fields might translate to: ```sql (identifier CONTAINS '-20200313-') AND ((details.driver = 'SAI') OR (details.driver = 'RIC')) ``` Properties in the session are referenced in typical JSON-path style (e.g. `"identifier"` or `"details.foo"`). The RTA query dialect (defined by [JSON Schema](../../api/query.schema.json)) is essentially a small subset of the [MongoDB Query Syntax](https://docs.mongodb.com/manual/reference/operator/query/). This is sufficient to meet all current and likely future requirements for basic session browsing and search, and is easy to parse and transpile to other query languages, such as SQL. ```json { "identifier": {"$contains":"-20200313-"}, "details.driver": {"$in": ["SAI", "RIC"]} } ``` === "POST" ```bash curl -X POST -d "query=%7B%22identifier%22%3A%7B%22%24contains%22%3A%22-20200313-%22%7D%2C%22details.driver%22%3A%7B%22%24in%22%3A%5B%22SAI%22%2C%22RIC%22%5D%7D%7D" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` ??? important "Specification Guidance" * The query language is specified by a [JSON schema](../../api/query.schema.json) * Library support is provided in the _MAT.OCS.RTA.Model_ [NuGet Package](../../downloads.md#nuget-packages) * ATLAS does not currently use all aspects of this dialect, but it MUST be supported in its entirety * Queries SHOULD be supported on as many session model properties as possible — including `"details"` ### Conditions #### Comparisons The simplest condition is a key-value pair representing an equality: ```json {"type": "aaa"} ``` This can also be encoded with a relational comparison operator: ```json {"type": {"$eq": "aaa"}} ``` | Operator | Meaning | |----------|--------------------------| | $eq | Equal to | | $gt | Greater than | | $gte | Greater than or equal to | | $lt | Less than | | $lte | Less than or equal to | | $neq | Not equal to | These work on all the data types. `null` comparisons are allowed but should be limited to `$eq` and `$neq`. #### Text Matches Text matches have the same structure, but only work on strings: ```json {"identifier": {"$contains": " 2021-"}} ``` | Operator | Meaning | |-------------|-------------| | $startsWith | Starts with | | $endsWith | Ends with | | $contains | Contains | !!! note The text match operators diverge from MongoDB syntax. There is no regular expression operator. #### Aggregation Aggregate operators `$in` and `$nin` accept arrays of values: ```json {"details.driver": {"$in": ["SAI", "NOR"]}} ``` | Operator | Meaning | |-------------|-------------| | $in | In list | | $nin | Not in list | #### Negation There is also a negation operator — `$not` — which operates on filter conditions. ```json {"$not": {"type": "test"}} ``` ### Criteria Conditions can be assembled into a list of criteria. A single equality comparison is the simplest case, but multiple equality matches are implicitly a logical AND: ```json {"type": "foo", "details.thang": "bar"} ``` This can also be written with an explicit logical operator, operating on a list of criteria: ```json {"$and": [{"type": "foo"}, {"details.thang": "bar"}]} ``` | Operator | Meaning | |-------------|---------------------------------| | $and | All of the criteria must match | | $or | Any of the criteria must match | | $nor | None of the crtieria must match | ## Extra Properties The service may have an extended data model offering more properties beyond those provided by default. Additional properties can be discovered by [querying the metamodel](metamodel.md#discovery). !!! example The Toolkit [Session Service](../../services/rta-sessionsvc/README.md) does not send properties under `extDetails` unless requested by the client. To request additional properties (e.g. `#!json "extDetails.CarSetup.RideHeightFront"` and `#!json "extDetails.CarSetup.RideHeightRear"`): === "POST" ```bash curl -X POST -d "prop=extDetails.CarSetup.RideHeightFront&prop=extDetails.CarSetup.RideHeightRear" -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions" ``` --- # Folders !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Folders RTA can represent sessions in a virtual folder structure. These could represent a file system, or could be used creatively to describe: * Recently acquired data * Schedule (e.g. sporting events or public transport) * Sources of data * Anomalies detected by algorithms * Saved queries and data sets !!! info ATLAS doesn't yet exploit this feature, but will do so in a future release. The specification is stable and ready to be used by web interfaces and surrounding systems. ## Concepts ### What is a Folder? Folder do not have to represent file system paths. A folder could be a virtual filter for 'recent' data, or for saved queries, or provide alternate ways to organise the data (by source, by location, etc.) Each folder has an unique identity, which could be a path, or a GUID. ATLAS does not assume any particular naming convention. ### Filtering by Folder Sub-tree Some of the REST API calls — such as listing sessions and requesting the metamodel — accept a `folder` argument. This provides the root for a recursive search: * When a `folder` is specified, all session in that folder and sub-folders are in scope; * If no `folder` is specified, the entire tree is in scope. ## Session Service walk-through See also: [gRPC API](../../services/rta-sessionsvc/grpc.md) ### Creating a Folder Tree The root folder always has an empty identity string, and cannot be modified. The service will accept any folder identity token format. The following example uses a path with `::` separators, but a GUID or some other path format will work as well. In this example, we create a folder structure like this: | +-- animals "Animals" | | | +-- animals::felines "Felines" | +-- plants "Plants" === "C# Sample" Creating the folder structure: ```c# using var sessionChannel = GrpcChannel.ForAddress("http://localhost:2652"); var sessionClient = new SessionStore.SessionStoreClient(sessionChannel); await sessionClient.PutSessionFolderAsync(new PutSessionFolderRequest { Identity = "animals", Label = "Animals", Description = "All creatures great and small" }); await sessionClient.PutSessionFolderAsync(new PutSessionFolderRequest { Identity = "plants", Label = "Plants", Description = "Sometimes called 'vegetables'" }); await sessionClient.PutSessionFolderAsync(new PutSessionFolderRequest { ParentIdentity = "animals", Identity = "animals::felines", Label = "Felines", Description = "Big cats and small cats" }); ``` !!! tip This is an idempotent service call, so it can also be used to: * rename a folder, by re-defining it with a different label; * move a folder, by re-defining its parent identity. === "JSON from REST API" Reading the children under the root folder: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/session-folders/"` ```json [ { "identity": "animals", "label": "Animals", "description": "All creatures great and small", "hasChildren": true }, { "identity": "plants", "label": "Plants", "description": "Sometimes called 'vegetables'", "hasChildren": false }, ] ``` Reading the children under `#!json "animals"`: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/session-folders/animals"` ```json [ { "identity": "animals::felines", "label": "Felines", "description": "Big cats and small cats", "hasChildren": false } ] ``` ### Sessions and Folders By default, sessions are created in the root folder. Creating a session in the `animals` folder, and then adding it to the `plants` folder: === "C# Sample - Creating in a folder" Creating in `animals`: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", CreateIfNotExists = new() { Identifier = "Example", Timestamp = DateTimeOffset.Now.ToString("O") }, Updates = { new SessionUpdate { SetFolders = new() { Identities = { "animals" } } } } }); ``` Now this session will appear listings for the root folder or `animals` folder, but not `plants`: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?folder=animals"` ```json { "sessions": [ { "identity": "example", "state": "unknown", "timestamp": "2021-05-19T13:10:04+00:00", "identifier": "Example" } ] } ``` `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?folder=plants"` ```json { "sessions": [] } ``` === "C# Sample - Adding to another folder" Adding to `plants` with a `PutFolders` operation: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { UpdateFolders = new() { Identities = { "plants" } } } } }); ``` After this call, the session is present in both `animals` and `plants`: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?folder=animals"` ```json { "sessions": [ { "identity": "example", "state": "unknown", "timestamp": "2021-05-19T13:10:04+00:00", "identifier": "Example" } ] } ``` `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions?folder=plants"` ```json { "sessions": [ { "identity": "example", "state": "unknown", "timestamp": "2021-05-19T13:10:04+00:00", "identifier": "Example" } ] } ``` === "C# Sample - Removing from a folder" Removing from `plants` with a `DeleteFolders` operation: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { DeleteFolders = new() { Identities = { "plants" } } } } }); ``` After this call, the session is only present in `animals` again. !!! info A session must be in a folder at all times. If a session is removed from all folders, the session is re-parented into the root folder. Folder membership is essentially a symbolic link: a session can appear in multiple folders without being physically copied. ### Deleting folders This symbolic link approach extends to folder deletion: Deleting folders does not delete sessions. If a session would be left without a parent folder, it is re-parented in the root folder — which cannot be deleted. === "C# Sample" Deleting the `animals` subtree and `plants` folder: ```c# await sessionClient.DeleteSessionFoldersAsync(new DeleteSessionFoldersRequest { Identities = { "animals", "plants" } }); ``` --- # Session Metamodel !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Session Metamodel The metamodel[^1] enables the service to describe what session properties are available and what values they can take. !!! info The [Session Service](../../services/rta-sessionsvc/README.md) generates a metamodel from session properties. There is scope for still-deeper integration for implementations that manage complex simulation models or perform federated queries into other systems. For example: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/session-metamodel"` ```json { "details": { "label": "Details", "properties": { "Engineer": { "type": "string", "constraints": { "values": [ "Joe", "Robert" ] }, "capabilities": { "filter": true, "sort": true } } // ... } } } ``` ATLAS uses this summary to drive the Session Browser filters, so that [queries](queries.md) can run server-side. In addition to constraints, the metamodel can include formatting rules for display and indicate whether the server supports _filtering_ or _sorting_ for a property. ## Capabilities The session model already defines some properties — such as `"state"`, `"timestamp"` and `"type"`. For each of these properties, the server can declare its capabilities: ```json { "state": { "filter": true, "sort": true }, "timestamp": { "edit": true, "filter": true, "sort": true } } ``` ## Constraints The metamodel can also describe the aggregate values a property has, or can take: ```json { "type": { "capabilities": { "filter": true, "sort": true }, "constraints": { "values": [ "Stream", "Model" ], "minLength": 1, "maxLength": 32 } } } ``` In this example, the `type` property: * is valid for sorting * is valid for filtering, and the client can offer `#!json "Stream"` and `#!json "Model"` as pre-set values * is valid for editing, but must take a value of `#!json 1`-`#!json 32` characters in length !!! info ATLAS uses `"values"` but currently does not use the other constraints. Greater support will be added in future releases. ## Display Pre-defined properties have an established description and display convention. However, the server can define arbitrary properties within `"details"` and `"extDetails"`, and describe them in the metamodel: ```json { "details": { "size": { "label": "Size", "description": "Size of the vehicle", "type": "string", "capabilities": { "filter": true, "sort": true }, "constraints": { "values": [ "small", "medium", "large" ], "onlyValues": true } }, "weight": { "label": "Weight", "description": "Just how heavy is this thing?", "type": "number", "capabilities": { "filter": true, "sort": true }, "display": { "unit": "kg", "format": "%.2f" }, "constraints": { "minimumExclusive": 0.0 } } } } ``` !!! info ATLAS currently does not use the display rules, but may do so in future releases. !!! warning The [Session Service](../../services/rta-sessionsvc/README.md) does not currently provide a way to describe display rules. ## Discovery `details` and in particular `extDetails` can represent very large meta-models — such as a detailed hardware and software setup parameters. The service may choose not to expose some properties to the client by default, but still make them available for discovery. Querying the metamodel produces results like this: `#!bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/session-metamodel/properties?query=details."` ```json [ { "id": "details.appearance", "label": "Appearance" }, { "id": "details.taste", "label": "Taste" }, { "id": "details.size", "label": "Size", "description": "Size of the vehicle" }, { "id": "details.weight", "label": "weight", "description": "Just how heavy is this thing?" } ] ``` [^1]: "model of the model" --- # Data Bindings !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Data Bindings When deploying [Microservices](../tutorials/microservices/index.md), you need to create a binding linking sessions to data. Deployments can have more than one data service for different backends. A request for data needs to hit the correct data service, and use a meaningful identifier — for example a file path, or query expression. This binding is captured in the [Session Service](../../services/rta-sessionsvc/README.md) and the [Gateway Service](../../services/rta-gatewaysvc/README.md) uses it to proxy incoming requests. !!! info Data Bindings are not needed for [RTA Server](../../services/rta-server/README.md). ## Request Flow Typical ATLAS request flow: 1. Fetch the session model — `/rta/v2/sessions/` 2. Fetch the configs bound to the session — `/rta/v2/configs/` 3. Fetch samples and events based on the configs — `/rta/v2/sessions//data//` For [RTA Server](../../services/rta-server/README.md), these activities are all handled by a single process, so no data binding is needed. But for a microservices deployment, ATLAS talks to the [Gateway Service](../../services/rta-gatewaysvc/README.md) which: 1. Fetches the session model from the [Session Service](../../services/rta-sessionsvc/README.md) 2. Fetches the configs from the [Config Service](../../services/rta-configsvc/README.md) 3. Fetches the data from a data service (e.g. [Toolkit Data Service](../../services/rta-datasvc/README.md), or [Influx Data Service](../../services/rta-influxdatasvc/README.md), or another data adapter): 1. Gets the session data bindings from the [Session Service](../../services/rta-sessionsvc/README.md) 2. Requests data for a _data identity_ from a _data source_ 3. Streams the data back to the client — with transformations if indicated by the data binding So each data binding must at least specify: * The _data source_ — which tells the [Gateway Service](../../services/rta-gatewaysvc/README.md) which data service to talk to; * The _data identity_ — which is passed to the data service; !!! info The _data source_ and _data identity_ are private: the [Gateway Service](../../services/rta-gatewaysvc/README.md) does not expose them via the REST API. * _data source_ is just a token — e.g. `influx`; * _data identity_ could be a file path, guid, query — or something else meaningful to the data service. ## Creating Data Bindings Use the [Session Service gRPC API](../../services/rta-sessionsvc/grpc.md) to `CreateOrUpdateSession` and populate `data_bindings`: === "C# Sample — Simple" This minimal binding will need a configured data service for the `influx01` _data source_ — e.g. [Influx Data Service](../../services/rta-influxdatasvc/README.md) — and will use the specified _data identity_ as a filter expression to select data for the session: ```c# await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetDataBindings = new() { DataBindings = { new DataBinding { Key = new() { Source = "influx01", Identity = "someField='abc123'" } } } } } } }); ``` === "Time Ranges" This pair of bindings captures discontinuous time ranges from the same _data source_ and _data identity_: ```c# hl_lines="20 21 31 32" await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetDataBindings = new() { DataBindings = { new DataBinding { Key = new() { TimeRef = startTime, Source = "influx01", Identity = "someField='abc123'" }, SessionStartTime = startTime, SessionEndTime = startTime + 599_999_999L }, new DataBinding { Key = new() { TimeRef = startTime + 900_000_000L, Source = "influx01", Identity = "someField='abc123'" }, SessionStartTime = startTime + 900_000_000L, SessionEndTime = startTime + 999_999_999L } } } } } }); ``` !!! note In this case, a `time_ref` is added to the `key` to make it unique. The value of this additional key element is arbitrary: in this example it is matched to the data time range. !!! important As a result, the [Gateway Service](../../services/rta-gatewaysvc/README.md) will need to inspect and potentially re-write data as it is streamed to the client to clip the data strictly to the specified bounds. This is required because the RTA specification allows (and encourages) data services to serve data beyond the specified time-bounds so this clipping does not need to be performed on all requests, and so that data chunks have predictable cache coordinates. Data transformation can impact data transfer throughput, so avoid using time bounds like this unless it is necessary. === "Time Shifts" This binding indicates that the session time is 300 milliseconds later than the data time: ```c# hl_lines="19 19" await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetDataBindings = new() { DataBindings = { new DataBinding { Key = new() { Source = "influx01", Identity = "someField='abc123'" }, SessionTimeOffset = 300_000_000L } } } } } }); ``` As a result, the [Gateway Service](../../services/rta-gatewaysvc/README.md) will need to re-write data as it is streamed to the client to add 300 milliseconds to all timestamps. This is the main difference between this service and a conventional reverse proxy: it has some purpose-built data transformations. !!! important Data transformation can substantially impact data transfer throughput. It is better to apply corrections upstream where possible. === "Multiple Sources" This binding is a composite of data from two separate data services: ```c# hl_lines="16 16 33 33" await sessionClient.CreateOrUpdateSessionAsync(new CreateOrUpdateSessionRequest { Identity = "example", Updates = { new SessionUpdate { SetDataBindings = new() { DataBindings = { new DataBinding { Key = new() { Source = "influx01", Identity = "someField='abc123'" }, DataChannelBindings = { new DataChannelBinding { FirstDataChannel = 0, FirstSessionChannel = 0, LastSessionChannel = 49 } } }, new DataBinding { Key = new() { Source = "influx07", Identity = "someOtherField='xxxyyyzzz'" }, DataChannelBindings = { new DataChannelBinding { FirstDataChannel = 0, FirstSessionChannel = 50, LastSessionChannel = 59 } } } } } } } }); ``` This also involves a transformation: data from the second binding is channel-shifted so that it is in the range `#!json 50`-`#!json 59`. Data from the first binding can be proxied unchanged. !!! important Session time-range and configuration must reflect the result after transformation. In this example, a session might have _two_ configuration bindings — one for each underlying data source — and use a channel offset of `#!json 50` on the second binding. This would mean that ATLAS requests channels in the range `#!json 0`-`#!json 59` and the requests are proxied accordingly by the [Gateway Service](../../services/rta-gatewaysvc/README.md). ## Fetching Data Bindings The [Gateway Service](../../services/rta-gatewaysvc/README.md) needs to be able to request data bindings from the [Session Service](../../services/rta-sessionsvc/README.md): ```bash curl -H "Accept: application/json" "http://localhost:2650/rta/v2/sessions/example/data-bindings" ``` ```json [ { "key": { "source": "influx01", "identity": "someField='abc123'", "timeRef": 0 }, "sessionStartTime": null, "sessionEndTime": null, "sessionTimeOffset": 0, "dataChannelBindings": [ { "firstDataChannel": 0, "firstSessionChannel": 0, "lastSessionChannel": 49 } ] }, { "key": { "source": "influx07", "identity": "someOtherField='xxxyyyzzz'", "timeRef": 0 }, "sessionStartTime": null, "sessionEndTime": null, "sessionTimeOffset": 0, "dataChannelBindings": [ { "firstDataChannel": 0, "firstSessionChannel": 50, "lastSessionChannel": 59 } ] } ] ``` This REST call is only used between these two services. It is never called by ATLAS and should not be exposed in the public API since data bindings might contain sensitive information. --- # Configuration Basics !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Configuration Basics ATLAS/RTA _Configuration_ is a rich schema decribing the telemetry. ATLAS is designed so that a stream of telemetry might contain data from multiple devices and models, assembled by different teams — so configuration is divided into _Apps_. In the Configuration API (from the _MAT.OCS.Configuration_ [NuGet Package](../../downloads.md#nuget-packages)), you can define one or more apps at a time, containing: * [Channels and Parameters](channels-parameters.md) — describing sampled data * [Parameter Groups](parameter-groups.md) — forming a parameter tree * [Conversions](conversions.md) — transforming and formatting data for display * [Event Definitions](event-defs.md) — describing events Here is a simple `#!json "demo"` app, published to the Toolkit [Config Service](../../services/rta-configsvc/grpc.md): === "C# Sample" All the code samples use the _MAT.OCS.Configuration_ [NuGet Package](../../downloads.md#nuget-packages). A JSON example is provided with each sample for reference if you are working in another language. ```c# using Grpc.Net.Client; using MAT.OCS.Configuration; using MAT.OCS.Configuration.Builder; using MAT.OCS.RTA.Model.Net; using MAT.OCS.RTA.Toolkit.API.ConfigService; // leave this channel open across service calls so it can be reused using var configChannel = GrpcChannel.ForAddress("http://localhost:2662"); var configClient = new ConfigStore.ConfigStoreClient(configChannel); var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { Channels = { new ChannelBuilder(0u, 0L, DataType.Double64Bit, ChannelDataSource.Timestamped) }, ChildParameters = { new ParameterBuilder("param:demo", "param", "Example Parameter") { MinimumValue = -1500.0, MaximumValue = +1500.0, WarningMinimumValue = -1000.0, WarningMaximumValue = +1000.0, ChannelIds = {0u} } }, EventDefinitions = { new EventDefinitionBuilder(8, "0008:demo Example Event") { Priority = EventPriority.High } } } } }.BuildConfiguration(); // identifier can be reused as long as config does not change const string configIdentifier = "17115312-F294-43E4-8027-4BE5823923A9"; // serialize and publish (as FFC, by default) await configClient.PutConfigAsync(configIdentifier, config); // alternatively, as JSON // await configClient.PutConfigAsync(configIdentifier, config, MediaTypes.JsonConfig); ``` === "JSON" Configuration can be published as [JSON](serializing.md#json), which is described by a [JSON schema](../../api/config.schema.json). ```json { "demo": { "tree": { "params": [ "param:demo" ] }, "parameters": [ { "id": "param:demo", "name": "param", "desc": "Example Parameter", "min": -1500.0, "max": 1500.0, "warnMin": -1000.0, "warnMax": 1000.0, "conv": "1to1:demo", "channels": [ 1 ] } ], "channels": [ { "id": 1, "source": "timestamped" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ], "events": [ { "id": 8, "code": "0008", "desc": "0008:demo Example Event", "pri": "high", "convs": [] } ] } } ``` --- # Channels and Parameters !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Channels and Parameters _Parameters_ are the abstraction for accessing sampled data. Each parameter can actually be an aggregate of multiple channels — for example, sampled periodically at multiple rates. !!! example Motion Applied hardware loggers can capture data at multiple rates, using triggers to define conditions for high-rate collection. For example, when logging twisting forces within a gearbox, high-rate sampling is only required when changing gear, and can be triggered for a period by the gear-change event. This makes optimal use of bandwidth and storage in the logging hardware and telemetry uplink. Each channel can also be presented with multiple parameters. ## Channels Channel properties are: `Id` _(unsigned 32-bit integer)_: : Unique id within a configuration, numbered contiguously from `#!json 0` or `#!json 1`. Each parameter should reference one or more channels by id. `Interval` _(signed 64-bit integer)_: : Interval in nanoseconds between samples, or `#!json 0` if the channel is not periodically sampled. `Data Type` _(enum)_: : Data type/precision. Signed/Unsigned integers for 8-bit, 16-bit, and 32-bit; Floating point at 32-bit and 64-bit precision. `Data Source` _(enum)_: : Data source, which for RTA should be one of `Timestamped`, `Periodic` or `RowData`. * [Timestamped Data](../data/index.md#timestamped-data) is variable-rate and has a delta-encoded timestamp for each sample; * [Periodic Data](../data/index.md#periodic-data) is sampled at a fixed rate — though sampling may not be continuous; * [Row Data](../data/index.md/#row-data) encodes sections of memory — usually from an embedded controller; Channels can also be assigned a `Name` — sometimes used to indicate a memory address. !!! note The Configuration API and JSON schema permit some additional values for the `Data Type` and `Data Source`. These have specialized uses and are not supported with RTA at this time. ## Parameters `Identifier`: : Globally-unique identifier, which should be qualified with the app name: `someParameter:myapp` The prefix (name-part) should follow the same rules as variables in C-like languages: * Use a-z, A-Z, 0-9 and underscore * Don't start with a number This ensures that the parameter can be used in functions in all situations without problems. `Name`: : Human-readable name. Usually the same as the `Identifier`, minus the app name qualifier. ATLAS will show this name in the Parameter Browser. As with `Identifier`, we recommend using only C-like variable names to ensure compatibility with functions. `Description`: : Human-readable description. ATLAS will show this description in the Parameter Browser. ### Channel Ids Each parameter should reference one (or more) channels, by Id. The channels must belong to the same app. === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("param:demo", "param", "Example Parameter") { ChannelIds = { 1u } } }, Channels = { new ChannelBuilder(1u, 0, DataType.Double64Bit, ChannelDataSource.Timestamped) } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "param:demo" ] }, "parameters": [ { "id": "param:demo", "name": "param", "desc": "Example Parameter", "conv": "1to1:demo", "channels": [ 1 ] } ], "channels": [ { "id": 1, "source": "timestamped" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` ### Conversion Every parameter must reference a [Conversion](conversions.md), and the Configuration API will check that it is in the same app. But, for convenience, the Configuration API will automatically add a 1:1 [Rational Conversion](conversions.md#rational-conversions) if no conversion is specified, as shown in this example: === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("param1:demo", "param1", "Example Parameter 1"), new ParameterBuilder("param2:demo", "param2", "Example Parameter 2") { ConversionName = "exampleConv:demo" } }, Conversions = { new TextConversionBuilder("exampleConv:demo") { Table = { [0] = "false", [1] = "true" }, DefaultValue = "file not found", Format = "%s" } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "param1:demo", "param2:demo" ] }, "parameters": [ { "id": "param1:demo", "name": "param1", "desc": "Example Parameter 1", "conv": "1to1:demo" }, { "id": "param2:demo", "name": "param2", "desc": "Example Parameter 2", "conv": "exampleConv:demo" } ], "conversions": [ { "name": "exampleConv:demo", "unit": "", "format": "%s", "text": { "keys": [ 0.0, 1.0 ], "values": [ "false", "true" ], "default": "file not found" } }, { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` ### Range (Min/Max) It is quite important to specify the valid data range, as this will enable ATLAS to auto-scale displays. There are two ranges: * `Minimum Value` and `Maximum Value` represent the minimum and maximum expected physical range (defaulting to `#!json 0.0`); (`"min"`/`"max"` in JSON) * `Warning Minimum Value` and `Warning Maximum Value` represent the minimum and maximum outside which a warning should be triggered — defaulting to the physical range if not specified; (`"warnMin"`/`"warnMax"` in JSON) === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("param:demo", "param", "Example Parameter") { MinimumValue = -1500.0, MaximumValue = +1500.0, WarningMinimumValue = -1000.0, WarningMaximumValue = +1000.0 } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "param:demo" ] }, "parameters": [ { "id": "param:demo", "name": "param", "desc": "Example Parameter", "min": -1500.0, "max": 1500.0, "warnMin": -1000.0, "warnMax": 1000.0, "conv": "1to1:demo" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` ### Format Parameter values are formatted using simple C-style format strings — for example, `"%5.3f"` ??? info "Supported Format Types" | type | description | |------|-------------------------------------------------| | `i` | signed decimal integer | | `d` | signed decimal integer | | `u` | unsigned decimal integer | | `f` | signed decimal floating point | | `g` | shortest representation (%e or %f) | | `G` | shortest representation (%E or %F) | | `e` | scientific notation — lower-case | | `E` | scientific notation — upper-case | | `o` | unsigned octal integer | | `x` | unsigned hexadecimal integer — lower-case | | `X` | unsigned hexadecimal integer — upper-case | | `c` | character | | `s` | string | === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("p1:demo", "p1", "Parameter 1") { ConversionName = "myconv:demo" }, new ParameterBuilder("p2:demo", "p2", "Parameter 2") { ConversionName = "myconv:demo", FormatOverride = "%2.1f" } }, Conversions = { new RationalConversionBuilder("myconv:demo") { Format = "%8.2f" } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "p1:demo" ] }, "parameters": [ { "id": "p1:demo", "name": "p1", "desc": "Parameter 1", "conv": "myconv:demo" }, { "id": "p2:demo", "name": "p2", "desc": "Parameter 2", "format": "%2.1f", "conv": "myconv:demo" } ], "conversions": [ { "name": "myconv:demo", "unit": "", "format": "%8.2f" } ] } } ``` ### Unit Like the format, parameter units (of measure) are defined in the [Conversion](conversions.md) rule, but can also be overridden in the parameter definition. In this example, parameters 1 and 2 units are `"rad"` and `"deg"`, respectively: === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("p1:demo", "p1", "Parameter 1") { ConversionName = "myconv:demo" }, new ParameterBuilder("p2:demo", "p2", "Parameter 2") { ConversionName = "myconv:demo", UnitOverride = "deg" } }, Conversions = { new RationalConversionBuilder("myconv:demo") { Unit = "rad" } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "p1:demo", "p2:demo" ] }, "parameters": [ { "id": "p1:demo", "name": "p1", "desc": "Parameter 1", "conv": "myconv:demo" }, { "id": "p2:demo", "name": "p2", "desc": "Parameter 2", "unit": "deg", "conv": "myconv:demo" } ], "conversions": [ { "name": "myconv:demo", "unit": "rad" } ] } } ``` ### Offset An `Offset` can be applied to the parameter value — for example, to set a zero point. The offset is applied _after_ conversions. === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("param:demo", "param", "Example Parameter") { OffsetValue = +18.6 } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "param:demo" ] }, "parameters": [ { "id": "param:demo", "name": "param", "desc": "Example Parameter", "offset": 18.6, "conv": "1to1:demo" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` ### Shifting and Masking A `Bit Shift` and `Data Bit Mask` can be applied to select specific bits from a channel. * +ve shift is a right-shift * -ve shift is a left-shift * the mask is applied after shift, but before conversions This can be used to extract many boolean values or flags from a packed section of telemetry. !!! info Shifting and Masking can only be applied to integer channels. ???+ example "Worked Example" Bit Shift: `4` Data Bit Mask: `0x1` Channel value (u16): `7638` ``` value bits: 0001 1101 1101 0110 (0x1DD6) shifted value: 0000 0001 1101 1101 mask: 0000 0000 0000 0001 (0x0001) result value: 0000 0000 0000 0001 (0x0001) ``` === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("param:demo", "param", "Example Parameter") { ChannelIds = {1u}, BitShift = 4, DataBitMask = 0x1 } }, Channels = { new ChannelBuilder(1u, 0L, DataType.Unsigned16Bit, ChannelDataSource.RowData) } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "param:demo" ] }, "parameters": [ { "id": "param:demo", "name": "param", "desc": "Example Parameter", "mask": "1", "shift": 4, "conv": "1to1:demo", "channels": [ 1 ] } ], "channels": [ { "id": 1, "type": "u16" } ], "conversions": [ { "name": "1to1:demo", "unit": "", "format": "%5.3f" } ] } } ``` !!! note The `"mask"` is serialized in hexadecimal. --- # Parameter Groups (tree) !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Parameter Groups — Making a Parameter Tree Parameters can be arranged in a tree. Apps are the top-level parameter groups. By convention, these typically represent independent sub-systems or simulation models. Parameter groups have child parameters, but can also contain child parameter groups, forming a tree. === "C# Sample" Like parameters, group identifiers have the same app name suffix to ensure uniqueness. ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("hydro") { Description = "Hydraulics", ChildParameterGroups = { new ParameterGroupBuilder("rat:hydro", "RAM Air Turbine") { ChildParameters = { new ParameterBuilder("ratDeployed:hydro", "ratDeployed", "RAT deployed") } }, new ParameterGroupBuilder("ptu:hydro", "Power Transfer Unit") { // .. } } } } }.BuildConfiguration(); ``` !!! tip In the C# API, the `Application` and `ParameterGroup` classes — after building configuration — both implement `IParameterTreeGroup`. This interface provides description and tree-traversal abstractions. === "JSON" Parameters are serialized as a flat list, and referenced from the `"tree"` by identifier. ```json { "hydro": { "desc": "Hydraulics", "tree": { "groups": [ { "id": "rat:hydro", "desc": "RAM Air Turbine", "params": [ "ratDeployed:hydro" ] }, { "id": "ptu:hydro", "desc": "Power Transfer Unit" } ] }, "parameters": [ { "id": "ratDeployed:hydro", "name": "ratDeployed", "desc": "RAT deployed" } ] } } ``` !!! info Parameters can be added into more than one parameter group, so that they appear in the tree in multiple places. This can be useful where a parameter could fit into more than one logical category. --- # Conversions !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Conversions Conversions are rules for transforming raw data into calibrated engineering data. For some systems, these are the same thing; for other systems, some correction is needed. The three key conversion types are: * [Rational Conversions](#rational-conversions) — which apply a correction function * [Table Conversions](#table-conversions) — which provide a lookup table * [Text Conversions](#text-conversions) — which convert discrete enumerated values to text They have some common properties: `Name`: : Identifies the conversion so it can be referenced from a Parameter or Event Definition. Should have the app name (e.g. `#!json "someConversion:myapp"`) as a suffix to ensure uniqueness. `Format`: : C-style format string (e.g. `#!json "%5.2f"` or `#!json "Turbine activated: %s"`). `Unit`: : Unit of measure (e.g. `#!json "deg"` or `#!json "°"`). Parameters inherit the `Format` and `Unit` from their conversion rule, though they can override it. ## Rational Conversions Use Rational Conversions to apply simple corrections raw values, such as gain or offset. Raw data are mapped onto engineering values using six coefficients in a standard formula: `eng = (c1 * raw^2 + c2 * raw + c3) / (c4 * raw^2 + c5 * raw + c6)` A 1:1 conversion therefore has coefficients `[0,1,0, 0,1,0]`. This is the default conversion if none is specified. This example applies a `+5.0` offset: === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { ChildParameters = { new ParameterBuilder("offsetExample:demo", "offsetExample", "Offset example") { ConversionName = "offsetExample:demo" } }, Conversions = { new RationalConversionBuilder("offsetExample:demo") { C1 = 0, C2 = 1, C3 = +5.0, C4 = 0, C5 = 1, C6 = 0 } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "tree": { "params": [ "offsetExample:demo" ] }, "parameters": [ { "id": "offsetExample:demo", "name": "offsetExample", "desc": "Offset example", "conv": "offsetExample:demo" } ], "conversions": [ { "name": "offsetExample:demo", "unit": "", "format": "%5.3f", "rational": { "coeffs": [ 0.0, 1.0, 5.0, 0.0, 1.0, 0.0 ] } } ] } } ``` ## Table Conversions Use Table Conversions to map raw data onto engineering values using lookups. This is useful when there is no simple mathematical relationship between the raw values and the corresponding engineering values. Linear interpolation can be indicated if the raw values are not discrete. === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { Conversions = { new TableConversionBuilder("tableExample:demo") { Table = Tuple.Create( new double[]{1, 2, 3}, new double[]{1, 20, 40} ), Interpolated = false } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "conversions": [ { "name": "tableExample:demo", "table": { "raw": [ 1.0, 2.0, 3.0 ], "calibrated": [ 1.0, 20.0, 40.0 ], "interpolated": false } } ] } } ``` ## Text Conversions Use Text Conversions to map discrete numerical values to text — aka _"enumerations"_. === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { Conversions = { new TextConversionBuilder("textExample:demo") { Table = { [1] = "one", [2] = "two", [3] = "three" }, DefaultValue = "?", Format = "%s" } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "conversions": [ { "name": "textExample:demo", "format": "%s", "text": { "keys": [ 1.0, 2.0, 3.0 ], "values": [ "one", "two", "three" ], "default": "?" } } ] } } ``` --- # Event Definitions !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Event Definitions Instantaneous changes are captured using Events. ATLAS has a dedicated display for events, and renders them in the waveform timeline alongside sampled data. ==picture?== !!! info "How are events different from sampled data?" _Sampled data_ is generally a measure of a value varying continuously within a time interval, whereas an _event_ is generated at a precise moment. For example: an _event_ can capture a switch position change at the moment it happens, together with a value indicating the new position; _sampled data_ would typically check the switch position at intervals and record the current state as a value. Events are parameterized — they can have associated values or text. Like parameters for sampled data, events have an _Event Definition_ in configuration, which provides extra description and formatting rules for the data. ## Properties `Id` _(integer)_: : Unique id. This should be unique amongst all apps — which can be awkward, but is really necessary for ATLAS to work correctly in all scenarios, including hardware logging where only the ID is present in the data stream. `Id Code` _(text)_: : Alternate id for display and filtering. Not currently used by ATLAS — see note below. `Description` _(text)_: : Human-readable description of the event. By convention, this is a short title. The event itself can carry further text when raised. This should include the Id Code at the start &mdash see note below. `Priority` _(enum)_: : Priority (severity), which can then be used for filtering. Can be one of: `Debug`, `Low`, `Medium`, `High`. `Conversions`: : [Conversion](conversions.md) names — in order — to format the data associated with events when they are raised. Events can contain values or text. If there is no text, the values (if present) are formatted and concatenated to status text using the conversions — commonly using [Text Conversions](conversions.md#text-conversions). If text is present in the event, ATLAS will render that instead. The number and order of conversions must match the data. ATLAS is currently limited to handling up to three data values in each event. !!! important `Id Code` is currently not used by ATLAS. To get the expected result, we recommend formatting the `Description` to include the `Id` and app name as a prefix: * `Id` as four-digit hex * `:` separator * App name * space * Description text For example: `#!json "01AF:app Some event"` === "C# Sample" ```c# var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { EventDefinitions = { new EventDefinitionBuilder(1, "Turbine State Changed") { Priority = EventPriority.High, ConversionNames = { "turbineBool:demo" } } }, Conversions = { new TextConversionBuilder("turbineBool:demo") { Table = { [0] = "false", [1] = "true" }, DefaultValue = "?", Format = "Deployed: %s" } } } } }.BuildConfiguration(); ``` === "JSON" ```json { "demo": { "events": [ { "id": 1, "code": "0001:demo", "desc": "Turbine State Changed", "pri": "high", "convs": [ "turbineBool:demo" ] } ], "conversions": [ { "name": "turbineBool:demo", "unit": "", "format": "Deployed: %s", "text": { "keys": [ 0.0, 1.0 ], "values": [ "false", "true" ], "default": "?" } } ] } } ``` --- # Serializing JSON/FFC !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Serializing Configuration RTA provides two configuration formats: * JSON (`application/vnd.mat.config+json`) * FFC (`application/vnd.mat.config+ffc`) ## JSON The JSON format is defined in the [JSON schema](../../api/config.schema.json). ??? info "JSON Conventions" * Properties can be omitted if they are `#!json null` or empty arrays/maps * Properties are _camelCase_ — e.g. `"timeRange"` instead of `"TimeRange"` * Enumerations are expressed as _camelCase_ string constants — e.g. `#!json "closed"` instead of `#!json "Closed"` For .NET, library support is provided via Motion Applied's [NuGet packages](../../downloads.md#nuget-packages). To convert configuration to JSON, add a package reference for _MAT.OCS.Configuration.Json_ and use the serializer like this: ```c# using MAT.OCS.Configuration.Json; using Newtonsoft.Json; var json = JsonConvert.SerializeObject(config, Formatting.Indented, new ConfigurationJsonConverter()); ``` When publshing to the [Config Service](../../services/rta-configsvc/grpc.md), the _MAT.OCS.RTA.Toolkit.API.GrpcClients_ package [handles JSON serialization automatically](publishing.md): ```c# using MAT.OCS.RTA.Toolkit.API.ConfigService; // handles serialization automatically await configClient.PutConfigAsync(configIdentifier, config, MediaTypes.JsonConfig); ``` ## FFC FFC is a Motion Applied proprietary binary format with support for internal indexing and selective parsing. It is supported in C#, F#, C++ VB.NET for .NET Core and .NET Framework, via Motion Applied's [NuGet packages](../../downloads.md#nuget-packages). If you are using another language, serialize configuration as JSON. !!! info This is the default format when publishing to the [Config Service](../../services/rta-configsvc/README.md) using the _MAT.OCS.RTA.Toolkit.API.GrpcClients_ package. To serialize to FFC explicitly, add a package reference for _MAT.OCS.FFC.ConfigSerializer_ and use the serializer like this: ```c# using MAT.OCS.FFC.ConfigSerializer; await using var ms = new MemoryStream(); FastConfigSerializer.Serialize(config, "RTA", identifier).ToStream(ms); var ffc = ms.ToArray(); ``` When publshing to the [Config Service](../../services/rta-configsvc/grpc.md), the _MAT.OCS.RTA.Toolkit.API.GrpcClients_ package [handles FFC serialization automatically](publishing.md):: ```c# using MAT.OCS.RTA.Toolkit.API.ConfigService; // handles serialization automatically await configClient.PutConfigAsync(configIdentifier, config); ``` --- # Publishing to the Config Service !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Publishing to the Toolkit Config Service The [gRPC API](../../services/rta-configsvc/grpc.md) provides two service calls to publish configuration: * `PutConfig` — works for configs up to about 4 MiB * `PutConfigStream` — is more complex to use, but works with configs of any size The _MAT.OCS.RTA.Toolkit.API.GrpcClients_ package has these methods, and also a convenience method which handles the serialization to [JSON](serializing.md#json) or [FFC](serializing.md#ffc) at the same time. === "C# Sample: Publish and Serialize" ```c# using MAT.OCS.Configuration.Builder; using MAT.OCS.RTA.Model.Net; using MAT.OCS.RTA.Toolkit.API.ConfigService; var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { // ... } } }.BuildConfiguration(); var configIdentifier = Guid.NewGuid().ToString(); using var configChannel = GrpcChannel.ForAddress("http://localhost:2662"); var client = new ConfigStore.ConfigStoreClient(configChannel); // handles FFC serialization automatically await configClient.PutConfigAsync(configIdentifier, config); // or use this call to explicitly serialize as JSON // await configClient.PutConfigAsync(configIdentifier, config, MediaTypes.JsonConfig); ``` === "C# Sample: Publish < 4 MiB" ```c# using MAT.OCS.Configuration.Builder; using MAT.OCS.Configuration.Json; using MAT.OCS.RTA.Toolkit.API.ConfigService; using Newtonsoft.Json; var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { // ... } } }.BuildConfiguration(); // serialize some config to JSON var json = JsonConvert.SerializeObject(config, new ConfigurationJsonConverter()); var configIdentifier = Guid.NewGuid().ToString(); using var configChannel = GrpcChannel.ForAddress("http://localhost:2662"); var client = new ConfigStore.ConfigStoreClient(configChannel); // suitable for up to a bit less than 4 MiB await client.PutConfigAsync(new PutConfigRequest { Identifier = id, ContentType = mediaType, Data = ByteString.CopyFromUtf8(json) }); ``` === "C# Sample: Publish > 4 MiB" ```c# using MAT.OCS.RTA.Toolkit.API.ConfigService; var config = new ConfigurationBuilder { Applications = { new ApplicationBuilder("demo") { // ... } } }.BuildConfiguration(); // serialize some config to JSON var json = JsonConvert.SerializeObject(config, new ConfigurationJsonConverter()); var ms = new MemoryStream(new UTF8Encoding(false).GetBytes(json)); // no BOM // use a new identifier every time if the config is not always the same var configIdentifier = Guid.NewGuid().ToString(); using var configChannel = GrpcChannel.ForAddress("http://localhost:2662"); var client = new ConfigStore.ConfigStoreClient(configChannel); using var call = client.PutConfigStream(new CallOptions()); var rs = call.RequestStream; // publish the identifier as the first message await rs.WriteAsync(new ConfigStreamMessage { Info = new ConfigStreamMessage.Types.ConfigInfo { Identifier = id, ContentType = mediaType } }); // publish config a bit at a time - 4 KiB is fine var buffer = new byte[4096]; var r = 0; while ((r = ms.Read(buffer, 0, buffer.Length)) > 0) { var msg = new ConfigStreamMessage { Piece = ByteString.CopyFrom(buffer, 0, r) }; await rs.WriteAsync(msg); } await rs.CompleteAsync(); await call.ResponseAsync; ``` --- # Data Basics !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Data Basics RTA supports four data representations: | Representation | Columnar[^1] | Variable Rate | Text Support | |---------------------------------------|--------------|---------------|--------------------| | [Timestamped Data](#timestamped-data) | yes | yes | enum only | | [Periodic Data](#periodic-data) | yes | no | enum only | | [Row Data](#row-data) | no | yes | enum only | | [Event](#event) | yes | yes | enums or free-text | [^1]: i.e. one parameter/channel at a time These are all defined in the [rta.model.data](../protobuf/model_data.md) protobuf schema. Pre-compiled classes are available for .NET in our [NuGet packages](../../downloads.md#nuget-packages), or you can use the [protobuf compiler](https://github.com/protocolbuffers/protobuf/releases) to generate idiomatic code from the schema in a range of languages. All RTA data timestamps are measured in nanoseconds since the Unix Epoch (1970-01-01 00:00:00Z). !!! important Since data timestamps are an offset from **UTC**, you must subtract the timezone offset from local timestamps — or use a library function that handles timezones. For example, these ISO 8601 timestamps are all the same offset (`1609502400000000000`): * 2021-01-01T13:00:00+01:00 * 2021-01-01T12:00:00Z * 2021-01-01T04:00:00-08:00 !!! info * 1 second = 1,000,000,000 nanoseconds * 1 millisecond = 1,000,000 nanoseconds * 1 tick (Windows) = 100 nanoseconds === "C# Timestamps Example" ``` C# var ts = DateTimeOffset.Parse("2021-01-01T13:00:00+01:00"); var nanos = (ts - DateTimeOffset.UnixEpoch).Ticks * 100; Console.WriteLine("Timestamp: " + nanos); ``` ``` Timestamp: 1609502400000000000 ``` ## Timestamped Data This representation encodes short bursts of samples, at a variable rate, for one channel. ### Schema ``` protobuf message TimestampedData { // Channel Id. uint32 channel_id = 1; // Start timestamp, in nanoseconds relative to the Unix epoch. sfixed64 start_timestamp = 2; // Multiplier (ns) to apply to timestamp deltas. int64 timestamp_deltas_scale = 3; // Timestamp deltas (ns / multiplier) for each sample relative to the previous one. // The first delta is always zero. // If the interval is constant, all subsequent deltas should be identical. // If the multiplier were set to the interval, the subsequent deltas would all be 1. // This scheme allows for efficient variable-length encoding. repeated int64 timestamp_deltas = 4; // Buffer of samples. Every sample must be encoded to alike and to the same width // so the buffer is splittable; the rest of the encoding is defined in config. bytes buffer = 5; } ``` Combining the delta-encoding scheme with protobuf variable-length integer encoding results in a compact, compressible message. ??? example "Worked Example" _Input data:_ | Timestamp | Channel 16 (`double`) | |----------------------|-----------------------| | 2021-05-04T11:50:55Z | 12.7 | | 2021-05-04T11:50:59Z | 14.9 | | 2021-05-04T11:51:04Z | 21.2 | _Encoded:_ | Field | Value | Note | |--------------------------|-----------------------|--------------------------------------------------------------| | `channel_id` | `16` | Can only represent a single channel at a time. | | `start_timestamp` | `1620129055000000000` | Nanoseconds since Unix epoch (UTC). | | `timestamp_deltas_scale` | `1000000000` | Input data has timestamps at 1 second resolution. | | `timestamp_deltas` | `[0, 4, 5]` | Note that the deltas are relative to each other. | | `buffer` | `0x 6666666666662940 CDCCCCCCCCCC2D40 3333333333333540` | Little-endian bytes (hex). | !!! important Bursts of data must never overlap each other. Data is served from the [REST API](../../api/index.md#operation/get-timestamped-data) in chunks using this list type: ```protobuf message TimestampedDataList { // Zero or more timestamped data items. repeated TimestampedData timestamped_data = 1; } ``` ### Code Samples === "C# Sample — `double` values" Using _MAT.OCS.RTA.API_ to encode `double`-precision data: ``` C# static TimestampedData Encode(uint channelId, long[] timestamps, double[] samples) { var burst = new TimestampedData { ChannelId = channelId }; burst.SetTimestamps(timestamps); burst.Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(samples.AsSpan())); return burst; } ``` !!! notes * `SetTimestamps` is an extension method, setting the `start_timestamp`, `timestamp_deltas_scale` and `timestamp_deltas`. The scale is calculated from the [Greatest Common Divisor](https://www.freecodecamp.org/news/euclidian-gcd-algorithm-greatest-common-divisor/). This has a neligible peformance impact. There is a corresponding `FillTimestamps` method to unpack timestamps into an array. * [`MemoryMarshal.AsBytes(...)`](https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.memorymarshal.asbytes?view=net-5.0) is the most efficient way to cast values to a buffer. [`Buffer.BlockCopy`](https://docs.microsoft.com/en-us/dotnet/api/system.buffer.blockcopy?view=net-5.0) is the .NET Framework alternative but involves an extra array and copy operation. * Memory allocation is the biggest performance bottleback. Both `SetTimestamps` and `ByteString.CopyFrom` allocate on the heap and this cannot be avoided using the C# generated protobuf classes, but try to reuse the `timestamps` and `samples` arrays or rent them from an [ArrayPool](https://docs.microsoft.com/en-us/dotnet/api/system.buffers.arraypool-1?view=net-5.0). The configuration should look like this: ``` C# new ChannelBuilder(channelId, 0L, DataType.Double64Bit, ChannelDataSource.Timestamped) ``` !!! info * The `Interval` should always be `0` for Timestamped Data * The `DataType` must match the values array type (`double[]` in this sample) * The `DataSource` must be `ChannelDataSource.Timestamped` * The `ByteOrder` should be little-endian — but this is already the default === "C# Sample — `ushort` values" Representing `ushort` (unsigned 16-bit) values is almost exactly the same: ``` C# static TimestampedData Encode(uint channelId, long[] timestamps, ushort[] samples) { var burst = new TimestampedData { ChannelId = channelId }; burst.SetTimestamps(timestamps); burst.Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(samples.AsSpan())); return burst; } ``` But the configuration does need to be updated for ATLAS: ``` C# new ChannelBuilder(channelId, 0L, DataType.Unsigned16Bit, ChannelDataSource.Timestamped) ``` === "C# Sample — reading `TimestampedDataList`" ```c# private static void DecodeList(TimestampedDataList dataList) where T : struct { foreach (var data in dataList.TimestampedData) { var sampleCount = data.TimestampDeltas.Count; var timestamps = ArrayPool.Shared.Rent(sampleCount); try { data.FillTimestamps(timestamps); // valid up to sampleCount var samples = MemoryMarshal.Cast(data.Buffer.Span); // consume timestamps and samples ... } finally { ArrayPool.Shared.Return(timestamps); } } } ``` ## Periodic Data This representation encodes short bursts of data sampled at regular intervals, for one channel. ### Schema ```protobuf message PeriodicData { // Channel Id. uint32 channel_id = 1; // Start timestamp, in nanoseconds relative to the Unix epoch. sfixed64 start_timestamp = 2; // Interval between samples (ns). int64 interval = 3; // Number of samples in the buffer. int32 samples = 4; // Buffer of samples. Every sample must be encoded to alike and to the same width // so the buffer is splittable; the rest of the encoding is defined in configuration. bytes buffer = 5; } ``` This schema can be significantly more efficient than [Timestamped Data](#timestamped-data) because it does not need to represent timestamps for each value. It is also a bit easier to encode. The `interval` field enables a burst to be split by timestamp without reference to configuration. !!! tip This representation is ideal for samples from hardware or software with an accurate schedule. Clock-drift and other timing discontinuities can be handled by starting a new burst. ??? example "Worked Example" _Input data:_ | Timestamp | Channel 16 (`double`) | |----------------------|-----------------------| | 2021-05-04T11:50:55Z | 12.7 | | 2021-05-04T11:50:60Z | 14.9 | | 2021-05-04T11:51:10Z | 21.2 | _Encoded:_ There is a timing discontinuity (skipped sample), so the data must be split into two bursts: | Field | Value | Note | |-------------------|-----------------------|------------------------------------------------| | `channel_id` | `16` | Can only represent a single channel at a time. | | `start_timestamp` | `1620129055000000000` | Nanoseconds since Unix epoch (UTC). | | `interval` | `5000000000` | Timestamps at 5 second intervals. | | `samples` | `2` | Number of samples. | | `buffer` | `0x 6666666666662940 CDCCCCCCCCCC2D40` | Little-endian bytes (hex). | | Field | Value | Note | |-------------------|-----------------------|------------------------------------------------| | `channel_id` | `16` | Can only represent a single channel at a time. | | `start_timestamp` | `1620129070000000000` | Nanoseconds since Unix epoch (UTC). | | `interval` | `5000000000` | Timestamps at 5 second intervals. | | `samples` | `1` | Number of samples. | | `buffer` | `0x 3333333333333540` | Little-endian bytes (hex). | !!! important Bursts of data must never overlap each other. Data is served from the [REST API](../../api/index.md#operation/get-periodic-data) in chunks using this list type: ```protobuf message PeriodicDataList { // Zero or more periodic data items. repeated PeriodicData periodic_data = 1; } ``` ### Code Samples === "C# Sample — `double` values" Using _MAT.OCS.RTA.API_ to encode `double`-precision data: ``` C# static PeriodicData Encode(uint channelId, long startTimestamp, long interval, double[] samples) { return new PeriodicData { ChannelId = channelId, StartTimestamp = startTimestamp, Interval = interval, Samples = samples.Length, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(samples.AsSpan())) }; } ``` !!! notes * [`MemoryMarshal.AsBytes(...)`](https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.memorymarshal.asbytes?view=net-5.0) is the most efficient way to cast values to a buffer. [`Buffer.BlockCopy`](https://docs.microsoft.com/en-us/dotnet/api/system.buffer.blockcopy?view=net-5.0) is the .NET Framework alternative but involves an extra array and copy operation. * Memory allocation is the biggest performance bottleback. `ByteString.CopyFrom` allocates on the heap and this cannot be avoided using the C# generated protobuf classes, but try to reuse `samples` array or rent it from an [ArrayPool](https://docs.microsoft.com/en-us/dotnet/api/system.buffers.arraypool-1?view=net-5.0). The configuration should look like this: ``` C# new ChannelBuilder(channelId, interval, DataType.Double64Bit, ChannelDataSource.Periodic) ``` !!! info * The `Interval` should always match the interval in the `PeriodicData` messages * The `DataType` must match the values array type (`double[]` in this sample) * The `DataSource` must be `ChannelDataSource.Periodic` * The `ByteOrder` should be little-endian — but this is already the default === "C# Sample — `ushort` values" Representing `ushort` (unsigned 16-bit) values is almost exactly the same: ``` C# static PeriodicData Encode(uint channelId, long startTimestamp, long interval, ushort[] samples) { return new PeriodicData { ChannelId = channelId, StartTimestamp = startTimestamp, Interval = interval, Samples = samples.Length, Buffer = ByteString.CopyFrom(MemoryMarshal.AsBytes(samples.AsSpan())) }; } ``` But the configuration does need to be updated for ATLAS: ``` C# new ChannelBuilder(channelId, interval, DataType.Unsigned16Bit, ChannelDataSource.Periodic) ``` === "C# Sample — reading `PeriodicDataList`" ```c# private static void DecodeList(PeriodicDataList dataList) where T : struct { foreach (var data in dataList.PeriodicData) { var startTimestamp = data.StartTimestamp; var samples = MemoryMarshal.Cast(data.Buffer.Span); // ... } } ``` ## Row Data This representation encodes packed buffers covering multiple channels. ### Schema ```protobuf message RowData { // Channel Ids, in the order they are present in the buffer. repeated uint32 channel_ids = 1; // Timestamp, in nanoseconds relative to the Unix epoch. sfixed64 timestamp = 2; // Buffer of samples. // The encoding may vary by channel, as defined in configuration. bytes buffer = 3; } ``` This encoding cannot be parsed without reference to configuration. ??? example "Worked Example" _Input data:_ | Timestamp | Channel 16 (`double`) | Channel 17 (`ushort`) | Channel 18 (`float`) | |----------------------|-----------------------|-----------------------|----------------------| | 2021-05-04T11:50:55Z | 12.7 | 234 | 0.7 | _Encoded:_ | Field | Value | Note | |--------------------------|-----------------------|--------------------------------------------| | `channel_ids` | `[16, 17, 18]` | Can represent multiple channels at a time. | | `timestamp` | `1620129055000000000` | Nanoseconds since Unix epoch (UTC). | | `buffer` | `0x 6666666666662940 EA00 3333333F` | Little-endian bytes (hex). | !!! important Channels need to be consistently grouped together — for example, channels `[16, 17, 18]` above must be the same in all bursts. Data is served from the [REST API](../../api/index.md#operation/get-row-data) in chunks using this list type: ```protobuf message RowDataList { // Zero or more row data items. repeated RowData row_data = 1; } ``` ### Code Samples === "C# Sample" Using _MAT.OCS.RTA.API_ to encode data as in _Worked Example_ above: ``` C# static RowData Encode(long timestamp, uint[] channelIds, double x, ushort y, float z) { const int xOffset = 0, xLength = sizeof(double); const int yOffset = xOffset + xLength, yLength = sizeof(ushort); const int zOffset = yOffset + yLength, zLength = sizeof(float); Span bytes = stackalloc byte[xLength + yLength + zLength]; BitConverter.TryWriteBytes(bytes.Slice(xOffset, xLength), x); BitConverter.TryWriteBytes(bytes.Slice(yOffset, yLength), y); BitConverter.TryWriteBytes(bytes.Slice(zOffset, zLength), z); return new RowData { ChannelIds = {channelIds}, Timestamp = timestamp, Buffer = ByteString.CopyFrom(bytes) }; } ``` The configuration should look like this: ``` C# var xCh = new ChannelBuilder(16, 0L, DataType.Double64Bit, ChannelDataSource.RowData); var yCh = new ChannelBuilder(17, 0L, DataType.Unsigned16Bit, ChannelDataSource.RowData); var zCh = new ChannelBuilder(18, 0L, DataType.FloatingPoint32Bit, ChannelDataSource.RowData); ``` !!! info * The `Interval` should always be 0, since the data is non-periodic * The `DataType` must match the values array type (`double[]` in this sample) * The `DataSource` must be `ChannelDataSource.Periodic` * The `ByteOrder` should be little-endian — but this is already the default === "C# Sample — reading `RowDataList`" ```c# private static void DecodeList(RowDataList dataList) { foreach (var data in dataList.RowData) { var timestamp = data.Timestamp; var packedSamples = data.Buffer.Span; // ... } } ``` ## Event This representation is a moment in time with some associated data. For example, this could represent a switch being toggled. ATLAS treats events quite differently to sampled data: * dedicated display for listing and filtering * rendered on the waveform timeline, rather than the plot area * can capture the event multiple times in the same instant Events can carry status text — or up to three values which are then formatted to text. ### Schema ``` protobuf message Event { // Unique event definition id. // This should correspond to an Event Definition in configuration. int32 event_definition_id = 1; // App name/identifier as a qualifier in case there is ambiguity. string app_name = 2; // Timestamp, in nanoseconds relative to the session epoch. sfixed64 timestamp = 3; // Text representation of the event (may be empty). // If empty, this will be substituted for formatted raw_data string status_text = 4; // Raw data values associated with the event. // The corresponding Event Definition specifies conversions // to be applied to this data to produce formatted status_text (if needed). repeated double raw_data = 5; } ``` Events are described in configuration, by `event_definition_id` (located in an app with the corresponding `app_name`). !!! important We recommend that event definition ids are globally-unique, to avoid tool compatibility issues. Data is served from the [REST API](../../api/index.md#operation/get-events) in chunks using this list type: ```protobuf message EventsList { // Zero or more events. repeated Event events = 1; } ``` ### Code Samples === "C# Sample with Text" ``` C# static Event EncodeWithText(int eventDefinitionId, string appName, long timestamp, string text) { return new Event { EventDefinitionId = eventDefinitionId, AppName = appName, Timestamp = timestamp, StatusText = text }; } ``` The configuration should look like this: ``` C# new ApplicationBuilder(appName) { EventDefinitions = { new EventDefinitionBuilder(eventDefinitionId, $"{eventDefinitionId:X4}:{appName} This is an example event") { Priority = EventPriority.Medium } } }; ``` !!! important Format the event definition description to include the `Id` and `AppName` as a prefix, as shown: * Event Definition Id as four-digit hex * ':' separator * App name This avoids a known issue with ATLAS event filtering and may become unnecessary in future. === "C# Sample with Values" ``` C# static Event EncodeWithValues(int eventDefinitionId, string appName, long timestamp, double[] values) { return new Event { EventDefinitionId = eventDefinitionId, AppName = appName, Timestamp = timestamp, RawData = { values } }; } ``` The configuration is unchanged: ``` C# new ApplicationBuilder(appName) { EventDefinitions = { new EventDefinitionBuilder(eventDefinitionId, $"{eventDefinitionId:X4}:{appName} This is an example event") { Priority = EventPriority.Medium } } }; ``` To change the default formatting of the event data, add ==Text Conversions==. --- # Schema Mappings !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Schema Mappings Schema Mappings describe transformations from a data store schema to the RTA domain model. ATLAS does not know anything about how the data is stored. [Configuration](../configuration/index.md) downloaded from the [Config Service](../../services/rta-configsvc/README.md) provides the client with a catalog of parameters and all the information it needs to make requests to the [REST API](../../api/index.md#tag/Data-API) — but it doesn't contain any information about database tables and fields, or file paths and offsets. ## Data Adapters The Toolkit [Data Service](../../services/rta-datasvc/README.md) uses the RTA domain model natively, so no mapping is required. Implementing a Data Adapter service — such as the Toolkit [Influx Data Service](../../services/rta-influxdatasvc/README.md) — do typically need a mapping. When the data adapter receives a request, it is expressed in terms of: * data identity * channels * time range (optional) The adapter must map this onto the data source schema to retrieve the data. Schema Mappings published to the [Schema Mapping Service](../../services/rta-schemamappingsvc/README.md) provide a generalizable way to do this. ## Example This example is from the [InfluxDB Data Adapter Tutorial](../tutorials/influx/review.md#defining-the-schema-mapping): === "C# Sample" ```c# var schemaMapping = new SchemaMapping { Properties = { ["dialect"] = "influxql", ["database"] = "somedb", ["measurement"] = "somedata" }, FieldMappings = { new FieldMapping {SourceField = "alpha", TargetChannel = 0u}, new FieldMapping {SourceField = "beta", TargetChannel = 1u}, new FieldMapping {SourceField = "gamma", TargetChannel = 2u} } }; ``` === "Schema Mapping (as JSON)" ```json { "properties": { "database": "rtademo", "dialect": "influxql", "measurement": "data" }, "field_mappings": [ { "source_field": "alpha", "target_channel": 0, "properties": {}, "source_field_data_type": "", "target_channel_data_type": "TARGET_CHANNEL_DATA_TYPE_UNSPECIFIED" }, { "source_field": "beta", "target_channel": 1, "properties": {}, "source_field_data_type": "", "target_channel_data_type": "TARGET_CHANNEL_DATA_TYPE_UNSPECIFIED" }, { "source_field": "gamma", "target_channel": 2, "properties": {}, "source_field_data_type": "", "target_channel_data_type": "TARGET_CHANNEL_DATA_TYPE_UNSPECIFIED" } ], "event_mappings": [] } ``` ### Properties The `properties` can convey information that applies to all fields and events. In this example, the `"database"` and `"measurement"` are needed as part of the InfluxDB query. ### Field Mappings Fields are primarily described by the mapping from `source_field` to `target_channel`. `properties` can be present on each field, too. In the [Custom Data Adapter Tutorial](../tutorials/data-adapter/review.md#schema-mappings), each field has a `"table"` property. `source_field_data_type` and `target_channel_data_type` are optional. A data adapter might need to perform type conversion when it supports data types that are not in the RTA specification. These fields are pre-defined to make it easier to create general-purpose type-conversion library code. ### Event Mappings Events are primarily described by the mapping from `source_event` to `target_event_app` and `target_event_id`. Like the Field Mappings, `properties` can be present on each Event Mapping. `source_event_data_fields` is optional but pre-defined to help bind event data. ## Using the Schema Mapping Service Schema Mappings are identified by two elements: * `data_source` — identifying the data service (effectively a namespace) * `data_identity` — identifying the data at the data service These must be synchronized between the data ingest scripts or pipelines — where the Schema Mapping is published — and the data service — where it is retrieved. These two elements are also used in the [Data Bindings](../sessions/data-bindings.md), where they need to be synchronized across the data ingest and [Gateway Service](../../services/rta-gatewaysvc/README.md). To avoid confusion, you should use the same _data source_ and _data identity_ pair in both places. !!! info Refer to the [Schema Mapping Service gRPC API](../../services/rta-schemamappingsvc/grpc.md) for all the available methods. Publishing a Schema Mapping: === "C# Sample" ```c# await schemaMappingClient.PutSchemaMappingAsync(new PutSchemaMappingRequest { DataSource = dataSource, DataIdentity = dataIdentity, SchemaMapping = schemaMapping }); ``` !!! info It is safe to update Schema Mappings. They are not shared, and the service caching strategy is designed to allow updates. The service offers both `Get` and `Query` operations. Queries filter the Schema Mapping so that the response only includes selected Field Mappings, and optionally Event Mappings: === "C# Sample" ```c# var channels = ChannelRangeSet.Parse("1,A-F"); var schemaMapping = await schemaMappingClient.QuerySchemaMappingAsync(new QuerySchemaMappingRequest { DataSource = dataSource, DataIdentity = dataIdentity, SelectChannels = channels.ToString(), SelectEvents = true }); ``` --- # Chunks !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Chunks The [RTA API](../../api/index.md) streams data to the client in _Chunks_. Each chunk represents one[^1] [Channel](../configuration/channels-parameters.md#channels), and covers a contiguous span of records. A chunk is the minimum unit of data returned by a request. Given the [Constraints](#constraints) listed below, clients must expect that a response will over-fetch beyond requested time-bounds. ## Column-Oriented Data This is an example of _column-oriented data_. Some databases and file formats work this way natively, such as [Apache Parquet](https://parquet.apache.org/). Many stores are _row-oriented_ — including most relational databases. ??? example Imagine a database table with a time column and three data columns: | Time | alpha | beta | gamma | |----------|-------|-------|-------| | 12:10:05 | `A01` | `B01` | `C01` | | 12:10:15 | `A02` | `B02` | `C02` | | 12:10:25 | `A03` | `B03` | `C03` | | 12:10:35 | `A04` | `B04` | `C04` | | 12:10:45 | `A05` | `B05` | `C05` | | 12:10:55 | `A06` | `B06` | `C06` | | 12:11:05 | `A07` | `B07` | `C07` | | 12:11:15 | `A08` | `B08` | `C08` | | 12:11:25 | `A09` | `B09` | `C09` | | 12:11:35 | `A10` | `B10` | `C10` | | 12:11:45 | `A11` | `B11` | `C11` | | 12:11:55 | `A12` | `B12` | `C12` | | 12:12:05 | `A13` | `B13` | `C13` | | 12:12:15 | `A14` | `B14` | `C14` | If we started collating records together into [Periodic Data](index.md#periodic-data), a single result might look like this: | Channel | Start Time | Samples | Result Buffer | |---------|------------|---------|----------------| | alpha | 12:10:05 | 3 |`[A01 A02 A03]` | A chunk is based on list types ([`PeriodicData` ➔ `PeriodicDataList`](index.md#periodic-data)), so each would contain multiple results. If each chunk covered one minute, requesting the _alpha_ channel could look like this: | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | | alpha | 12:12:05 | 12:12:15 | `[A13 A14]` | To turn most row-oriented data into chunks: 1. Collate samples into [`PeriodicData`](index.md#periodic-data) or [`TimestampedData`](index.md#timestamped-data) results; 2. Collate results into Lists (e.g. [`PeriodicData` ➔ `PeriodicDataList`](index.md#periodic-data)) and [encode to chunks](#using-the-api) as shown below. ## Using the API The _MAT.OCS.RTA.Model_ [NuGet Package](../../downloads.md#nuget-packages) has a `Chunk` class that holds the data in serialized and compressed form. For example: ```c# var data = new PeriodicDataList { PeriodicData = { // ... add results here } }; var chunkData = ChunkData.EncodePooled(ChunkDataMemoryPool.Shared, data, new[] {channelId}); var chunk = new Chunk(dataStartTime, dataEndTime, chunkData); ``` This applies fast LZ4 compression by default and uses a Memory Pool for low-overhead buffer reuse. !!! info "Compression" [Protobuf](https://developers.google.com/protocol-buffers) is already a compact binary serialization. Some parts of the data schema — such as [`TimestampedData`](index.md#timestamped-data) — use delta-compression so that protobuf [variable length encoding](https://developers.google.com/protocol-buffers/docs/encoding) produces a more compact output with more repeating byte sequences. These repeating sequences are then further compressed with LZ4, which is a very fast algorithm that typically achieves 2:1 compression on this data with no noticeable loss of throughput. Make sure you respect the [Constraints](#constraints) listed below. The _MAT.OCS.RTA.Services.AspNetCore_ [NuGet Package](../../downloads.md#nuget-packages) provides formatters to send a `ChunkedResult` (wrapping a `Chunk` stream) back to the client as the [`application/vnd.mat.protobuf+chunked` wire-format](../protobuf/net_chunks.md). ## Constraints This section describes some important constraints, using RFC 2119 language (MUST, SHOULD, etc). These are important if you are implementing a [Data Service](../../../../key-functionality/integrate/rta/introduction/data-services.md). ### Size Limits To ensure performance and reliability in all components: * Each result (e.g. [`PeriodicData`](index.md#periodic-data)) SHOULD contain up to 128 samples and MUST NOT exceed 10,000 samples. * Each list SHOULD serialize to no more than 64 KiB before compression and MUST NOT exceed 4 MiB. * Each list SHOULD cover 1-100 seconds and MUST NOT exceed 1000 seconds. ### Time Ordering Data MUST be returned in time order: * Samples MUST be ordered within results * Results MUST be ordered within chunks * Chunks MUST be ordered within channels The ordering of channels with respect to each other does not matter. For example, if returning data for _alpha_ and _beta_ in the [example above](#example), all of these orderings are valid: === "Example 1" | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | | alpha | 12:12:05 | 12:12:15 | `[A13 A14]` | | beta | 12:10:05 | 12:10:55 | `[B01 B02 B03]` `[B04 B05 B06]` | | beta | 12:11:05 | 12:11:55 | `[B07 B08 B09]` `[B10 B11 B12]` | | beta | 12:12:05 | 12:12:15 | `[B13 B14]` | === "Example 2" | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | beta | 12:10:05 | 12:10:55 | `[B01 B02 B03]` `[B04 B05 B06]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | | beta | 12:11:05 | 12:11:55 | `[B07 B08 B09]` `[B10 B11 B12]` | | alpha | 12:12:05 | 12:12:15 | `[A13 A14]` | | beta | 12:12:05 | 12:12:15 | `[B13 B14]` | === "Example 3" | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | beta | 12:10:05 | 12:10:55 | `[B01 B02 B03]` `[B04 B05 B06]` | | beta | 12:11:05 | 12:11:55 | `[B07 B08 B09]` `[B10 B11 B12]` | | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | beta | 12:12:05 | 12:12:15 | `[B13 B14]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | | alpha | 12:12:05 | 12:12:15 | `[A13 A14]` | This constraint enables a client to resume an interrupted download, since it can track progress through a request. ### No Overlaps When a session has reached a `closed` state: * Each result (e.g. [`PeriodicData`](index.md#periodic-data)) MUST NOT overlap any other result. * Each chunk MUST NOT overlap any other chunk. These constraints do not apply while a session is still `open`, since it may be difficult to maintain consistency between the [REST API](../../api/index.md) and [Live Streaming Data](../live/index.md), and new data may force re-chunking to maintain the [Size Limits](#size-limits) constraint. ### Stable Chunks Chunks MUST have stable boundaries once the session has closed. This means that if a sample is included in request, it always appears in the same chunk with the same start and end time. Using the [example above](#example): | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | | alpha | 12:12:05 | 12:12:15 | `[A13 A14]` | If the request bounds are _start: 12:10:10.00_, _end: 12:11:05.00_, the response is: | Channel | Start Time | End Time | Results | |---------|------------|----------|---------------------------------| | alpha | 12:10:05 | 12:10:55 | `[A01 A02 A03]` `[A04 A05 A06]` | | alpha | 12:11:05 | 12:11:55 | `[A07 A08 A09]` `[A10 A11 A12]` | This response includes samples outside the request bounds: * `A01` (12:10:05) is included because the unbounded response chunk boundary is 12:10:05 * `A08`-`A12` (12:11:15 - 12:11:55) are included because `A07` is inside the (inclusive) request bounds so the whole chunk is returned There are two main approaches to implement this: time-based chunking: : Round the request time-bounds (if any) outwards — generally just by dropping precision. The _MAT.OCS.RTA.Services_ [NuGet Package](../../downloads.md#nuget-packages) provides a `ChunkTime` utility to help with this. This is easy to implement but requires some knowledge of the expected data rate so the chunks are neither too small nor exceed the [Size Limits](#size-limits). Data rates could be communicated by convention, configuration or the [Schema Mapping Service](../../services/rta-schemamappingsvc/README.md). !!! important The chunk start and end time must still reflect the data contained inside the chunk, rather than the expanded time bounds. This is important so that clients can accurately assess what data they have, particularly at the leading edge of a live streaming session. Consecutive chunks should appear to have gaps between them if placed on a timeline. storage-based chunking: : Split the request based on storage pages: for example, pages within a memory-mapped file. This can be more flexible when data rates vary signficantly, but typically needs the data to be stored by time so the [Time Ordering](#time-ordering) constraint is not broken. !!! info This constraint is annoying to fulfil, and _may_ be dropped in future if it can be shown that it does not impact caching. For now, assume that this requirement is mandatory. [^1]: Except for [Row Data](index.md#row-data), which represents a set of [channels](../configuration/channels-parameters.md#channels) encoded into a packed buffer --- # Live-Streaming Basics !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Live Streaming Basics RTA live streaming enables a client to follow the leading edge of a live session without polling. !!! important RTA live streaming is designed for use _downstream_ from storage, rather than as an upstream telemetry distribution protocol. The [REST API](../../api/index.md) must be available to provide [Configuration](../configuration/index.md), and ATLAS expects to be able to retrieve the [Session](../sessions/index.md) and read any flushed [Data](../data/index.md). ## WebSocket Communication When a user loads a [Session](../sessions/index.md) from the [REST API](../../api/index.md), the client checks to see if the session is in the `#!json "open"` state. In that case, if a [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) URI has been configured, the client tries to open a connection using the [Session Identity](../sessions/model.md#required-properties) as part of the URI. A WebSocket URI uses the `ws://` or `wss://` scheme (equivalent to `http://` and `https://`). !!! example For session identity `abc123`: wss://example.com/rta/v2/sessions/abc123/stream The server immediately starts sending session metadata updates using a [simple protobuf-based protocol](websockets.md), and clients can send requests to subscribe to events and data. The flow of data is selective and predominantly unidirectional, so it is both efficient and insensitive to connection latency. ## Redis Communication The [Stream Service](../../services/rta-streamsvc/README.md) provides a reference implementation of the [WebSocket protocol](websockets.md) and receives its data for distribution via [Redis](https://redis.io/), which acts as a broker between the ingest points and the edge services. This decoupling helps ensure that edge services can be scaled and fault-tolerant, and ingest processes can be short-lived and unaffected by client activity. Data is carried using a [simple protocol](redis.md) built on [Redis Streams](https://redis.io/topics/streams-intro), which also provides buffering to cover the gap between flushed data — which should be available via the [REST API](../../api/index.md) — and the leading edge. This gap will vary depending on the storage technology: it could be milliseconds or hours. The _MAT.OCS.RTA.StreamBuffer_ [NuGet Package](../../downloads.md#nuget-packages) (.NET Core) provides an implementation for ingest processes to send data to Redis, and the [protocol](redis.md) is documented for the benefit of ingest pipelines written in other languages. ## Complications The live stream may not originate in the same process that writes data to persistent storage. This is likely when integrating with existing infrastucture or using off-the-shelf connectors. This raises several complications: _Difficult to synchronize session metadata_ : The process writing to persistent storage might have more metadata available than the process creating the live data stream. The client needs to mitigate this by combining metadata from the REST API and WebSocket using some heuristics, and there are some [guidelines to mitigate this issue](redis.md#session-metadata-updates). _Difficult to track how much data has been flushed_ : Many storage technologies do not guarantee immediate consistency, and separation between ingest processes can make it nearly impossible to determine when data has been flushed with perfect accuracy. If the flush point is not known, there cannot generally be a seamless join between the REST API and WebSocket streaming data. There are strategies that might mitigate this in specific situations — such as inserting checkpoints or tracking progress based on time — but in general it is more robust simply to configure the server environment to buffer stream data in Redis for a fixed period based on the known system characteristics. The client needs to mitigate this by merging data where there is an overlap. --- # WebSocket Protocol !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # WebSocket Protocol The ATLAS client opens a dedicated [WebSocket](https://datatracker.ietf.org/doc/html/rfc6455) connection for each [Session](../sessions/index.md) in the `#!json "open"` state. All messages are binary payloads, with the content serialized according to the [net_stream.proto](../protobuf/net_stream.md) file. As soon as the client connects, the server sends the current [Session JSON](../sessions/index.md), and then the current session time-range and any subsequent updates. No events or data are sent until the client subscribes to them. The client can send a subscription request at any time, specifying channel numbers for selective data delivery. This widens any existing subscription. The server immediately starts following the leading edge of the session, but should also send data to backfill between the data that is available from the [REST API](../../api/index.md) up to the leading edge. The duration of this back-scan is configured server-side — and since it's unlikely to be a precise join, the client should deal with some overlap. Streamed data is not necessarily in time order: the data may arrive out of order at the ingest point, and back-fill will send data behind the leading edge. ## Connecting The URI structure is: wss://example.com/rta/v2/sessions/{identity}/stream The client must request this [sub-protocol](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#subprotocols): mat.ocs.rta.stream-lz4 ??? info "HTTP Headers" ``` GET /rta/v2/sessions/{identity}/stream HTTP/1.1 Host: example.com:443 Upgrade: websocket Connection: Upgrade Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== Sec-WebSocket-Version: 13 Sec-WebSocket-Protocol: mat.ocs.rta.stream-lz4 ``` Almost all servers and clients only support WebSockets over HTTP/1.1 Error Responses: `400 Bad Request` : If the request is not a WebSocket connection upgrade. `406 Not Acceptable` : If the request does not specify the `mat.ocs.rta.stream-lz4` sub-protocol. The initial handshake should complete successfully even if there is no session to stream. ## Initial Server Response The first message from the server must contain only the current Session JSON. If the session is not available to stream, the server cannot send the Session JSON and initiates a normal [WebSocket close handshake](https://datatracker.ietf.org/doc/html/rfc6455#section-7.1.2) without sending any other messages. The client should back-off and retry after confirming the session is still open by polling the [REST API](../../api/index.md). ## Server Messages Server messages are all `StreamDataBurst`: ```protobuf message StreamDataBurst { repeated StreamData data = 1; } ``` Each burst contains one or more `StreamData` messages, carrying: * Session metadata updates * Session time-range updates * Events and sample data ```protobuf // Base message for all stream data. message StreamData { reserved 1; oneof data { // Session JSON update. The format must match the web service specification. // This is the first message downloaded by a new web socket connection, so it is not possible // to establish a stream session until at least one Session JSON update has been streamed. // If collections of details, laps, config bindings etc are null - not empty - it is assumed // they are unavailable and they will be merged in from the Session JSON downloaded from the service. string session_json = 2; // Time range update. // This needs to be sent regularly so that the data viewer can pan smoothly to show // activity, even if no data subscription has been applied. // The overall session time range is expanded to cover this update. StreamTimeRange time_range = 3; // Timestamped Data rta.model.data.TimestampedData timestamped_data = 4; // Periodic Data rta.model.data.PeriodicData periodic_data = 5; // Row Data rta.model.data.RowData row_data = 6; // Event rta.model.data.Event event = 7; } } // Updates the session time range. message StreamTimeRange { // Start time (inclusive) in nanoseconds relative to the Unix epoch. sfixed64 start_time = 1; // End time (inclusive) in nanoseconds relative to the Unix epoch. sfixed64 end_time = 2; } ``` `SessionTimeRange` updates are sent at fairly high frequency (typically up to 10 Hz) to ensure the client can smoothly scroll displays to reflect live activity, even if the user is displaying data that is not updated that often. The serialized `StreamDataBurst` is compressed in a block using [LZ4](https://github.com/lz4/lz4). This burst representation is important for network efficiency, as the individual `StreamData` messages may be quite small and there may be significant opportunity for compression across a series of messages. The server will size the bursts to be about 64 KiB (before compression) without introducing significant latency. ## Client Messages Client messages are all `StreamRequest`: ```protobuf message StreamRequest { reserved 1; oneof option { // Widens the subscription. StreamSubscription subscribe = 2; } } ``` The `oneof option` makes the request extensible, but currently only defines a subscription request: ```protobuf message StreamSubscription { // Events mask. bool all_events = 1; // All data channels mask. bool all_data_channels = 2; // Specific data channels mask. repeated uint32 data_channel_ids = 3; } ``` Client messages are not compressed. ## Closing the Connection The WebSocket protocol defines a [close handshake](https://datatracker.ietf.org/doc/html/rfc6455#section-7) that must be followed. In practice, this means that the client requests closure and should continue to read messages until the server sends a corresponding close message — and the server may initiate closure at any time. --- # Streaming to Redis !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Streaming to Redis [Redis Streams](https://redis.io/topics/streams-intro) provides a buffer between telemetry ingest and the [Stream Service](../../services/rta-streamsvc/README.md) implementing the [WebSocket protocol](websockets.md). ???+ info "Why Redis?" [Redis Streams](https://redis.io/topics/streams-intro) is a similar design to [Apache Kafka](https://kafka.apache.org/). [Redis](https://redis.io/) was chosen because: * it's generally considered to be easier to deploy and manage, with few configuration options * it's easier to implement lookups for the current [Session JSON](../sessions/index.md) and time-range * it's then also available for caching data from the [REST API](../../api/index.md) (likely to be needed at some point) [Redis](https://redis.io/) can be run on Windows using [Docker](https://hub.docker.com/_/redis). [Redis](https://redis.io/) provides the communication layer, and this page defines a protocol layered over the top. !!! info This protocol is implemented by the _MAT.OCS.RTA.StreamBuffer_ [NuGet Package](../../downloads.md#nuget-packages) (.NET Core). ## Stream Management Messages are sent to named [Redis Streams](https://redis.io/topics/streams-intro). As a guideline, create a new stream for each source of data, and sub-divide further as needed. You can have as few or as many named streams as you like, but: _Streams need to be trimmed so the buffer doesn't grow indefinitely_ : This is done automatically by the [Stream Service](../../services/rta-streamsvc/README.md), but doesn't work effectively if new streams are being constantly created. They should be persistent. _Streams can't be read selectively_ : The [Stream Service](../../services/rta-streamsvc/README.md) will need to consume all data from a stream — even if it has a single client subscribed to a single session. If there are too many concurrent sessions, this becomes inefficient. As an example, if the source of the data is [Kafka](https://kafka.apache.org/), consider creating a stream for each topic partition. ## Session Metadata Updates ATLAS will not show a [Session](../sessions/index.md) in the Session Browser until it is available from the REST API, along with [Configuration](../configuration/index.md). The session needs to be in the `#!json "open"` state and have [configuration bindings](../sessions/model.md#configuration-bindings). The [WebSocket protocol](websockets.md) also requires the Session JSON to be available before any data can be streamed to a client, which means that the Session needs to be published _both_ via the REST API and via Redis and the WebSocket protocol. It might be difficult to keep these consistent — particularly if the ingest pipeline is split into different processes. Here are some guidelines: 1. Make sure your ingest process will publish the Session via the REST API promptly. If it's a slow-starting batch process, consider publishing a record (e.g. to the [Session Service](../../services/rta-sessionsvc/grpc.md)) at the start of live streaming — which can then be updated. 2. The [Session JSON model](../sessions/model.md) can leave the [details](../sessions/model.md#details-and-extended-details) and [configuration bindings](../sessions/model.md#configuration-bindings) set to `#!json null` in the stream message to ensure the client uses the version in the REST API — or vice versa. 3. The [Session time-range](../sessions/model.md/#time-range) does not need to be updated in the JSON at high frequency since there is a dedicated stream protocol message for this aspect. But it should ideally be updated in the REST API every 5-30 seconds so progress is indicated in the ATLAS Session Browser. 4. Make sure the Session state is updated to `#!json "closed"` in the stream and ultimately in the REST API. The WebSocket protocol is designed so that it doesn't matter if the two are slightly out of sync. ## Protocol There are three Redis data structures to update: * `stream:*` keys are [Streams](https://redis.io/topics/streams-intro) carrying compressed `StreamDataBurst` messages * `streams` is a [Set](https://redis.io/topics/data-types#sets) listing all the Streams * `sessions` is a [Hash](https://redis.io/topics/data-types#hashes) locating each Session and containing the latest JSON model and time-range ### Streams Follow a naming convention where each Redis Stream is named with the `stream:` prefix. For example: `rocket-engine` becomes `stream:rocket-engine` A Redis Message is made up of name-value pairs. For this protocol, they are: * [`"id"`](#message-id) — distinguishing the session * [`"data"`](#message-data) — LZ4-compressed protobuf payload The [`XADD` command](https://redis.io/commands/xadd) appends a message to a stream. #### Message `"id"` A Redis Stream can carry multiple concurrent [Sessions](../sessions/index.md), so each message has an `"id"` field specifying a stream session ID. This is a simple number. It should be tracked in a key named `lastStreamSessionId` and atomically polled and incremented using the [`INCRBY` command](https://redis.io/commands/incrby). #### Message `"data"` The core message payload in the `"data"` field is an [LZ4](https://github.com/lz4/lz4)-compressed [`StreamDataBurst`](../protobuf/net_stream.md), as described for the [WebSocket protocol](websockets.md#server-messages). !!! important Aim for messages that are roughly 64 KiB (before compression), and do not exceed 1 MiB. This ensures that the messages contain enough data that the protocol overhead is insignificant, but are small enough that the [Stream Service](../../services/rta-streamsvc/README.md) can process data efficiently by reusing memory buffers. ### Streams Set Maintain a `streams` [Set](https://redis.io/topics/data-types#sets) containing all stream names (without the `stream:` prefix). This enables the [Stream Service](../../services/rta-streamsvc/README.md) to discover all streams (for trimming) without scanning for keys — which is discouraged for clustered deployments. The [`SADD` command](https://redis.io/commands/sadd) adds an entry to a Set. ### Sessions Hash Maintain a `sessions` [Hash](https://redis.io/topics/data-types#hashes) containing a description of all sessions. Each entry should use the [Session Identity](../sessions/model.md#required-properties) as the key, and must define the following value fields: `"streamSessionId"` : * Corresponds to the `"id"` field in each stream message * Enables the [Stream Service](../../services/rta-streamsvc/README.md) to filter for the correct session `"streamName"` : * Corresponds to the stream name containing the session (without the `stream:` prefix) * Enables the [Stream Service](../../services/rta-streamsvc/README.md) to subscribe to the correct stream `"streamPosition"` : * Current stream position when the first stream message was appended, or `0-0` if the stream is new * Enables the [Stream Service](../../services/rta-streamsvc/README.md) to scan fewer messages when catching up to the leading edge `"lastUpdated"` : * Time (milliseconds since Unix epoch) this Hash was last updated * Enables a cleanup job to identify streams and sessions that haven't updated in a long time Every time a message is streamed to update the Session JSON, this field must be added or updated: `"json"` : * Session JSON * Enables the [Stream Service](../../services/rta-streamsvc/README.md) to give the client the latest Session model when it connects Every time a message is streamed to update the Session time-range, these fields must be added or updated as a pair: `"tStart"` : * Earliest timestamp in the Session, in nanoseconds since the Unix epoch `"tEnd"` : * Latest timestamp in the Session, in nanoseconds since the Unix epoch !!! important A Session is not ready to be streamed via the [WebSocket protocol](websockets.md) until the `"json"` field is available. It is not necessary to keep the time-range up to date in the JSON since this is covered by `"tStart"` and `"tEnd"`, but the JSON should be updated periodically in the REST API so users can see an indication of activity in the ATLAS Session Browser. The [`HMSET` command](https://redis.io/commands/hmset) updates the fields for a Hash entry. Entries in the `sessions` hash should be set to expire automatically after a reasonable interval. --- # Overview !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Protobuf Schemas RTA models size/performance-sensitive data in [Google Protocol Buffers](https://developers.google.com/protocol-buffers). ??? info "Why not _X?_" Some alternative frameworks are starting to gain traction, including: * [cap'n'proto](https://capnproto.org/) * [FlatBuffers](https://google.github.io/flatbuffers/) * [Apache Arrow](https://arrow.apache.org/) These achieve even better performance than Protobuf by using zero-copy data structures instead of techniques like variable-length integer compression. The principle is that the data serialization is larger, but still amenable to fast compression (e.g. with [LZ4](https://github.com/lz4/lz4)). Protobuf was chosen because it has a great mix of features, maturity, usability, RPC support, language support, and adoption — combined with good performance. Microsoft's investment in making gRPC a first-class citizen in Visual Studio and ASP.NET Core has been a major deciding factor. Memory allocation _is_ still a noticeable factor on a profiler — but typically only when all other sources of allocation and inefficiency have been eliminated. These schemas are pre-compiled in the _MAT.OCS.RTA.Model_ library, which is available via the Motion Applied [NuGet Packages](../../downloads.md#nuget-packages). If you need to compile them yourself for your preferred language, create this directory structure: * `protos/` * `API/` * [`model_data.proto`](model_data.md) * [`net_chunks.proto`](net_chunks.md) * [`net_stream.proto`](net_stream.md) Refer to the [protobuf documentation](https://developers.google.com/protocol-buffers/docs/overview#generating) for `protoc` instructions. --- # model_data.proto !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # rta.model.data This IDL describes all the types described in [Data Basics](../data/index.md). `model_data.proto` ``` protobuf --8<-- ../protos/API/model_data.proto --8<-- ``` --- # net_chunks.proto !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # rta.model.net.chunks This IDL is used as part of the protocol for downloading data from the [REST API](../../api/index.md#tag/Data-API). Each request transfers data in chunks. Wire format for each chunk (concatenated in a stream): | Field | Bytes | Data type | Description | |---------|---------------|--------------------------|--------------------------------------------------------------------------| | prefix | 4 | signed little-endian int | length of header | | header | _prefix_ | protobuf | `ChunkTransferHeader` — including `packed_length` (of the payload) | | payload | _from header_ | protobuf | _data list_ | `net_chunks.proto` ``` protobuf --8<-- ../protos/API/net_chunks.proto --8<-- ``` The `compression_delta` indicates LZ4=1 on the first chunk if it is used (strongly recommended). --- # net_stream.proto !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # rta.model.net.stream This IDL is used in [WebSocket](https://en.wikipedia.org/wiki/WebSocket) communications for live streaming data from the [Stream Service](../../services/rta-streamsvc/README.md). When the client initially connects, the server begins sending `StreamDataBurst` (_binary_) messages. These intially include only `session_json` and `time_range` updates, but the client can widen its subscription to include data by sending `StreamRequest` messages. ``` protobuf --8<-- ../protos/API/net_stream.proto --8<-- ``` --- # Overview !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Useful Utilities ## gRPC UI [gRPC UI](https://github.com/fullstorydev/grpcui) is exceptionally useful for exploring the [Toolkit Services](../services/index.md). Each service is self-describing, so you only need to connect to the service on the gRPC port, and create calls using a web browser. For example, connecting to the [Session Service](../services/rta-sessionsvc/README.md) on the default port: grpcui -plaintext localhost:2652 ## Postman and Insomnia [Postman](https://www.postman.com/) is a convenient tool and popular tool for testing REST web services — such as the RTA API. [Insomnia](https://insomnia.rest/) is a popular alternative. ## httpstat [httpstat](https://github.com/davecheney/httpstat) is useful for benchmarking — particularly when developing a [Data Service](../../../key-functionality/integrate/rta/introduction/data-services.md). ```bash httpstat -H "Accept: application/vnd.mat.protobuf+chunked" -o output.bin http://localhost/rta/v2/sessions/abc123/data/timestamped/0-7f ``` ``` HTTP/1.1 200 OK Server: Kestrel Content-Type: application/vnd.mat.protobuf+chunked Date: Wed, 28 Oct 2020 13:16:59 GMT Body read DNS Lookup TCP Connection Server Processing Content Transfer [ 0ms | 5ms | 396ms | 14388ms ] | | | | namelookup:0ms | | | connect:5ms | | starttransfer:402ms | total:14790ms ``` ## Protocol Buffers Compiler [protoc](https://github.com/protocolbuffers/protobuf/releases) will generate idiomatic data classes and serializers in a range of languages. !!! tip If you are working in .NET, Visual Studio has [integrated support for compiling protobuf](https://docs.microsoft.com/en-us/aspnet/core/grpc/dotnet-grpc) as part of your project. This works very well and is highly recommended. --- # Downloads !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Downloads ## Toolkit Services These [services](services/index.md) provide an implementation toolkit covering some or all of the functionality in the [RTA API Specification](api/index.md). Combined with the [libraries](#nuget-packages) to support [Data Service](../../key-functionality/integrate/rta/introduction/data-services.md) development, this can significantly reduce integration work. ### Docker Images are available from [Docker Hub](https://hub.docker.com/u/mclarenapplied): * [RTA Server](https://hub.docker.com/r/mclarenapplied/rta-server) — `mclarenapplied/rta-server` * [RTA Session Service](https://hub.docker.com/r/mclarenapplied/rta-sessionsvc) — `mclarenapplied/rta-sessionsvc` * [RTA Config Service](https://hub.docker.com/r/mclarenapplied/rta-configsvc) — `mclarenapplied/rta-configsvc` * [RTA Data Service](https://hub.docker.com/r/mclarenapplied/rta-datasvc) — `mclarenapplied/rta-datasvc` * [RTA Gateway Service](https://hub.docker.com/r/mclarenapplied/rta-gatewaysvc) — `mclarenapplied/rta-gatewaysvc` * [RTA Stream Service](https://hub.docker.com/r/mclarenapplied/rta-streamsvc) — `mclarenapplied/rta-streamsvc` * [RTA Schema Mapping Service](https://hub.docker.com/r/mclarenapplied/rta-schemamappingsvc) — `mclarenapplied/rta-schemamappingsvc` * [RTA Influx Data Service](https://hub.docker.com/r/mclarenapplied/rta-influxdatasvc) — `mclarenapplied/rta-influxdatasvc` ### Binaries * v2.0.0 — [Windows (64-bit)](https://github.com/mat-docs/RTA.Releases/releases/download/v2.0.0/toolkit-win-x64.zip) / [Linux (64-bit)](https://github.com/mat-docs/RTA.Releases/releases/download/v2.0.0/toolkit-linux-x64.zip) ## Code Samples Clone the sample project from [RTA.Examples](https://github.com/mat-docs/RTA.Examples) on GitHub. You will need a free GitHub account. ## NuGet Packages These .NET libraries reduce development effort to build and test an RTA service: MAT.OCS.RTA.API : Client API for testing and integration; Supports .NET Standard 2.0 and 2.1 MAT.OCS.RTA.Services.AspNetCore : By far the quickest way to get a new RTA service off the ground; Supports ASP.NET Core 2.1, 3.1 and 5.0 MAT.OCS.RTA.Toolkit.API.GrpcClients : Pre-compiled gRPC clients for the Toolkit Services; Supports .NET Standard 2.1 and .NET 5.0 Refer to the [Developer Guide](worked-guide/index.md) for more details. These libraries are all available from our [GitHub packages feed](https://github.com/mat-docs/packages). A free [GitHub](https://github.com/) account is required. === "RTA Libraries" | Package | Description | |-------------------------------------|----------------------------------------------------------------------------| | MAT.OCS.RTA.API | RTA client library, supporting the RTA REST API and WebSocket interface. | | MAT.OCS.RTA.Model | RTA data model, including classes modelling the API responses. | | MAT.OCS.RTA.Services | Service interfaces to be implemented by customer-specific backends. | | MAT.OCS.RTA.Services.AspNetCore | ASP.NET API Controllers and Formatters implementing the RTA specification. | | MAT.OCS.RTA.Toolkit.API.GrpcClients | Pre-compiled gRPC clients for use from services or utilities. | | MAT.OCS.RTA.Toolkit.API.GrpcServers | Pre-compiled gRPC servers/clients for use from services only. | | MAT.OCS.RTA.StreamBuffer | RTA stream buffering via Redis. | === "Supporting Libraries" | Package | Description | |-------------------------------------|----------------------------------------------------------------------------| | MAT.OCS.Configuration | ATLAS configuration model, including a builder API. | | MAT.OCS.Configuration.Json | Binds the configuration model to JSON. | | MAT.OCS.FFC | Fast Configuration API - a binary configuration format (FFC). | | MAT.OCS.FFC.Configuration.Format | Fast Configuration implementation. | | MAT.OCS.FFC.Configuration | Reads Fast Configuration (FFC) resources as MAT.OCS.Configuration. | | MAT.OCS.FFC.ConfigSerializer | Serializes MAT.OCS.Configuration models as Fast Configuration (FFC). | --- # FAQ !!! danger "Prototype Functionality" RTA was released as prototype functionality. As such, the content in these pages is subject to change based on feedback and roadmap changes. # Frequently Asked Questions (FAQ) ## Q: Do I need to write my integration in C#? No — the [API Specification](../api/index.md) is language-neutral. The [Toolkit Services](../services/index.md) communicate via [gRPC](https://grpc.io/), so they are also language-neutral. However, we do provide [NuGet Packages](../downloads.md/#nuget-packages) that will definitely speed up your development. They provide: * Model classes to help with JSON serialization * REST API stubs and service interfaces to help write back-end services * ASP.NET implementations of the network protocol — e.g. transferring data chunks * Pre-compiled gRPC clients for the services ## Q: Do I need to use the [Toolkit Services](../services/index.md)? No — but they can significantly reduce development work. The RTA session model can be awkward to retrofit into an existing environment, so you can save a lot of time by dropping-in a self-contained implementation. The [Session Service](../services/rta-sessionsvc/README.md) provides support for all the RTA features, including: * all session metadata, including extended details * query support, trans-piling requests to SQL * filter support, by producing the session metamodel * describing relationships between sessions * providing extra indirection to compose data from different services Adding self-contained microservices and some gRPC integration should be significantly less intrusive for your data pipeline. You _can_ probably achieve a more seamless integration by implementing the [RTA API Specification](../api/index.md) directly against stores in your data environment, which will reduce data duplication and synchronization logic. We recommend starting out with the implementation toolkit and reviewing after initial integration is complete. ## Q: Can I do data queries using this interface? No — only metadata queries (which could include aggregate values). RTA will happily integate into environments that do have more sophisticated query services. --- # News and Release Notes # News and Release Notes --- # 2026 # 2026 --- # 2025 # 2025 --- # Announcement # Announcement --- # Blog # Blog --- # Releases # Releases --- # Profiling ## Overview From version 11.2.3.460 onwards, ATLAS application now includes Jet Brains dotTrace and dotMemory profilers, which can be accessed via the Tools menu. This function allows users to take snapshots, which can be shared with the ATLAS development team for debugging purposes. Please use this feature only when specifically requested by the development team. !!! warning The profilers capture a large amount of information and will generate a large file. Please limit the amount of time metrics are being captured to minimise the file size. ## Capturing Performance Profile 1. Navigate to `Tools > Performance Profiler > Performance > Start`. ![PerformanceToolsMenu.png](assets%2Fprofiling%2FPerformanceToolsMenu.png) 2. Wait for the status bar to display `Performance Profiler Started`. ![StartingPerformance.png](assets%2Fprofiling%2FStartingPerformance.png) ![StartedPerformance.png](assets%2Fprofiling%2FStartedPerformance.png) 3. Perform the actions that are required to be profiled. 4. Stop the profiler by navigating back through the Tools menu. 5. Navigate to the snapshot folder via `Tools > Performance Profiler > View Snapshot Folder`. 6. Send the \*.dtp and the \*.dtp.000# to your Motion Applied support representative. ## Capturing Memory Profile 1. Navigate to `Tools > Performance Profiler > Memory > Start`. ![MemoryToolsMenu.png](assets%2Fprofiling%2FMemoryToolsMenu.png) 2. Wait for the status bar to display `Memory Profiler Started`. ![StartingMemory.png](assets%2Fprofiling%2FStartingMemory.png) ![StartedMemory.png](assets%2Fprofiling%2FStartedMemory.png) 3. Perform the actions that are required to be profiled. 4. Create snapshot as required via `Tools > Performance Profiler > Memory > Snapshot`. 5. Stop the profiler by navigating back through the Tools menu. 6. Navigate to the snapshot folder via `Tools > Performance Profiler > View Snapshot Folder` 7. Send the *.dmw to your Motion Applied support representative.