Understanding PMTs with Flatbuffers and Modern C++

Explore the use of Polymorphic Types (PMTs) with Flatbuffers and modern C++ to serialize complex data structures for efficient transmission between systems. Learn about the benefits of PMTs, their primary uses in GNU Radio, and how Flatbuffers offer a fast and efficient serialization format. Discover the evolution and advantages of using Flatbuffers in structuring data for better performance and compatibility.

• PMTs
• Flatbuffers
• Modern C++
• Serialization

lnesb Follow

Uploaded on Sep 16, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. PMTs with Flatbuffers and modern C++ John Sallay Josh Morman

2. Roadmap PMT Overview Data Serialization Modern C++ The New PMT Classes Next Steps

3. What is a PMT? Polymorphic Type Used to group arbitrary data types together. Think of a python dictionary or json. Includes serialization/deserialization functions. Allows us to send data over the network.

4. Why is a PMT? Primary uses in GNU Radio are: Tag samples in the data stream USRP rx_time tag Integer seconds uint64_t Fractional seconds - double Send messages between blocks Burst Detection Message (will include other fields) Modulation type string Bandwidth double Burst ID - integer

5. The interface can be inconsistent and hard to understand. Signed integer functions: pmt::from_long() and pmt:is_integer() Some common operations can require a lot of function calls with hard to remember names. Why do something new? Slow Performance PMT's are a custom idea only used in GNU Radio Many other projects have been thinking about similar problems over the past decade. We hope to leverage other work.

6. Serialization Transmitting complex data structures between systems, we need an invertible representation. PMT s provide serialization. Other Common Options Json Store data as strings. Easy to read/write. Poor performance due to string conversions. Protobufs Store data in predefined structures. Serialized format is dynamic, which saves space but has a performance cost. Flatbuffers Store data in serialized format. Fastest performance.

7. What are Flatbuffers? Memory and computationally efficient structure serialization format. Open Source and maintained by Google. Data is stored in the serialized format. No cost to convert data back and forth. Can access (deserialize) any element without having to analyze other elements. Similar to accessing a structure in C++. Structures can be used in C++, python, and most other popular languages. Structures can define optional and repeated fields Supports PMT data structures like maps and vectors.

8. Flatbuffers: Structure Evolution Allows for forwards and backwards compatibility of structures. Can add/deprecate structure fields. Older structure versions can still communicate with newer versions. Add A Field

9. How to Use Flatbuffers Create a schema file. Defines data structures and types. Similar to json schema. https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html Compile the schema. flatc is the Flatbuffers schema compiler. Produces header files that can be imported into c++ code. Can also generate code for most other popular languages. Include the generated header in your code.

10. Modern C++ Poorly defined term. Python dict for loop Usually defined as using features from c++11 and on. I think of it as - making c++ work more like python. Common Concepts: Range based for loop. C++ dict for loop Using templates to allow functions to be called with any valid data type. Declare complex variable types using auto keyword.

11. PMT Project and Classes Standalone PMT project under GNU Radio gnuradio/pmt: pmt (github.com) Can be built/used independent of any other GNU Radio source. Feel free to contribute. PMT Classes Define a PMT as a wrapper object around a flatbuffer. The wrapper defines "normal" functions and works like other c++ containers. Templated design should be extensible to many data structures.

12. PMT Scalar Wraps a single value. Includes signed/unsigned integers, floating point, and complex values. API allows for easy construction and for getting and setting the value.

13. PMT Vector Represents a contiguous array of values of the same underlying type. Analogous to an STL vector. API matches the STL vector API

14. PMT Map Represents a collection of key value pairs in an easily accessible format. For now, the keys must be strings. In the future, we hope to make it possible to use any PMT for the key. The value can be any PMT including scalars, vectors, other maps, etc.

15. Future Types The library is structured so that new PMT types can be added without having to modify existing types. PMT String Wrapper around character data. API should match STL string PMT Vector (Not uniform data) Stores an array of other PMTs. Similar to a python list No plan to support some current PMT types (such as pair and linked list)

16. Formatted Types Future Capability PMT messages can contain arbitrary data and types. In many cases it would be nice to lock down the format and have strong type checking on input and output PMTs. Like applying a json schema to incoming messages and tags. Flatbuffers can help Flatbuffers are serialized into packed structures type checking is free. Block definition would include input/output Flatbuffers structures. Note that data validation is a hard problem. For some applications, further validation will be needed for received data.

17. PMT Performance The new interface is easier to use but can t be slower than current code. We have done a few limited tests on PMT vectors and maps. In my experience, these are the most commonly used types. Vector Serialization results New code slower for large vectors Identified extra work that can be cut out.

18. PMT Performance PMT Map Element Access Flat lines at the bottom are new code. Sloped lines is the current code. Current implementation is very inefficient New code is significantly faster.

19. Where are we going from here? Code still needs some work to be ready for GNU Radio inclusion. Make sure that API matches STL containers. Add PMT types from previous slides. Make it easy to work with PMTs outside of GNU Radio. Flatbuffers libraries exist for many languages. We could write wrapper libraries for other languages. This would require a lot of maintenance and language knowledge. Alternatively write GNU Radio blocks that work with arbitrary Flatbuffers messages.

20. Web Server External Interface Example We could have a web service that receives user input. It produces a defined Flatbuffers message and sends it to a message broker or server. GNU Radio block accepts message and converts to Pmt. Message is passed to blocks in the flowgraph for processing. GNU Radio Flowgraph JavaScript based webservice Flatbuffers Receiver (Message Broker/Server) Rest of Flowgraph

21. Questions??? If you would like to contribute gnuradio/pmt: pmt (github.com)