Eric Blossom

eb@comsec.com

0.1 2005-01-20 0.2 2005-02-02 Updated for SWIG 1.3.24 0.3 2006-07-21 Clarification of 1:1 fixed rate vs item size 2004 2005 Free Software Foundation, Inc. This article explains how to write signal processing blocks for GNU Radio. This article assumes that the reader has basic familiarity with GNU Radio and has read and understood Exploring GNU Radio. There is a tarball of files that accompany this article. It includes the examples, DocBook source for the article and all the Makefiles etc it takes to make it work. Grab it at ftp://ftp.gnu.org/gnu/gnuradio or one of the mirrors. The file you want is gr-howto-write-a-block-X.Y.tar.gz. Pick the one with the highest version number. See http://comsec.com/wiki?CvsAccess for CVS Access. &gnuradio; provides a framework for building software radios. Waveforms -- signal processing applications -- are built using a combination of Python code for high level organization, policy, GUI and other non performance-critical functions, while performance critical signal processing blocks are written in C++. From the Python point of view, &gnuradio; provides a data flow abstraction. The fundamental concepts are signal processing blocks and the connections between them. This abstraction is implemented by the Python gr.flow_graph class. Each block has a set of input ports and output ports. Each port has an associated data type. The most common port types are float and gr_complex (equivalent to std::complex<float>), though other types are used, including those representing structures, arrays or other types of packetized data. From the high level point-of-view, infinite streams of data flow through the ports. At the C++ level, streams are dealt with in convenient sized pieces, represented as contiguous arrays of the underlying type. This article will walk through the construction of several simple signal processing blocks, and explain the techniques and idioms used. Later sections cover debugging signal processing blocks in the mixed Python/C++ environment and performance measurement and optimization. The example blocks will be built in the style of all &gnuradio; extensions. That is, they are built outside of the gnuradio-core build tree, and are constructed as shared libraries that may be dynamically loaded into Python using the "import" mechanism. &SWIG;, the Simplified Wrapper and Interface Generator, is used to generate the glue that allows our code to be used from Python. The C++ class &gr_block; is the base of all signal processing blocks in &gnuradio;. Writing a new signal processing block involves creating 3 files: The .h and .cc files that define the new class and the .i file that tells &SWIG; how to generate the glue that binds the class into Python. The new class must derive from &gr_block; or one of it's subclasses. Our first examples will derive directly from &gr_block;. Later we will look at some other subclasses that simplify the process for common cases. Before we dive into the code, &lets; talk a bit about the overall build environment and the directory structure that &well; be using. To reduce the amount of Makefile hacking that we have to do, and to facilitate portability across a variety of systems, we use the GNU autoconf, automake, and libtool tools. These are collectively referred to as the autotools, and once you get over the initial shock, they will become your friends. (The good news is that we provide boilerplate that can be used pretty much as-is.) automake automake and configure work together to generate GNU compliant Makefiles from a much higher level description contained in the corresponding Makefile.am file. Makefile.am specifies the libraries and programs to build and the source files that compose each. Automake reads Makefile.am and produces Makefile.in. Configure reads Makefile.in and produces Makefile. The resulting Makefile contains a zillion rules that do the right right thing to build, check and install your code. It is not uncommon for the the resulting Makefile to be 5 or 6 times larger than Makefile.am. autoconf autoconf reads configure.ac and produces the configure shell script. configure automatically tests for features of the underlying system and sets a bunch of variables and defines that can be used in the Makefiles and your C++ code to conditionalize the build. If features are required but not found, configure will output an error message and stop. libtool libtool works behind the scenes and provides the magic to construct shared libraries on a wide variety of systems. shows the directory layout and common files &well; be using. After renaming the topdir directory, use it in your projects too. We'll talk about particular files as they come up later. File/Dir Name Comment topdir/Makefile.am Top level Makefile.am topdir/Makefile.common Common fragment included in sub-Makefiles topdir/bootstrap Runs autoconf, automake, libtool first time through topdir/config Directory of m4 macros used by configure.ac topdir/configure.ac Input to autoconf topdir/src topdir/src/lib C++ code goes here topdir/src/lib/Makefile.am topdir/src/python Python code goes here topdir/src/python/Makefile.am topdir/src/python/run_tests Script to run tests in the build tree &gnuradio; uses a set of naming conventions to assist in comprehending the code base and gluing C++ and Python together. Please follow them. We've returned to a kinder, gentler era. We're now using the "STL style" naming convention with a couple of modifications since we're not using namespaces. With the exception of macros and other constant values, all identifiers shall be lower case with words_separated_like_this. Macros and constant values (e.g., enumerated values, static const int FOO = 23) shall be in UPPER_CASE. All globally visible names (types, functions, variables, consts, etc) shall begin with a "package prefix", followed by an underscore. The bulk of the code in GNU Radio belongs to the "gr" package, hence names look like gr_open_file (...). Large coherent bodies of code may use other package prefixes, but let's try to keep them to a well thought out list. See the list below. These are the current package prefixes: gr_ Almost everything. gri_ Implementation primitives. Sometimes we have both a gr_foo and a gri_foo. In that case, gr_foo would be derived from gr_block and gri_foo would be the low level guts of the function. atsc_ Code related to the Advanced Television Standards Committee HDTV implementation usrp_ Universal Software Radio Peripheral. qa_ Quality Assurance (Test code.) All class data members shall begin with d_foo. The big win is when you're staring at a block of code it's obvious which of the things being assigned to persist outside of the block. This also keeps you from having to be creative with parameter names for methods and constructors. You just use the same name as the instance variable, without the d_. class gr_wonderfulness { std::string d_name; double d_wonderfulness_factor; public: gr_wonderfulness (std::string name, double wonderfulness_factor) : d_name (name), d_wonderfulness_factor (wonderfulness_factor) { ... } ... }; All class static data members shall begin with s_foo. Each significant class shall be contained in its own file. The declaration of class gr_foo shall be in gr_foo.h and the definition in gr_foo.cc. By convention, we encode the input and output types of signal processing blocks in their name using suffixes. The suffix is typically one or two characters long. Source and sinks have single character suffixes. Regular blocks that have both inputs and outputs have two character suffixes. The first character indicates the type of the input streams, the second indicates the type of the output streams. FIR filter blocks have a three character suffix, indicating the type of the inputs, outputs and taps, respectively. These are the suffix characters and their interpretations: f - single precision floating point c - complex<float> s - short (16-bit integer) i - integer (32-bit integer) In addition, for those cases where the block deals with streams of vectors, we use the character 'v' as the first character of the suffix. An example of this usage is gr_fft_vcc. The FFT block takes a vector of complex numbers on its input and produces a vector of complex numbers on its output. For our first example &well; create a block that computes the square of its single float input. This block will accept a single float input stream and produce a single float output stream. Following the naming conventions, &well; use howto as our package prefix, and the block will be called howto_square_ff. We are going to arrange that this block, as well as the others that we write in this article, end up in the gnuradio.howto Python module. This will allow us to access it from Python like this: from gnuradio import howto sqr = howto.square_ff () We could just start banging out the C++ code, but being highly evolved modern programmers, &were; going to write the test code first. After all, we do have a good spec for the behavior: take a single stream of floats as the input and produce a single stream of floats as the output. The output should be the square of the input. How hard could this be? Turns out that this is easy! Check out . &qa_howto_1_listing; gr_unittest is an extension to the standard python module unittest. gr_unittest adds support for checking approximate equality of tuples of float and complex numbers. Unittest uses Python's reflection mechanism to find all methods that start with test_ and runs them. Unittest wraps each call to test_* with matching calls to setUp and tearDown. See the python unittest documentation for details. When we run the test, gr_unittest.main is going to invoke setUp, test_001_square_ff, and tearDown. test_001_square_ff builds a small graph that contains three nodes. gr.vector_source_f(src_data) will source the elements of src_data and then say that &its; finished. howto.square_ff is the block &were; testing. gr.vector_sink_f gathers the output of howto.square_ff. The run method runs the graph until all the blocks indicate they are finished. Finally, we check that the result of executing square_ff on src_data matches what we expect. The build tree is everything from topdir (the one containing configure.ac) down. The path to the install tree is prefix/lib/pythonversion/site-packages, where prefix is the --prefix argument to configure (default /usr/local) and version is the installed version of python. A typical value is /usr/local/lib/python2.3/site-packages. We normally set our PYTHONPATH environment variable to point at the install tree, and do this in ~/.bash_profile or ~/.profile. This allows our python apps to access all the standard python libraries, plus our locally installed stuff like GNU Radio. We write our applications such that they access the code and libraries in the install tree. On the other hand, we want our test code to run on the build tree, where we can detect problems before installation. We use make check to run our tests. Make check invokes the run_tests shell script which sets up the PYTHONPATH environment variable so that our tests use the build tree versions of our code and libraries. It then runs all files which have names of the form qa_*.py and reports the overall success or failure. There is quite a bit of behind-the-scenes action required to use the non-installed versions of our code (look at runtest for a cheap thrill.) Finally, running make check in the python directory produces this result: [eb@bufo python]$ make check make check-TESTS make[1]: Entering directory `/home/eb/gr-build/gr-howto-write-a-block/src/python' Traceback (most recent call last): File "./qa_howto.py", line 24, in ? import howto ImportError: No module named howto Traceback (most recent call last): File "./qa_howto_1.py", line 24, in ? import howto ImportError: No module named howto FAIL: run_tests =================== 1 of 1 tests failed =================== make[1]: *** [check-TESTS] Error 1 make[1]: Leaving directory `/home/eb/gr-build/gr-howto-write-a-block/src/python' make: *** [check-am] Error 2 [eb@bufo python]$ Excellent! Our test failed, just as we expected. The ImportError indicates that it can't find the module named howto. No surprise, since we haven't written it yet. Now that we've got a test case written that successfully fails, let's write the C++ code. As we mentioned earlier, all signal processing blocks are derived from gr_block or one of its subclasses. Let's take a look at . &gr_block_listing; A quick scan of gr_block.h reveals that since general_work is pure virtual, we definitely need to override that. general_work is the method that does the actual signal processing. For our squaring example we'll need to override general_work and provide a constructor and destructor and a bit of stuff to take advantage of the boost shared_ptrs. and are the header and c++ source. &howto_square_ff_h_listing; &howto_square_ff_cc_listing; Now we need a Makefile.am to get all this to build. is enough to build a shared library from our source file. We'll be adding additional rules to use &SWIG; in just a bit. If you haven't already, this is a good time to browse all the Makefile.am's in the build tree and get an idea for how it all hangs together. &src_lib_Makefile_1_am_listing; Now that we've got something that will compile, we need to write the &SWIG; .i file. This is a pared-down version of the .h file, plus a bit of magic that has python work with the boost shared_ptr's. To reduce code bloat, we only declare methods that &well; want to access from Python. We're going to call the .i file howto.i, and use it to hold the &SWIG; declarations for all classes from howto that will be accessible from python. It's quite small: &howto_1_i_listing; Now we need to modify src/lib/Makefile.am to run &SWIG; and to add the glue it generates to the shared library. &src_lib_Makefile_2_am_listing; make now builds everything successfully. We get a few warnings, but &thats; OK. Changing directories back to the python directory we try make check again: [eb@bufo python]$ make check make check-TESTS make[1]: Entering directory `/home/eb/gr-build/gr-howto-write-a-block/src/python' . ---------------------------------------------------------------------- Ran 1 test in 0.004s OK PASS: run_tests ================== All 1 tests passed ================== make[1]: Leaving directory `/home/eb/gr-build/gr-howto-write-a-block/src/python' [eb@bufo python]$ Victory! Our new block works! In our howto_square_ff example above, we only had to override the general_work method to accomplish our goal. gr_block provides a few other methods that are sometimes useful. Looking at general_work you may have wondered how the system knows how much data it needs to ensure is valid in each of the input arrays. The forecast method provides this information. The default implementation of forecast says there is a 1:1 relationship between noutput_items and the requirements for each input stream. The size of the items is defined by gr_io_signatures in the constructor of gr_block. The sizes of the input and output items can of course differ; this still qualifies as a 1:1 relationship. // default implementation: 1:1 void gr_block::forecast (int noutput_items, gr_vector_int &ninput_items_required) { unsigned ninputs = ninput_items_required.size (); for (unsigned i = 0; i < ninputs; i++) ninput_items_required[i] = noutput_items; } Although the 1:1 implementation worked for howto_square_ff, it wouldn't be appropriate for interpolators, decimators, or blocks with a more complicated relationship between noutput_items and the input requirements. That said, by deriving your classes from gr_sync_block, gr_sync_interpolator or gr_sync_decimator instead of gr_block, you can often avoid implementing forecast. When implementing your general_work routine, &its; occasionally convenient to have the run time system ensure that you are only asked to produce a number of output items that is a multiple of some particular value. This might occur if your algorithm naturally applies to a fixed sized block of data. Call set_output_multiple in your constructor to specify this requirement. The default output multiple is 1. gr_block allows tremendous flexibility with regard to the consumption of input streams and the production of output streams. Adroit use of forecast and consume allows variable rate blocks to be built. It is possible to construct blocks that consume data at different rates on each input, and produce output at a rate that is a function of the contents of the input data. On the other hand, it is very common for signal processing blocks to have a fixed relationship between the input rate and the output rate. Many are 1:1, while others have 1:N or N:1 relationships. Another common requirement is the need to examine more than one input sample to produce a single output sample. This is orthogonal to the relationship between input and output rate. For example, a non-decimating, non-interpolating FIR filter needs to examine N input samples for each output sample it produces, where N is the number of taps in the filter. However, it only consumes a single input sample to produce a single output. We call this concept "history", but you could also think of it as "look-ahead". gr_sync_block is derived from gr_block and implements a 1:1 block with optional history. Given that we know the input to output rate, certain simplifications are possible. From the implementor's point-of-view, the primary change is that we define a work method instead of general_work. work has a slightly different calling sequence; It omits the unnecessary ninput_items parameter, and arranges for consume_each to be called on our behalf. /*! * \brief Just like gr_block::general_work, only this arranges to * call consume_each for you. * * The user must override work to define the signal processing code */ virtual int work (int noutput_items, gr_vector_const_void_star &input_items, gr_vector_void_star &output_items) = 0; This gives us fewer things to worry about, and less code to write. If the block requires history greater than 1, call set_history in the constructor, or any time the requirement changes. gr_sync_block provides a version of forecast that handles the history requirement. gr_sync_decimator is derived from gr_sync_block and implements a N:1 block with optional history. gr_sync_interpolator is derived from gr_sync_block and implements a 1:N block with optional history. Given that we now know about gr_sync_block, the way howto_square_ff should really be implemented is by subclassing gr_sync_block. Here are the revised sources: , . The accompanying files contain the additional test code. &howto_square2_ff_h_listing; &howto_square2_ff_cc_listing; At this point, we've got a basic overview of how the system goes together. For more insight, I suggest that you look at the code of the system. The doxygen generated class hierarchy is a useful way to find things that might interest you. Sources and sinks are derived from gr_sync_block. The only thing different about them is that sources have no inputs and sinks have no outputs. This is reflected in the gr_io_signatures that are passed to the gr_sync_block constructor. Take a look at gr_file_source.{h,cc} and gr_file_sink.{h,cc} for some very straight-forward examples. If your block isn't working, and you can't sort it out through python test cases or a few printfs in the code, you may want to use gdb to debug it. The trick of course is that all of &gnuradio;, including your new block, is dynamically loaded into python for execution. Try this: In your python test code, after the relevant imports, print out the process id and wait for a keystroke. In another window run gdb and tell it to attach to the python process with the given process id. At this point you can set breakpoints or whatever in your code. Go back to the python window and hit Enter so it'll continue. #!/usr/bin/env python from gnuradio import gr from gnuradio import my_buggy_module # insert this in your test code... import os print 'Blocked waiting for GDB attach (pid = %d)' % (os.getpid(),) raw_input ('Press Enter to continue: ') # remainder of your test code follows... Another SNAFU you might run into is that gdb 6.2 isn't able to set breakpoints in the constructors or destructors generated by g++ 3.4. In this case, insert a call to the nop function gri_debugger_hook in the constructor and recompile. Load the code as before and set a break point on gri_debugger_hook. Oprofile is your friend. See http://oprofile.sourceforge.net.