In this module we will use the following components:

• STM32 NUCLEO-F072RB

• USB cable - 6" A/MiniB

• Adafruit I2S MEMS Microphone Breakout

In principle, any board from STM32 can be used for these exercises, as long as it is supported by , and exposes at least two I2S buses since both the microphone and the DAC (Stereo Decoder) require a dedicated I2S bus for audio transfers.

Prerequisites

• basic knowledge of C and Python programming

• a PC with a USB port (the microcontroller will be programmed and powered by your PC via a USB cable)

• and, of course, having completed the the previous DSP modules!

Hi, and welcome to this final module of the DSP course, in which we will learn how to build a real time signal processing system using a general-purpose microcontroller unit.

Developing DSP applications for a low-level device does not affect the theory behind the algorithms that we implement. However, the constraints imposed by a low-power CPU will demand that we pay particular attention to details such as code efficiency, memory use and input/output routines.

In addition to that, working "close to the metal", so to speak, will give us the chance to look in some detail at the often neglected physical aspect of DSP, namely, the actual electronic components and the signalling protocols that are used to move data around in a complex circuit.

We will focus this module on audio signal processing, so as to build a system whose functions we can immediately enjoy; in particular, we will design and implement a variety of voice scramblers, that is, devices that you can use to alter your speaking voice in real time and sound, say, like a chipmunk or like Darth Vader.

The skills and the experience needed to port a real-time signal processing algorithm to an embedded board are the province of the truly accomplished DSP engineers; in this module you will be able to get a first taste of the challenges and the struggles of the job, together with the priceless satisfaction of truly "making" a device come to life.

In this online book we will provide you with step-by-step instructions together with links to videos on that illustrate all the steps leading to the "finished product". Working with hardware can often prove overwhelming and dealing with the numerous protocols, lengthy documentation, and specific components can be frustrating at times. But, thanks to these notes and the videos, you should be able to navigate around these issues with ease and taste the fun and exciting side of practical DSP right away.

Even if you don't have access to the selected hardware platform, you will still be able to appreciate the main lessons on real-time, low-level programming and we still recommend you read the nevertheless.

We hope you'll find this tutorial instructive and entertaining!

In this eBook we will develop real-time signal processing algorithms for a specific piece of hardware, namely the STM32 NUCLEO-F072RB microcontroller board manufactured by STMicroelectronics (often abbreviated as ST). ST provides many inexpensive development boards that are used by hobbyists, students, and professionals to prototype countless applications.

In principle, any board from the STM32 family can be used for the exercises, as long as it exposes at least two I2S buses. You can find more information about this family of boards by reading the official documentation.

In order to facilitate the development of applications, ST provided full integrated development environments (IDEs) that you can use to program their boards. These tools can sometimes be overwhelming as they allow for a lot of customization, but they are meant to make your life easier! Attention to detail and reading the documentation will help you in setting up a successful workflow. We will cover these tools and their installation in the next section.

We will conclude this introductory part by illustrating in detail how to build a first simple application on the microcontroller.

Here are some useful shortcuts and debugging tips when using the IDE. You will find this section more useful later on, when you begin to be familiar with the programming environment.

Shortcuts

Below are some shortcuts we find particularly useful. For MacOs, replace "Ctrl" with "Command".

text and code Eric Bezzam, Adrien Hoffet and Paolo Prandoni For inquiries and information please write to [email protected]

You can download the code for the examples described in this gitbook here.

(c) LCAV, EPFL, 2020

Any real-time audio application running on the microcontroller will need to acquire data from a source (for instance, a microphone) and deliver data to an output sink (for instance, an digital-to-analog converter connected to a loudspeaker) that we can listen to. Here is a brief description of the components we selected.

The microphone breakout board

The component used to capture sound is the by Adafruit. The actual microphone on this mini-board produces an analog signal (continuous in time and amplitude) but the device also contains an Analog-to-Digital Converter that returns a digital signal (discrete in time and amplitude), which is the format we need in order to pass the data to our microcontroller. We will describe the component in more detail .

We will now start the most interesting part of this module, the one where we start implementing actual audio DSP algorithms on the microcontroller.

As we said in the beginning, the theme of our examples will be the design of increasingly more sophisticated voice transformers, that you can use in real time to modify the sound of your own voice.

Before proceeding with this section, you should download and play with the Voice Transformer Jupyter notebook that we prepared for this module. The notebook is also available in your Coursera workspace in module DSP4 if you prefer not to run it locally.

In the notebook you will find a theoretical explanation of the algorithms that we will try to implement in the microcontroller, together with the code and with sound examples that you can listen to.

Reading and understanding the notebook is fundamental to understanding the sections that follow since, from now on, we will focus solely on the implementation details associated to our specific hardware and on the need to implement the algorithms in strict real time.

A "passthrough" can be viewed as the audio processing equivalent of a "hello world" program. In this section we will program the Nucleo to simply pass the audio samples from the microphone to the DAC.

Using the CubeMX software, we will first update the configuration of the microcontroller. We will then guide you through the wiring and, finally, we will program our passthrough using the SW4STM32 software.

Highlighted boxes, as shown below, specify a task for which you need to find out the appropriate solution and implementation.

A passthrough is a great sanity check when first beginning with an audio DSP system. Moreover, it serves as a useful starting point for new projects, as we will see in the following chapters when we develop more complicated programs.

The microphone and the DAC are peripheral components external to the microcontroller board and therefore we need to understand two fundamental things:

• the protocol used by external peripherals to electrically transfer data to and from the microcontroller board; for audio, this is usually the I2S protocol

• the mechanism by which the data transfer is handled; in our case this will be a so-called DMA transfer.

In this section we will illustrate some common coding practices that are used in real-time DSP implementations and that will be used later in our examples.

Circular buffers

In in the we discussed some implementation issues related to discrete-time filters and, in particular, we talked about circular buffers.

As a quick recap, remember that if you need to store past values of a signal, the best solution is to use a circular buffer; assume that you need to access at most past values of the signal :

The microphone we are using measures an analog signal and returns a digital signal, which can be further processed by our microcontroller entirely in the digital domain. In order to playback or listen to this digital signal, it is necessary to convert it back to analog form; this can be done with a DAC (Digital-to-Analog Converter). We will be using Adafruit's , which contains a DAC, an audio jack for connecting headphones, and the necessary additional components. In the following subsections, we will explain the important inputs/outputs of the DAC we will be using, the I2S stereo output protocol our application will have to conform to, and an explanation about the breakout board from Adafruit.

DAC inputs/outputs

The DAC component in the Adafruit breakout is the by NXP, whose block diagram is shown below.

In this section we will implement the "alien voice" effect on the microcontroller. As shown in the , the alien voice effect is achieved simply by performing sinusoidal modulation on the input signal in order to shift the voice spectrum up in frequency.

Given a modulation frequency (in Hz) and an input sample we can compute each output sample instantaneously as:

where is the system's sampling frequency (in Hz). The modulation frequency must be kept small in order to preserve intelligibility; still, the resulting signal will be affected by aliasing and other artifacts that we cannot really control.

As mentioned in the Jupyter notebook, this voice transformer is great for real-time applications as it requires only a single multiplication per sample. This means that, compared to the passthrough project, we will not have to write too much new code. But the devil, as they say, is in the details!

.

We will now implement the second voice transformation method described in the Jupyter notebook, namely pitch shifting via granular synthesis.

While the alien voice transformer simply alters the quality of the voice, with a pitch shifter we will be able to move the perceived pitch of the speaker either down (to create a "Darth Vader" sound) or up (to create a "Chipmunks" voice).

We recommend you study the relevant section on the notebook carefully before proceeding, since the algorithmic details are going to be a bit trickier than what we have seen so far; please make sure you understand the theory and that you are comfortable with the offline implementation before tackling the real-time version of the transformer.

May the Force be with you!

Figure: Modified from .

The ST Nucleo board hosts a microcontroller that is both:

• highly configurable, in the sense that some of its electrical pins can be rerouted in software and assigned to specific function

• programmable at a high level, since we can use C code and use a compiler to produce the microcode that will be uploaded onboard

To handle this great flexibility, ST provides us with an integrated development environment (IDE) that we can use to manage both aspects of Nucleo programming. This is the , an Eclipse-based IDE for programming STM32 microcontrollers. From the description webpage:

When discussing the code architecture of a generic real-time audio device, we already remarked that if our processing callback is too slow with respect to the frequency of the DMA transfers, we will run into a condition called buffer underflow (or overflow, if you look at it from the point of view of the input DMA).

It's therefore very important to make sure that our processing is fast enough and find out if and where the code is using up a lot of time. Fortunately, the microcontroller provides us with functionalities that gives the possibility to monitor that.

Timers

The HAL library includes a function uint32_t HAL_GetTick(void); which will return the number of ticks since the start of the microcontroller in milliseconds. Unfortunately we cannot use this tool because a resolution of one millisecond is too large for most audio sampling frequencies. For instance, with MHz the period to perform one operation is, thus the micro-second granularity is way too slow.

In order to have a finer timebase, we will use the Nucelo's onboard , whose full technical details can be found . Briefly, all computing boards (and microcontroller are no exception) possess an internal clock that provides a reference timebase signal; this timebase is usually generated by a . The onboard timer is a roll-over counter that is incremented in lockstep with the timebase signal, often via a that can be used to lower its frequency, since the oscillator is usually very fast.

For our application, we will use a timer with a large counting capacity (32 bits) and we will set it to increment itself every microsecond.

Setting up the timer

To set up the timer we will use CubeMX and then regenerate the initialization code. Open the CubeMX file by double clicking the .ioc file of the copied project it in the IDE project explorer.

In order to activate a timer, you need to set a "Clock Source". Open TIM2 in the Timers menu (TIM2 happens to be 32bit timer) and activate its clock by setting the Clock Source to "Internal Clock".

Next, we need to configure the timer in the configuration panel that appears:

Set the Counter Period to 0xFFFFFFFF; this ensures that the 32-bit timer counter only rolls around at its maximum value. You can leave the rest of the parameters as is for "TIM2". Finally, you can update the initialization code by saving the .ioc file.

Using the timer

In order to use the timer we configured, we will define a couple of macros to start and stop the timer and a global variable to keep track of the time that elapses between calls. Between the USER CODE BEGIN PV and USER CODE END PV comment tags, add the following lines. Note the volatile declaration for the timer, which underscores how this variable is a global variable modified by an interrupt service routine independently of the normal control flow of the rest of the code.

For instance, to benchmark the passthrough example, we can modify the Process function like so

Benchmarking live

In a real-time audio application the processing time cannot exceed the time between successive DMA calls; if this is not the case, we have a so-called buffer underflow which results in extremely corrupted audio. We will use our benchmarking timer to make sure we are within the limits.

To check the actual time used by our processing function we will use an extremely convenient facility provided by the STM32 IDE, namely the possibility to monitor the live value of the variables in our code while the code runs.

Pull up the passthrough example and modify the processing function as shown in the previous section by inserting the START_TIMER and STOP_TIMER macros. Then launch the application in the debugger.

In the debugging window in the top right corner of the screen, select the "live variables" tag and add the variable timer_value_us.

You can see that the passthrough code takes about 33 microseconds to execute, which is well below the maximum available time. This is good news!

Solutions

Assuming you have successfully implemented the , you can simply copy and paste that project from within the STM32CubeIDE environment. We recommend choosing a name with the current date and "alien_voice" in it. Remember to delete the old binary (ELF) file inside the copied project.

Lookup table

Remember that in the passthrough we set up the system to work with a sampling frequency of 32KHz and a sample precision of 16 bits. Here we will use a modulation frequency of 400Hz for the frequency shifting, so that the digital frequency is a rational multiple of as in the :

The values for one period of the digital sinusoid can be encoded in a

Breakpoint and watch

The first and maybe most instinctive way to check if a code is working as expected is to put a breakpoint at a critical line of code. In that way it is possible to check if the micro-controller is going through a certain instruction and to do a step-by-step execution of the code starting from the breakpoint.

A breakpoint is added by a double click on a line number in the code window. It can be added either during execution (debug session already started) or during editing. When a breakpoint is reached, the view jumps to the breakpoint's line and you will see the following view:

In the previous section we implemented a basic granular synthesis voice transformer that lowers the pitch of the input voice. In this section we will address some remaining issues, namely:

• implement an effect that raises the pitch of the voice (aka the "Chipmunks" effect)

• properly initialize the buffer as a function of the pitch change

• optimize the code a little more

The Chipmunks

To raise the pitch of the voice we need to set to values larger than one. As we have seen, this makes the effect noncausal, which we need to address by introducing some processing delay.

The way to achieve this is to place the audio buffer's input index forward with respect to the output index; let's do this properly by creating an initialization function for the buffer that takes the resampling factor as the input.

By now you know where to place this code but don't forget to

• add the following line to the file main.h between the /* USER CODE BEGIN Includes */ tags.

• declare the function prototype in the USER CODE BEGIN PFP block

• call the function before launching the DMA transfers:

Switching between effects

We can use the blue button on the Nucleo board to switch between Darth Vader and the Chipmunks; to do so, define the following constants at the beginning of the code

and modify the user button callback like so:

Final optimizations

In the main processing loop, we are performing two checks on the value of grain_m per output sample. However, in the current implementation, both the stride and the taper lengths are multiples of the size of the DMA half-buffer. This allows us to move these checks outside of the processing loop and perform them once per call rather than once per sample

Solutions

Gain

One thing that you might have noticed from the passthrough example is that the output signal is not very loud. To correct this, we can add a gain factor to the processfunction that multiplies each signal sample by a constant.

In order to take advantage of the architecture of the microcontroller's internal multiplier, it is recommended to use factors that are a power of 2 since in this case a multiplication corresponds to a simple binary shift of the integer values to the left. We measured$1\mu s$difference in processing time when tested with the first voice transformer algorithm.

Removing the DC offset

In general, in DSP applications we assume that input signals are zero mean. This is no different in the case of our microphone, so that, if there is no sound, we expect a sequence of zeros. If you actually look at the input samples, however, you will almost certainly find out that this is not so. In fact, the internal circuitry in the microphone almost always adds a voltage offset, and sometimes different microphones (of the same manufacturer) will have different offsets. We typically call this shift in the waveform a .

DC offsets are highly undesirable since they limit the dynamic range of our system; in other words, we are "wasting" binary digits on a constant that serves no purpose.

We have talked about DC offset removal in in the . Recall that a DC component corresponds to a nonzero frequency value at so the idea is to use a filter with a zero in A very simple example is the so-called FIR "DC notch" whose CCDE is simply

Unfortunately this filter has the very poor frequency response shown here and, while good as a first approximation, it is not really recommended if audio quality is important to you.

A better filter is obtained by using a an IIR DC notch which, while marginally more expensive computationally, provides a much flatter frequency response over the audio frequency band:

When is close to (but less than) one, we can get a magnitude response like this:

Tasks solutions

Now that we have initialized the different peripherals that we will use to interface with the outside world (from the point of view of the microcontroller), we are ready to wire everything up! Make sure that the STM32 board is not powered, i.e. unplugged, while connecting the microphone and DAC breakout boards.

For this task, we will have to refer to the card provided with the STM32 board (see below) and the image of the chip on the "Pinout" tab of our CubeMX project (further below).

Adafruit I2S MEMS Microphone Breakout

As previously mentioned, make sure that the STM32 board is powered off! We can then begin by connecting the microphone's ground pin. In electronics, it is common practice to first ground a component/circuit.

We can now connect the supply voltage pin.

Previously, we configured I2S2 for the microphone so we will have to connect the following pins (see image of chip from "Pinout" tab for the names on the left side of the arrow) to the corresponding pins on the microphone breakout board (right side of the arrow):

• I2S2_SD DOUT

• I2S2_CK BCLK

• I2S2_WS LRCL

Finally, we configured an additional GPIO pin in order to select whether we would like the microphone to be assigned to the left or right channel.

Adafruit I2S Stereo Decoder

As previously mentioned, make sure that the STM32 board is powered off! We can then begin by connecting the DAC's power supply, starting with the ground pin.

Previously, we configured I2S1 for the DAC so we will have to connect the following pins to the appropriate pins on the DAC breakout board:

• I2S1_SD

• I2S1_CK

• I2S1_WS

Moreover, we configured an additional GPIO pin in order to mute the output.

• MUTE

With everything correctly wired up, we can proceed to the passthrough on the SW4STM32 software!

Tasks solutions

Sadly we cannot connect all the wires for you or double check your connections. However we did our best to help you with this wiring by making a step-by-step video accessible at this .

The Nucleo board has a user programmable push button. We will now use it as an ON/OFF button for the alien voice effect.

Configuration

The idea is to use the push button to call an asynchronous routine in our code. To do that, we need to configure the button to trigger an interrupt and then we need to catch the interrupt in our code.

Go into CubeMX by clicking on the ioc file in your alien voice project; in the left panel click on "System > NVIC" and enable the line "EXTI line 4 to 15" by checking the corresponding checkmark. The pin PC13 is linked to EXTI13 in the hardware of the microcontroller. Interrupts are used because they provide a very fast access to the core of the system and thus a very fast reaction.

Still in CubeMX, verify that the label for pin PA5 is "LD2" and the label for pin PC13 is "B1".

Add the following state variable to the USER CODE BEGIN PV section

and add the following interrupt handler to the USER CODE BEGIN 0 section:

The interrupt handler toggles the variable user_button and switches the LED on when its value is true.

Benchmarking

Now that we have an ON/OFF button, we can use the to see how expensive it is to compute the alien voice.

Solution

In this section we will guide you step by step through the process of coding a simple application for the microcontroller, connecting the board, and running the application on the microcontroller. This first application does not use any peripherals and simply makes an onboard LED blink, it will be a basic project template that we can reuse many times later.

Please note that, if you get stuck, you can always download the working STM32 projects for the examples in this gitbook here.

Open the IDE and select a workspace

1) Open the STM32CubeIDE that you just installed in the previous section.

2) Select a workspace, this will be the folder where all projects are going to be stored. You can create multiple workspaces if you work on different projects.

3) Once you select a valid folder you can Launch the IDE.

Create a new project

The first time you open the software, you will be prompted by the screen shown below. If a pop-up appears, asking if you would like to initialize all peripherals with their default mode, simply press Yes.

Press the Start new STM32 project button in order to launch CubeMX and start initializing the project.

If you have a workspace that already contains a project, the new project button is in the top left corner.

Configuring the hardware with CubeMX

We will be using CubeMX's graphical interface to generate the initialization steps for the board and its peripherals. Once the board is configured, the IDE will translate our configuration choices into automatically-generated C code.

Board selection

When all necessary downloads are completed, you should eventually see something similar to the screenshot below. Click on the Board Selector tab in the top toolbar to the left.

Make sure the "Board Selector" tab is the active one (top-left corner) and look for our board, the "NUCLEO-F072RB" (you can use the "Part Number Search" facility). Double-click the board in the search results. Note that if you are using a different ST board that , you should select the model you actually have.

Chose an appropriate name for the project, including the date, project goal etc, and leave the options as default.

When clicking next, you will see a pop-up asking if you want to initialize all peripherals to their default mode: this applies to the external circuits that may have been added to the Nucleo board. Peripheral initialization will be relevant later, when we add a microphone and an audio output module, but in this case we are only using an onboard LED and a button. Press Yes in any case.

When the board has loaded, you should see something similar to the following screenshot:

Extend the central pane if it was hidden, because it will be needed later!

Code generation

When a Nucleo template is selected and all peripheral initialized to their default values, the blue button B1 and the LED LD2 are already configured; this is sufficient for our first project.

We are now ready to generate the initialization code. Save your project by pressing CTRL+ S. The project will be automatically generated if a modification was made; in this case, since we did not change the layout, you may have to trigger the code generation by pressing Alt + K. CubeMX will generate some C files, using HAL libraries, that encode all the settings that were selected via the GUI.

HAL is short for Hardware Abstraction Layer and it is a set of libraries provided by ST to help developers produce portable code, i.e. code that can be used across the whole family of STM32 boards. For more information on HAL, please refer to .

The user application

From the "Project Explorer", open the file "Src/main.c"; this is the code automatically generated by CubeMX and it will look like so:

If you look at the C code, you can notice matched commented lines that read USER CODE BEGIN and USER CODE END; it is only between these tags that you should write your code!

All other lines of code have been generated automatically by CubeMX according to the configuration we specified via the graphical tool. If you go back and change some of the configuration parameters, CubeMX will overwrite all the code that is not between the USER CODE tags!

Blinking an LED

We will now program the board to perform a simple task - make an onboard LED blink!

In the code, look for the infinite loop between the comments USER CODE BEGIN WHILE and USER CODE END WHILE, add the following lines to the body of the loop:

HAL_GPIO_TogglePin and HAL_Delay are commands provided by the ST HAL library for toggling the voltage level on a pin and to pause execution, respectively. Remember that you can always look for the definition of a function or of a variable by pressing Ctrl and clicking the function/variable.

The first command toggles the value of the pin corresponding to the LED at pin LD2; this turns the LED on for one iteration of the while loop and off for the next iteration. In order to actually be able to observe the LED blinking we must set a delay between each toggle operation, otherwise the blinking would be too fast to be perceived. This is what the second command accomplishes by placing a delay of 1 second; the argument of the function HAL_Delay is indeed in milliseconds.

Building the project

Before plugging in the board, let's try building the project. This can be done by pressing the hammer icon on the top toolbar, or by using the shortcut Ctrl + B ("Command + B" on MacOS). Make sure you are building for the Debug target and for the correct project.

In the figure below, we can see the two signs of a successful build:

• A "Binaries" folder was created, as can be seen in the "Project Explorer", and it contains an ELF file corresponding to our project. It should have the same name as your project. If this does not appear, it may be necessary to refresh the project by right-clicking the project directory and selecting Refresh (or using the shortcut F5).

• There are no errors in the "Console" pane.

Now we can program the board! Plug the board to your computer using the USB Type-A to Mini-B cable. A couple LEDs of the board should light up as it is being powered by your computer.

Debugging the code

Click on the bug icon from the toolbar and select Debug As > STM32 MCU C/C++ Application (see below).

If there are no debug configurations available from the menu, set up a configuration first by choosing "Debug configurations..." and clicking on the STM32 Cortex-M option.

If this is your first time debugging in this workspace, you should see a pop-up similar to the one below appear. Click "Yes" as this perspective will be very useful, and you can check the box for "Remember my decision" so that this pop-up does not appear again.

If something similar to the following error appears:

make sure the board is properly plugged in and/or try another USB port.

If the Nucleo's firmware is outdated, you might be requested to update it, shown by the following pop-up:

Just press OK and then Yes.

When the Nucleo is reconnected. First press Open in update mode, and then Upgrade the firmware of your Nucleo.

After the upgrade, you can press again on the bug button to resume debbuging.

A view similar to the one below should then appear. This is the typical "Debug perspective" in Eclipse.

Your program should be momentarily paused as is the case in the figure above at Line 90. You can continue the program by pressing the Resume button as pointed out above.

You should now observe the green "LD2" LED (see below) blinking!

Figure: Top view of a NUCLEO board. Red arrow pointing out the location of "LD2" LED. .

Terminating the program

In order to properly stop the debugger, it is also necessary to disconnect from the board. Both can be done by pressing the Disconnect button on the top toolbar (see below).

Finally, you can switch back to the normal perspective by pressing the button to the left of the bug icon in the top-right corner (see below).

For the input we will use the I2S MEMS Microphone Breakout by Adafruit; in the following we will refer to this part simply as the Adafruit mic. In the following subsections, we will explain the key inputs and outputs of the MEMS microphone component, the I2S input protocol for the data transfer, and what is meant by a "breakout board".

Overview of the MEMS microphone pins

For portable devices, digital MEMS microphones are the popular choice for audio capture since they integrate both the analog microphone and the analog-to-digital converter that samples and quantizes the audio. MEMS is short for MicroElectroMechanical System, a process technology used to create tiny integrated devices or systems that combine mechanical and electrical components; MEMS are small, cheap, and easy to integrate into one's desired application.

The connectors on a MEMS microphone are the following:

The basic input pins are:

• VDD: (usually) 3.3V to power the device.

• GND: ground.

• CLK: an external "clock" signal that drives the sampler in the A/D circuit. The sampling frequency for the Adafruit mic is , that is, the input clock should be 64 times the desired audio sampling frequency.

A standard MEMS microphone typically returns a PDM (Pulse-Density Modulation) signal. This is essentially a 1-bit, 64-oversampled signal that requires downsampling and filtering in order to obtain a PCM (Pulse-Code Modulation) signal. PCM is the format typically used for storing and processing audio and it is indeed the format that we want to provide to the microcontroller. You can read more about PDM and PCM and and you can play with one-bit, oversampled signals .

Luckily for us, the MEMS component in the Adafruit mic already provides us with a PCM signal (the circuit implements a decimator and a low-pass filter), which it outputs in the that we have seen in the previous section. Each sample is encoded over 32 nominal bits (that is, the binary words is 32-bit long) and word synchronization requires an additional input signal:

• WS: a "word select" signal whose level transitions mark the beginning of a binary word; since there will be a data word per audio sample, the frequency for the WS signal must be equal to the sampling frequency, that is, equal to the CLK frequency divided by 64. Since two MEMS microphones can be connected in parallel to provide an interleaved stereo signal, the following convention is used: when WS goes HIGH, the MEMS whose SEL signal is HIGH will start to transmit while the MEMS whose SEL is LOW will remain in a tri-state output (essentially disconnected); conversely, when WS goes LOW, the MEMS whose SEL is low will start to transmit. Note that, because of the interleaving, the sampling frequency will need to be twice the nominal value.

I2S timing diagram example

Let's look at an example timing diagram from the single Adafruit microphone we will be using. We assume we have configured our microphone to be the left channel (that is, we set SEL=0).

Figure: I2S MEMS microphone output timing diagram. The output data format is I2S, 24 bit, 2's complement, MSB first. p. 7 of .

From the figure above, we can make several observations:

1. After WS switches to LOW, we receive the first bit of information on the DATA line from the microphone, since SEL=0. When WS switches to HIGH (meaning a word is expected from the right channel microphone) the left channel microphone stays disconnected from the data bus.

2. Each new bit is received at a rising edge and held for an entire period of CLK.

3. The first 18 bits after a rising or falling edge of the WS signal corresponds to actual audio data, starting with the Most-Significant Bit (MSB) and finishing with the Least-Significant Bit (LSB).

I2S wiring example

In general, two MEMS microphones are usually connected in parallel according to the following diagram; the component called "IS2 Master" would be our microcontroller. The terms "master" and "slave" are quite common in electronics to describe the device which acts as the controller, and the devices(s) that are being controlled, respectively. See for more information on the terminology.

Figure: I2S MEMS microphone wiring for stereo use. Note that in our exercises we will be using a mono, i.e. one channel, setup. p. 7 of .

Some important observations can be made:

• The DATA lines for the two microphones are connected to each other and are supplied as a single input to the I2S Master.

• The SEL input for each microphone is set differently: SEL=VDD for the right-channel microphone and SEL=GND for the left-channel microphone. This is absolutely essential if two microphones are to share the same DATA line, as we explained before.

• The two microphones use the same BCLK (aka CLK

In this module, we will only use a single microphone, but the wiring from the microcontroller to the MEMS is identical.

Adafruit breakout

From the diagram above, we can observe that a MEMS microphone requires several additional components (capacitors and resistors) on top of the several inter-connecting wires. Instead of taking care of this part ourselves we can simply use a pre-made breakout board. With a breakout board, all the necessary components are pre-installed and we simply provide the connections for the signals/ports that need to interact with our microcontroller. In the case of our microphone, all the components (microphone, resistors, capacitors) are soldered on a compact board and convenient access can be given to the following signals:

1. VDD and GND: provided by the microcontroller to power the microphone.

2. WS and BCLK: generated by the microcontroller for the I2S transfer.

3. SEL: wired by the user to either VDD or GND to configure the microphone appropriately.

It is possible to design your own breakout boards using CAD tools for PCBs (Printed Circuit Boards). But for popular components like microphones, it is easy to find breakout boards that have already been designed. is a great place to find such boards and other cool electronics for personal projects, along with very well-explained user guides. The is the component that perfectly fits our needs.

The key technique behind simple pitch shifting is that of playing the audio file either more slowly (to lower the pitch) or faster (to raise the pitch). This is just like spinning an old record at a speed different than the nominal RPM value.

In the digital world we can always simulate the effect of changing an analog playing speed by using fractional resampling; for a refresher on this technique please refer to . Resampling, however, has two problems:

• the pitch of speech is changed but so is the overall speed, which we do not want;

• the ratio of output to input samples for the operation is not one, so it cannot be implemented in real time.

The initialization code we generated in the blinking LED example will need to be updated as it does not perform the setup for the two I2S buses that we will need to communicate with the microphone and the DAC.

Create a new project

First, let's make a copy of our working LED blinking project. We want to keep tracks of old projects in order to be able to go back to a known working configuration if something is not functioning anymore. To copy the project use the "Project Explorer" of the SW4STM32 software. Open the project you want and do a simple copy/paste operation. When you paste the project, a pop-up will ask you to rename the copied project: we recommend choosing a name that includes the current date and the word "passthrough" in it for bookkeeping purposes.

To finish the copying process:

• make sure that the binary file of the original project is removed by deleting the .elf file in the Binaries folder of the new project.

• rename the .ioc file with the name of the project

Now we are ready to update the initialization code. From the project explorer, click on the IOC file of the new project and open the CubeMX configurator.

Enable and configure the I2S buses

When the IOC file has successfully loaded, you should see something similar to the figure below. On the left-hand column, select "Multimedia" and expose the I2S1 and I2S2 selectors.

I2S1 (DAC)

Let's begin by setting up the I2S channel that communicates with the DAC. Click on I2S1 and select the "Half-Duplex Master" for the Mode in the top middle panel.

You should see several pins highlighted in green: after enabling an I2S bus, the interface shows in green the electrical pins in the microcontroller that will be devoted to the signals used in the I2S protocol. Recall that an I2S bus uses three pins according to the :

1. Clock (CK).

2. Word select (WS).

3. Serial data (SD).

Move your attention now to the "Configuration" panel below; we'll need to set up the structure of the data that transits on the bus (bits per word and per frame) and the data rate.

Select the "Parameter Setting" tab and set the transmission mode to "Mode Master Transmit" and the Communication Standard to "I2S Philips".

Now let's configure the DMA transfers. Select the "DMA Settings" tab and press "Add". Adjust the settings so that DMA Request is set to "SPI1_TX, Data Width is set to "Half Word" and Mode is set to "Circular", as in the screenshot below. Note that the DMA stream can differ if you are using a different microcontroler as it is dependent on the physical implementation of the internal circuitry.

I2S2 (microphone)

Repeat the previous steps for I2S2 with the following differences:

• set the Transmission Mode to "Mode Master Receive"

• set the DMA request to "SPI2_RX

Finally, complete the configuration:

As a final sanity check, click on "NVIC" under "System" in the left column and ensure that the interrupts are enabled for both selected DMA channels, as below.

Configure the GPIO pins

The configuration we have done so far would be sufficient in order to create an audio passthrough. However, we will configure two more pins of the microcontroller so that we can programmatically:

1. Mute the DAC.

2. Assign the microphone to the left or the right channel.

Go back to the "Pinout" tab, as seen below.

By clicking on any of the pins, you should be able to see the different functions that particular pin can assume, see below.

We are interested in using two pins as "GPIO_Output" (GPIO stands for "General-Purpose Input/Output") in order to output a HIGH or LOW value to the Adafruit breakout boards. Set the pins "PCO" and "PC1" to "GPIO_Output" (see below). You can reset a pin to having no function by selecting "Reset_State".

Just as in the case of variables in a program, we should give meaningful names to our GPIO pins. We will rename "PC0" and "PC1" as "MUTE" and "LR_SEL" respectively. You can rename a pin by right-clicking it and selecting "Enter User Label" (see below).

Update initialization code

If you now save the IOC file (or if you change perspective) the source code will be updated:

If you have any of the source files open on SW4STM32, they should refresh automatically to reflect the settings you have changed in CubeMX. Remember that this is why you should not add or modify any section in the code outside of the USER CODE BEGIN and USER CODE END comments; outside of these tags, all code will usually replaced by a change in configuration.

With the peripherals and initialization code updated, we can proceed to !

Tasks solutions

Coding a "real-world" DSP application on dedicated hardware is a bit of a shock when we are used to the idealized world of theoretical derivations and nowhere is the disconnect more profound than when we need to take numerical precision explicitly into account.

float vs. int

Floating point numbers, as implemented in most architecture today, free us of the need to explicitly consider the range of the numeric values that appear in our algorithm. Although not without caveats, a 64-bit double in C is pretty much equivalent to an ideal real number for most intents and purposes, with a dynamic range (that is, the ratio between the smallest and largest numbers that can be represented) in excess of .

We are building a real-time system, so the output data rate will necessarily be equal to the input data rate. In the previous section we saw that grains are produced via a periodic pattern whose period is equal to the stride length. It would make perfect sense, therefore, to set the length of the DMA buffer equal to the stride and let that be the cadence of the processing function.

Unfortunately this simple approach clashes with the capabilities of the hardware and so we need to trade resources for some extra code complexity: welcome to the world of embedded DSP!

Memory limitations

If we play around with the Jupyter notebook implementation of granular synthesis, we can quickly verify that the voice changer works best with a grain length of about 30ms and an overlap factor of about 50%. Using the formula derived in the previous section, this gives us a grain stride of 20ms.

Now, remember that the smallest sampling frequency of our digital microphone is 32KHz so that 20ms correspond to 640 samples. Each sample is 2 bytes and the I2S protocol requires us to allocate a stereo buffer. This means that each DMA half-buffer will be

Since we need to use double buffering for DMA, and since we need symmetric input and output buffers, in total we will need to allocate over 10KB of RAM to the DMA buffers alone; when we start adding the internal buffering required for computation, we are going to quickly exceed the 16KB available on the Nucleo F072RB!

(As a side note, although 16KB may seem ludicrously low these days, remember that small memory footprints are absolutely essential for all devices that are not a personal computer. The success of IoT hinges upon low memory and low power consumption!)

To avoid the need of large DMA buffers, we will implement granular synthesis using the following tricks:

• to save memory, all processing will be carried out on a mono signal;

• we will use a single internal circular buffer that holds enough data to build the grains; we have seen in the previous section that we need a buffer at most as long as the grain. Using mono samples, this will require a length of 1024 samples, for a memory footprint of 2 KBytes.

• we will fill the internal buffer with short DMA input transfers and compute a corresponding amount of output samples for each DMA call; DMA transfers can be as short as 16 or 32 samples each, thereby reducing the amount of memory required by the DMA buffers.

The code

To code the granular synthesis algorithm, copy and paste the Alien Voice project from within the STM32CubeIDE environment. We recommend choosing a name with the current date and "granular" in it. Remember to delete the old binary (ELF) file inside the copied project.

Here, we will set up the code for the "Darth Vader" voice transformer and will consider more advanced modifications in the next section.

DMA size

As we explained, the idea is to fill the main audio buffer in small increments to save memory. To this end, set the DMA half-buffer size to 32 samples in the USER CODE BEGIN PV section:

Grain size and taper

We will use a grain length of samples which corresponds to about 30ms for a sampling rate of 32KHz. The overlap is set at 50%, i.e., we will use a tapering slope of samples. The resulting grain stride is .

Add the following lines to the USER CODE BEGIN 4 section in main.c, where the values for the tapering slope are those computed by your Python function:

Main buffer

We choose the buffer length to be equal to the size of the grain, since anyways the voice transformer doesn't sound too good for . With a size equal to a power of two, we will be able to use bit masking to enforce circular access to the buffer. Add the following lines after the previous ones:

With these values the buffers are set up for causal operation (i.e., for lowering the voice pitch); we will tackle the problem of noncausal operation later.

You can now examine the memory footprint of the application by compiling the code and looking at the "Build Analyzer" tab on the lower right corner of the IDE. You should see that we are only using less than 30% of the onboard RAM.

Processing function

This is the main processing function:

The processing loop uses an auxiliary function Resample(uint16_t m, uint16_t N) that is supposed to return the interpolated value .

A simplistic implementation is to return the sample with integer index closest to :

Benchmarking

Since our processing function is becoming a bit more complex than before, it is interesting to start benchmarking its performance.

Remember that, at 32KHz, we can use at most per sample; we can modify the timing function to return the number of microseconds per sample like so:

If we now use the , we can see that the current implementation (with the full fractional resampling code) requires between and per sample, which is well below the limit. The oscillation between the two values reflects the larger computational requirements during the tapering slope.

Solutions

In this section, we will guide you through programming the microcontroller in order to implement the passthrough. Many of the concepts in this section lay the foundations for how to structure and code a real-time audio application on the microcontroller. In later sections we will build more complex processing functions, but the architecture of the code will remain the same.

In the previous section, you should have copied the blinking LED project before updating the IOC file with CubeMX. From the SW4STM32 software, open the file "Src/main.c" in the new project; we will be making all of our modifications here.

Macros

In programming a microcontroller, it is customary to define preprocessor macros to set the values of reusable constants and to concisely package simple tasks that do not require much logic and flow control and for which, therefore, a function call would be overkill. See for more on macros and preprocessor directives when programming in C.

Macros are usually defined before the main function; we will place our macros between the USER CODE BEGIN Includes and USER CODE END Includes comment tags.

The MUTE macro

As an example, we will begin by creating macros to change the logical level of the MUTE pin. As in the blinking LED example, we will be using HAL library calls in order to modify the state of the MUTE GPIO pin.

Note how the MUTE pin that we configured before automatically generates two constants called MUTE_GPIO_Port and MUTE_Pin, which is why we suggested giving meaningful names to pins configured with the CubeMX tool.

If you press "Ctrl" ("Command" on MacOS) + click on MUTE_GPIO_Port or MUTE_Pin to see its definition, you should see how the values are defined according to the pin we selected for MUTE. In our case, we chose pin PC0 which means that Pin 0 on the GPIO C port will be used. The convenience of the CubeMX software is that we do not need to manually write these definitions for the constants! The same can be said for LR_SEL.

The Channel Select macro

We will now define two more macros in order to assign the MEMS microphone to the left or right channel of the I2S bus, using the LR_SEL pin we defined previously. As before, you should place these macros between the USER CODE BEGIN Includes and USER CODE END Includes comments.

Private variables (aka Constants)

In most applications we will need to set some numerical constants that define key parameters used in the application.

These definitions are also preprocessing macros and they are usually grouped together at the beginning of the code between the USER CODE BEGIN PV and USER CODE END PV comment tags.

We will now define a few constants which will be useful in coding our application. Before defining them in our code, let's clarify some of the terminology:

1. Sample: a sample is a single discrete-time value; for a stereo signal, a sample can belong either to the left or right channel.

2. Frame: a frame collects all synchronous samples from all channels. For a stereo signal, a frame will contain two samples, left and right.

3. Buffer length: a buffer is a collection of frames, stored in memory and ready for processing (or ready for a DMA transfer). The buffer's length is a key parameter that needs to be fine-tuned to the demands of our application, .

Audio Parameters

Add the following lines to define the frame length (in terms of samples) and the buffer length (in terms of frames):

SAMPLES_PER_FRAME is set to 2 as we have two input channels (left and right) as per the I2S protocol.

Since our application is a simple passthrough, which involves no processing, we can set the buffer length - FRAMES_PER_BUFFER - to a low value, e.g. 32.

Data buffers

Again, as explained in Lecture 2.2.5b in the , for real-time processing we normally need to use alternating buffers for input and output DMA transfers. The I2S peripheral of our microcontroller, however, conveniently sends two interrupt signals, one when the buffer is half-full and one when the buffer is full. Because of this feature, we can simply use an array that is twice the size of our target application's buffer and let the DMA transfer fill one half of the buffer while we simultaneously process the samples in the other half.

Finally, we can create the input and output buffers as such:

Private function prototypes

In this section we will declare the function prototypes that implement the final application. The code should be placed between the USER CODE BEGIN PFP and USER CODE END PFP comment tags.

Main processing function

Ultimately, the application will work by obtaining a fresh data buffer filled by the input DMA transfer, processing the buffer and placing the result in a data buffer for the output DMA to ship out. We will therefore implement a main processing function with the following arguments:

1. a pointer to the input buffer to process

2. a pointer to the output buffer to fill with the processed samples

3. the number of samples to read/write.

The resulting function prototype is:

This will be the main processing function which will be invoked by the interrupts raised by the DMA transfer every time either the first or the second half of the buffer has been filled.

DMA callback functions

As previously mentioned, the STM32 board uses DMA to transfer data in and out of memory from the peripherals and issues interrupts when the DMA buffer is half full and when it's full.

The HAL family of instructions allows us to define triggered by these interrupts. Add the following function definitions for the callbacks, covering the four cases of two input and output DMAs times two interrupt signals:

Note that the Rx callbacks (that is, the callbacks triggered by the input DMAs), have an empty body and only the Tx callbacks (that is, the ones driven by the output process) perform the processing via our process function.

This is a simple but effective way of synchronizing the input and the output peripherals when we know that the data throughput should be the same for both devices. Of course we can see that if the process function takes too long, the buffer will not be ready in time for the next callback and there will be audio losses. In the next chapter, we will introduce a mechanism to monitor this.

You can read more about the HAL functions for DMA Input/Output for the I2S protocol in the comments of the file "Drivers/STM32F0XX_HAL_Driver/Src/stm32f0xx_hal_i2s.c" from the SW4STM32 software:

The user application

Between the USER CODE BEGIN 4 and USER CODE END 4 comment tags, we will define the body of the process function which, in this case, implements a simple passthrough.

Initial Setup

Between the USER CODE BEGIN 2 and USER CODE END 2 comment tags, we need to initialize our STM32 board, namely we need to:

1. un-mute the DAC using the macro defined .

2. set the microphone to either left or right channel using the macro defined .

3. start the receive and transmit DMAs with HAL_I2S_Receive_DMA and HAL_I2S_Transmit_DMA respectively.

This is accomplished by the following lines:

We can now try building and debugging the project (remember to press Resume after entering the Debug perspective). If all goes well, you should have a functioning passthrough and you should be able to hear in the headphones the sound captured by the microphone.

Going a bit further

If you still have time and you are curious to go a bit further, we propose to make a modification to theProcess function. In the current implementation, since the input is mono and the output is stereo, you may have noticed that only one output channel carries the audio while the other is silent. Wouldn't it be nice if both had audio, thereby converting the mono input to a stereo output?

Note: remember to copy your project before making any significant modifications; that way you will always be able to go back to a stable solution!

Congrats on completing the passthrough! This project will serve as an extremely useful starting point for the following (more interesting) applications. The first one we will build is an . But first, let's talk about some key issues in real-time DSP programming.

Solutions