FRDM-Stylophone

Introduction

Our project will be using the FRDM-KL46Z board to create a musical instrument similar to a stylophone. Users will use the touch slider to play notes of different pitches and duration while notes are output concurrently from our computer/laptop speakers. On the slider, different areas or capacitance will output different pitches based on a predetermined range, and depending on how long the slider is touched will lead to different note durations. Users can press the buttons on the board to change the range of notes that play. Enjoy making music with our FRDM-Stylophone!

System Overview

Diagram

Overview

Our project does not use any external hardware but relies on a computer with a USB port. The touch slider is used for user input which measures values of capacitance. We use UART and serial communication to send a signal from the C-program to the python file to decode the capacitance values into integers. The python file will play a note depending on which area of the touch slider is pressed. These sounds will be played concurrently as the user is pressing the touch slider. It will generate three different types of sounds. When it first runs the code, it would play the default random notes. But, if button SW3 or SW1 is pressed, it would change the notes to three different notes that allow us to play Mary Had a Little Lamb or Hot Cross Buns respectively.

Approach

We were able to get the values and approximate the capacitance values to set the ranges on the touch slider. We used the DigiKey demo code (Using the Capacitive Touch Sensor on the FRDM-KL46Z) and modified it to check what value is within a certain area on the touch slider. We also used the canvas demo code to generate a connection between the UART and serial communication, which decoded and printed the values in the python terminal. After a few observations by playing around with the touch slider, we were able to see certain ranges that can be used to separate the slider into three sections corresponding to different notes. The ranges and notes on a specific note are shown below in Diagram 1.

To play a specific note, we used a built-in Python library called PyGame. We use the midi module in the PyGame library to generate a specific midi-note that corresponds to a note on the piano. After choosing the notes we desired, we set it to output that note on the specific area being touched. Then a set of areas with specific notes would act like a piano that plays the sound for a music piece. While a specific area on the touch slider is being touched and a note is being played, the LED would toggle to tell the user that a note is outputting concurrently with the touch. For LED and sound to display/output concurrency, we use threading on the Python platform. Threading minimized the delay between decoding the values from C to Python and outputting the notes based on the decoded values.

Obstacles

We ran into a few obstacles while implementing this project. We had trouble setting the ranges to separate the regions of the touch slider. The touch slider is very sensitive and depending on how much pressure you press the touch slider, the number would be different. This is due to the capacitance of the touch slider. We figured out a way to minimize this discrepancy by using a rubber stylus that is designed for touch screen devices. Using the stylus gave us more accurate approximate values of the touch slider’s capacitance when choosing ranges to set for the music notes.

Another issue we had was outputting the sound. First, we used VLC sound to output the sound from a ten second mp3 file but the note was not a constant sound and the audio would drop off towards the end. We did not want to find multiple mp3 files to our liking, so we tried using the beep function in the winsound library which played a beep depending on a given frequency and duration. However, that did not solve the problem of playing a plain note and using winsound limited users to only those with a Windows OS. We asked the TA and professor for help on how to improve the output sound and it was recommended that we use the PyGame library. The PyGame midi module allowed us to select a specific instrument such as the piano and output a note by setting a specific value corresponding to a midi-note.

Delays were a main problem that we faced. We realized the connection between C-programming and Python using UART and serial communication caused a noticeable delay. It takes a delay of a few seconds to adapt the signals sent from the board to Python, so the sound would sometimes keep on playing even after we stopped pressing the touch slider. This was due to the constant update from the board to python and the delay of the python sound functions. We were able to solve this problem by using threading to minimize the delays from continuously decoding incoming values from the board and playing notes at the same time which generated a concurrency effect.

Project Video Demo

FRDM-Stylophone Final Video

Detailed Technical Description

In C, we initialized the UART port, touch slider, LEDs, and made the SW1 and SW3 buttons into I/O pins by setting up the appropriate pins. We also set up default values of y=0 and for the LEDs to be off at the start.

The touch slider was implemented by using a touch slider demo from DigiKey which would read the capacitance value from the touch slider when the user touches the touch slider.

In order to connect the embedded system and output, we used the UART demo code to take the raw data when the user touches the touch slider and send it to Python via UART and adding a short delay to slow down the rate at which Python was receiving data so that there would not be any problems in Python when decoding the data.

We used the reset, SW1, and SW3 buttons on the board to change the combination of notes that the board could play back. We implemented this by using the switch demo code that uses interrupts for the SW1 and SW3 buttons. For example, if the SW1 was pressed, we would assign the variable y to be 1, reset the flag, and send that value to Python. The value of y corresponds to a certain range of notes. In this case, since y=1, the three notes that would play would be G7, A7, and B7. Similarly, for SW3, it would change the value of y to 2 and the corresponding notes that would play from Python would be C4, D4, and E4. The reset button would set the value of y to be 0 and this would play random notes.

Otherwise, if no buttons are pressed, the green LED will turn on and the red LED will toggle to signify that the user is playing different notes when they press the touch slider.

In Python, we used threading to make the sound play concurrently as the user was pressing the touch slider. Our first thread decoded the data that was sent via serial connection and UART from the board to Python. Python would read each character and then decode each byte and print the numbers to the console. Then, we made a variable called pressed which would signify which button was pressed by the user if any.

For our second thread, once the input from the board was decoded for both the button input value and touch slider input value, we would enter an infinite loop that first compared the value pressed to 0 , 1, and 2. Then, depending on which button, if any were pressed, we compare the touch slider input value to the preset ranges we coded. For example, if the user pressed SW1 and the value of the touch slider was between 25 and 35, then Python would use the PyGame library and module midi to calculate the frequency from the capacitance value of the touch slider. Afterwards, it would play the sound for that particular note such as C4 in this example for 0.5 seconds. The sound was played by using the PyGame library and using the midi module. The functions we used were player.note_on() and player.note_off() to get the sound and the time.sleep() function was used to play the sound for a specified amount of time which in our case was 0.5 seconds.

The threading in Python was implemented by having the decode function and sound function start by using t1.start() and t2.start() and then waiting for the threads to finish by using t1.join() and t2.join().

Testing

We conducted a lot of testing. We first tested if the UART and serial connection between our C-program and Python file worked properly by following the serial example demo code from canvas. We sent the values from the touch slider to our python file and printed them in the python terminal.

After setting up the ranges, we checked that the ranges were set and worked as expected by using print statements in Python to make sure it printed the right value for a specific area. For example, it would print A, B, or C depending on the area you touch on the touch slider. The area is separated as shown in diagram 1. After that, we tested to see if the correct note outputted for the corresponding area by pressing the touch slider and listening to the notes.

Sometimes, we noticed the note played for the wrong section due to inconsistent pressure being applied. The only way to test that the sound is outputting properly is by playing with the touch slider and running the code every time to test the sound. Also, in our testing, we would try different taps or how frequently we touch the slider. We would do one tap, multiple taps, and keep touching the slider for a few seconds to test if the sound is outputting concurrently. As a user is touching the slider, the LED would toggle to show that a note is outputting and it is touching the slider. This also tests whether the signal is being updated and constantly sent from C to Python. After all these tests on checking the ranges, sound output, and concurrency, a music piece can be played by touching the slider!

Lastly, to test whether the notes change when a button is pressed (SW1 or SW3) we used print statements to see if it gets the signal that whichever button is pressed. For example, when it is the default the pressed variable is set as 0. When SW3 is pressed, pressed would change its value to 1 and it would print “button one” to show that the button is pressed and the notes change corresponding to the button. But, if SW1 is pressed, pressed would change its value to 2 and print “button two” to show that button is pressed and the notes would change. The print statements were very helpful for debugging purposes and allowed us to see which parts of the codes are not working.

Additional Resources Used

PyGame API
UART/Serial Code Example from Canvas
Using the Capacitive Touch Sensor on the FRDM-KL46Z
Switch Demo Code from Canvas

Work Distribution

We worked together by meeting up in the mornings and conducting a three hour work session to get everyone on the same page. In the early stages of working on the project, we divided different tasks to research what components we can use in C, Python, and components on the FRDM - KL46Z. We have a shared google document so everyone has access to the updated information. Some tasks we individually had were: Becky worked on how to get the LCD working, however, we realized UART and displaying the LCD did not work, so we did not include this component. Then, we brainstormed together and decided to do something with the buttons. Katherine looked into the specific notes number on PyGame to get the desired notes and made sure the code is well commented. Zi worked on the write-up for the website to thoroughly describe the project and edited the video. Becky drew the diagrams that can be used to include on the website. Katherine proofread and edited the website write-up. Throughout the process of brainstorming and coding/debugging sessions, we would work together and step through it together to see what could be the problem. Even though members were assigned specific roles, worked on all stages of the project together such as, how to output sound, incorporating the LEDs, and switch buttons. During the times we were not able to meet up, we worked on assigned tasks individually and kept each other updated. Overall, we each had our own role and achieved the goal of this project to output a fabulous music piece by using the touch slider!!! :)