Hardware Prototyping and Development

The Concept

TacTile is a touch-sensitive fabric matrix intended to function as a New Interface for Musical Expression (NIME). This musical controller is meant primarily for personal use by me, ideally for composition and improvisation. It is supposed to satisfy the following design criteria:

1. Sensory Interaction (Tactile, Visual and Aural)
2. Responsiveness
3. Portability

How Does It Work?

The design features 2 mutually perpendicular sets of parallel strips of conductive fabric separated by a piezoresistive material (velostat, in this case). Both sets of strips connect to a microcontroller, one set to digital pins, the other to analog with the latter reading off values.

Metal snaps are used to bridge the connection between soft conductive fabric and hard wires. The wires are a 2 ribbons of 10 wires with female header pin sockets. These are connected to another 2 ribbons with male headers on both sides which go to a breadboard. This extra length allows the microcontroller to be placed further away from the fabric, ensuring a flat and even playing surface.

The function is based on a voltage divider circuit. When no pressure is applied, the velostat is resistive and no (or minimal) current passes through. As pressure is applied, the resistance of the velostat lowers and more current passes through, increasing the analog input value being received at the microcontroller.

These values are interpreted and sent to either Processing or Max/MSP for further processing where every value is used to populate one cell of a matrix to either generate visuals or to produce sounds.

The sound generation relies on X and Y position which are extracted using computer vision (see this link for further details).

Project Context

The design and code draws on openly available resources from and owes significant thanks to Maurin Donneaud, Plusea, Kobakant/How to Get What You Want and Tom Igoe.

The basic tutorial and starting code for this project can be found on Kobakant at this link. The more advanced version, which wasn’t followed step-by-step, however was a very good source of information and useful point of reference is this Instructables article on the rSkin by Plusea.

A few key first inspirations and points of reference for this project were:

eTextile

Maurin Donneaud developed this interface called the eTextile Sensor. I first found it while sifting through the NIME repository in a paper he submitted in 2017 along with Paul Strohmeier at this link.

The hardware and software he uses are both more complex than the ones I employ. My interface can be looked at as a direct descended of his – albeit much more simplified. His documentation can be found online at this link and is intended to be open source and freely available to professionals and novices. I used these resources as well as the Kobakant Matrix References to reverse-engineer his design and develop my NIME.

Maurin has a very sophisticated blob-tracking software that detects not only multiple touches, but also fine variations and wide ranges of pressure. He has worked on improvements and variations on this same idea for a long time and has made an XY Fabric Controller (website and GitHub Resource), an eTextile Synth and various layouts for it.

Madrona Labs Soundplane Model A

The Soundplane is an expressive musical instrument designed and developed by Madrona Labs. In their own words:

“The Soundplane Model A is a computer music controller with the sensitivity and feel of an acoustic instrument. It detects a wide range of touches on its walnut playing surface, from a light tickle to a very firm press. Unlike a MIDI keyboard, which typically sends out just one velocity value at the start of a note, the Soundplane communicates three dimensions of information, x, y and pressure, over the entire duration of every touch.” [source] https://madronalabs.com/hardware

I used Adam Tindale’s Soundplane as a device for sensory exploration to see which new expressive possibilities could open up. What I learned from this experience and further thoughts on it can be viewed at: [NIME Post Link]

But to summarise, it gave me an interesting window into a new way of making music. The key insights it gave me were a visceral sense for and appreciation of the role of touch in making sound and music, one of the ways in which that can be leveraged using Max and the importance of mappings.

Novation Launchpad Pro

The Novation Launchpad Pro is a widely available commercial MIDI controller. It is designed to work with many popular professional Digital Audio Workstations (DAWs) and is extremely flexible in how one can incorporate it into their workflow. It features multiple kinds of messages and modes for music creation including mixer control, launching clips and tracks, playing notes and scales, drum machine control, chord mode and a sequencer.

This MIDI controller was also used by me during reading week to further explore how mappings and deep integration influence the interaction experience and process of making music. I found this study to be very immersive and played multiple sessions with it over a few hours without having to ever look at my computer’s screen.

https://novationmusic.com/products/launchpad-pro-mk3

Sensel Morph

The Sensel Morph is an innovative touch-based flexible controller that changes into various modes depending on the current magnetic overlay being used. Although its recently been discontinued, its an interesting and well-loved example of how a fresh take on not just touch-based interfaces for music control, but HCI in general.

https://morph.sensel.com/pages/video

Honourable Mentions

Other notable projects and products that were explored in this domain are:

• Joué Board
• ROLI Lightpad Blocks
• Monome Grid
• Embodme Erae Touch

Parts, Materials and Technical Assets

Parts and Materials List

The following parts and materials were used in the making of this project:

1. Conductive Fabric – Ripstop
2. Cotton Fabric Scraps
3. Snaps & Setter Kit by ArtMinds (Item # 10043798)
4. 10 pin Ribbon Wire with F Headers (Creatron) x2
5. Jumper Wire Ribbon with M/M Header Pins 10 x2 (Creatron)
6. Solder (flux core, lead-free)
7. Arduino Nano 33 IOT
8. Teensy 3.2
9. Micro USB Data Cable
10. Alligator Clips
11. Masking Tape
12. Painter’s Tape

Equipment

As for equipment, I used:

1. Speedy 300 Laser Cutter
2. Drill Press to press fit the metal snaps
3. Scissors
4. Clothes Iron
5. Hammer
6. Soldering Iron
7. Brass Wool
8. Metal File
9. Cricut Maker 3
10. Cricut Purple Backing Mat and Tools

Circuit Diagram

The circuit diagram is fairly simple: the top set of conductive fabric and wires connect to digital pins 0-9 (brown to black, shown here on the right side of the Teensy 3.2) and the bottom set connects to analog pins A9-A0 (brown to black, shown on the left side of the controller).

Code

The Arduino and Processing Code as well as the Max/MSP patches can be found on GitHub at:

Design Criteria Assessment

The design criteria for this project and their assessment are as follows:

1. Sensory Interaction (Tactile, Visual and Aural) The interface allows one to interact with it and reads tactile information. It also outputs visual and aural feedback to inform the user of what result their actions are producing. All three dimensions, while good, could use some improvement:
   • Tactile – Having some embossing or ridges to allow users to sense where their finger is without looking could improve touch-based feedback and work synergistically with the other senses
   • Visual – While good for the time being and given time constraints, in the future, LEDs or other visual media (like projectors perhaps) could be incorporated so that users rely less on a laptop screen. This might improve the immersive aspect and ironically put sound as well as the creative aspect in focus.
   • Sound – Another consequence of the time constraints, the sound currently only reacts to change in the X direction. Y and Z direction (or area/pressure) can be leveraged to allow more dimensions of expression. The pitch ranges and intermediate intervals are flexible currently but quantising them to standard pitches would allow one to actually play tunes, songs and beats – significantly improving usability and in many more contexts.
2. Responsiveness The design is responsive to single touches and there is no perceptible latency between touch, information communication, processing and sound generation. This means that the device does what you expect it to, and does so when expected. This could be improved upon by incorporating multi-touch to play many sounds simultaneously – something which is expected when studying user behaviour, but missing at the moment.
3. Portability The design, when properly assembled, is easy to fold or roll up, carry and reinstall at another location. This works well because most of the material used is fabric, light metal and plastic, and the cables are managed well. A smaller breadboard and/or protoboard/PCB could be used to further improve portability. A small pocket on the panel itself to house the microcontroller and conductive traces to it could improve this even further.

Challenges and Successes

Challenges

• Microcontroller The biggest challenge was the microcontroller. I wrestled with incorrect readings for weeks because I was using the Arduino Nano 33 IoT. This is not supposed to have INPUT_PULLUP on the analog pins which was an essential part of my code. After trying every possible alternative, I finally switched the microcontroller to a Teensy 3.2 and it worked instantly.
• Snaps The type of metal snaps I used are the ring type. These have one pair of pieces that press fits to one piece of fabric and another pair that press fits onto the second piece of fabric. The pair with the smooth outer part actually has a loose metal ring inside which ‘catches’ the other half. This part being loose made the electrical connection unreliable – so I soldered the wire to the inside of it. This, however, changed its shape and made it so that it wouldn’t fit onto the other piece. So I had to tighten the two pieces in a vice to get them to mate properly (this made some of the snaps bend and deform in uncomfortable ways). The second set of snaps that I soldered the wires to turned out better: I put only a tiny bit of solder onto the very outside edge to keep the ring from moving and establish a strong electrical connection. This set mated with the other halves much more easily and cleanly, although a few still needed a gentle push from the vice.
• Laser Cutting
  • I decided to laser cut the black backing fabric. This was from scraps that I’d picked up from the OCAD U Textile Studio so I didn’t know what kind of fabric it was and therefore what laser cutting settings to use for it. I also had enough to run tests, but didn’t have time to do that so I went with one pass and hoped for the best. The cut didn’t go all the way through so even after laser cutting, I had to manually cut and punch holes into the appropriate places – but at least this gave me some accurate markings and a reference.
  • The other major issue with this was that the laser cutter burnt the fabric. Not only did this make the edge quite frayed, it also produced an extremely strong burnt smell that was hard to ignore. I was only able to fix it by making sure I washed the fabric with soap.
  • In my rush to get the job finished before the XFab Lab closed, I also screwed up here by not mirroring the top fabric panel to match the arrangement of the top conductive fabric strips. The asymmetrical layout made it so I couldn’t just get away with flipping the panel over. This had the unintended benefit of exposing some portion of the lower strips and wiring – giving users a window into (and me an easy way to lift the top panel to show them) how the device worked – a detail which many people appreciated, regardless of them realising whether it was intentional or not.
  • Laser cutting the conductive fabric also worked in strange ways. There were some rounded corners that refused to cut – although they still produced very faint markings. The laser cutter cut out one or two circles but ignored every other curve in the file for some reasons. No solution at the machine itself really fixed this, I had to manually cut these out and thankfully they were only very short segments. This did have the benefit of the strips not completely separating from the rest of the fabric which helped in both transportation and transferring them onto the backing fabric.
• Cricut Settings and Transfer
  • The Cricut gave me similar issues with the conductive fabric. The very first set I cut using it turned out perfectly. I purposely cut it with the conductive side facing up because I wanted to be able to get it off with minimal (or preferably no) lateral movement that disturbed the arrangement. I chose to transfer it onto a makeshift ‘patch’ of painters tape and stick it onto the black fabric using the clothes iron.
  • The second time I did this, I thought placing the fabric conductive side down would make it easier to transfer onto the black fabric (I couldn’t be more wrong – how did I think I would get it off from the sticky mat or apply heat from the clothes iron to the reverse side of either the black fabric or the cricut mat?). Despite using the exact same settings as the last time (unless someone messed with them without my knowledge) the conductive fabric never cut all the way through. It seemed as if the machine only cut through the iron-on backing of the fabric and not the weave of the fabric itself – the part that actually holds it together and gives it strength. In the end, this piece of conductive fabric didn’t cut properly and it looked like cutting through it manually wasn’t going to work either. This was a very costly lesson learnt.
• Max/MSP Serial Limit The 10×10 matrix worked well in Processing but refused to work with Max. The method I’m currently using is to send tab-separated values of every intersection point (100 for a 10×10 matrix) per reading to Max which then sequentially puts each reading into a cell in a matrix object in the program. This seems to top out at roughly 60-62 values and refuses to take in any more values. I still haven’t completely solved this issue except to scale down to a 7×7 matrix (for a total of 49 values, 8×8 is 64 and just high enough that it crosses the apparent limit).

Successes

• I got the interaction and sound itself to work! After weeks of not having it work, the excitement of getting it to actually do something I wanted was amazing (and a great relief).
• The visitors at OPEN Show really liked the project and I received some great encouragement, excitement, feedback and constructive criticism from them about the project. It also worked as amazing user testing and will inform my future explorations in this realm – most notably that whether they realised it or not, people expect it to be able to play multiple notes at the same time, to have the Y-axis modify the sound in a more meaningful way and to have quantisation (the ability to play discrete notes of a scale).
• My soldering technique has improved tremendously. This comes as a result of a few realisations: Keep the tip clean – I knew this already and that I much prefer the brass wool to the wet sponge for this, but I made this work perfectly with the Hako irons in XFab, but I just could NOT get the Weller irons in the studio to cooperate, no matter what I did (especially during the Solar Wearable project, I even tried tip cleaner). Until a good hack that I got told me to gently file the oxidation off till the tip was shiny (taking care not to go too deep and damage the protective finish). I then made sure to use flux core solder and tin the tip if I left it unused and exposed (even for a few minutes at a time) and I was done in a fraction of the time it took me earlier.

Next Steps

• Incorporating Visual Feedback
• More tactile feedback (different materials + laser cutting for embossed/debossed effect)
• Smoother tracking (possible solutions: varying conductive material and its sizing and/or spacing, inserting thicker middle layer to spread pressure out more across various conductive elements, using FSRs?)
• Multi-touch with multiple blob tracking
• Scaling up (sorting out Max/MSP serial communication limit issue or failing that, picking a different program/language [JS?]; more number of conductive elements, multiplexers?)
• Pitch quantisation, experiment with modes (like MIDI notes and/or control change and/or program change messages, MIDI MPE, OSC, possibly making a sequencer, etc), gesture recognition (like drawing circles to turn volume up or down)
• Develop this for a future independent study/thesis project(?)