ICM Final: The Impulse and the Response

For my final project in Intro to Computational Media, I decided to focus on the p5 Sound library’s convolution reverb functionality. Reference for that object can be found here. Technically, I have been interested in p5’s ability to handle sound in the browser. Conceptually, I have been interested in convolution reverb as a medium for artistic messaging. In terms of my broader practice, I was excited to not only produce work in a collaborative manner but also create a tool for artists to use.

 

The result is a two part final. One, is a piece titled “The Impulse and the Response”. Two, is a browser based tool to be given to artists in order to create more pieces of a similar nature as this one.

“The Impulse and the Response”

This piece consists of two interactive sound pieces to be played in the browser alongside explanatory text. Two artists, Édgar J. Ulloa and Daniela Benitez, provided audio performances. A slider is available to change the amount of reverb that affects the audio that is playing. The reverb is a re-creation of the sonic qualities of the inside of the Statue of Liberty. In collaboration, my prompt to the artists was, “If you could do a sound performance inside of the Statue of Liberty, what would it be?”

Screenshot of “The Impulse and the Response”

Traditionally, convolution reverb is used to easily recreate reverbs of sonically pleasing spaces. Even experimentally, non-traditional use of convolution reverb focuses on innovative sound design results. I was interested in the potential for political or social commentary. Both of my collaborators describe themselves as affected by the United State’s immigration policy, social attitudes towards immigration policy, and the general climate of xenophobia. They wished to make pieces addressing these issues.

Giving them the opportunity to virtually inhabit this symbolic space not only gives potential to critique the space and it’s multi-dimensional symbolism, but uses the act of recreation of the space and it’s virtualization to match the physical, psychological, and political state of flux that many immigrants can face.

Challenges

The artistic metaphor completely relies on the technical understanding of how a convolution reverb works. However, most official definitions of convolution reverb err on the technical side and as a result are not always helpful descriptions to the average person. With this in mind, I decided that a brief but thorough artist’s statement that explains convolution reverb was essential. With that in mind, I will link to the piece here so that you can read about convolution reverb and it’s relevance to the piece in detail.

The Impulse and the Response

First approaches

My initial idea was about the general potential of convolution reverb as an artistic message. I had less thought about the possibilities of recreating spaces, but more of putting audio “through” another sound. Since convolution reverb works by loading other sounds, called impulse responses, to create reverberations, you can put any sound into it for unexpected results. The sound of the arctic shelf cracking and falling into the ocean could be used as an impulse response in a sound piece about global warming, for example.

My first presentation on the idea revolved around police shootings. Loud pops and bangs are used to create traditional impulse responses, and on the professional level may be done with starter pistols. I had mocked up an interface for a piece that would use audio from the shooting of Keith Lamont Scott as the impulse response for a convolution reverb. The user would be prompted to read news articles about the shooting.

The general approach was to use convolution reverb to achieve a kind of empathy that may not be available in other forms of media. Having your voice brought into this “space” created by the reverb may more easily prompt you to think about yourself in that space. After discussion the general concept with Allison Parrish, it was apparent that there was the potential for very problematic usage of this approach. Not only in the procuring of media as appropriation, but the general feeling of a serious and fatal issue boiled down into a “sound toy” that you play with.

Allison’s advice to hold the technology and the message in separate spaces while meditating on potential concepts was invaluable, and lead me to the current incarnation of the project. I feel that what I have is a much more positive approach while still having a strong social message, and is empowering to artists. This revolves around the second part of my project.

Convolution Reverb Online

At the bottom of the piece is a link where you can create similar pieces of your own. The broader vision for this is that I can provide this link to Édgar, Daniela, and other potential collaborators so that they can create their own versions of this kind of piece without having to rely on sound engineers or programmers.

Here is a screenshot of an initial prototype:

In trying to achieve a user friendly user experience, I learned much about Javascript , p5, and browser capabilities in general. This has been very enlightening, and I am very glad I could work on practical Javascript usage in this project. Some decisions while not practical, offered good practice; like trying to program as much of the UI as I could inside of Javascript instead of pre-made HTML. Other learning points were great research; iOS does not allow Javascript access to the microphone. Not just Safari, but iOS as a system. And I wound up writing some code I imagine myself coming back to over and over, like file uploads that can use system menus as well as drag and drop functionality.

Screenshot of Convolution Reverb Online tool

Source Code for Convolution Reverb Online

The Experience

While there are things to revisit on this project in terms of design, layout, and code cleanliness, I am really happy with the results. It has been a while since I tried to use solely Javascript for interactive work, and being able to immerse myself in p5 has been a great re-introduction to what the browser is capable of. And more so, the ethos of p5 was fully felt. I was able to program something novel and expressive without the syntax and technical aspects getting in the way. My classmates’ wonderful feedback, and Allison Parrish’s guidance was so important and appreciated. I’m proud of the end result, and am motivated to continue exploring programming as an artistic skill. It also seems that this project will have a life after this class, which I am very excited about further developing.

PComp: Final Project Update

This past week we had a play testing session with one of the other Physical Computing sections. 

It was good to see people who hadn’t heard of my project before approach it for the first time. Also, it seems that the people with musical performance and production experience almost immediately understood the usage and potential. Very encouraging.

I’ve been a bit obsessed with having a low profile and slick fabrication of the Mic Cuff. The grip and feel of it is as much a functional interface question as it is about the aesthetic appeal. I went out on a limb with an idea, trying to work with bent acrylic.

I had done some measurements and cut a couple rectangles to work with. But I needed to practice on a scrap piece first and see how this might work.

Definitely takes a while to understand the feel of bending acrylic, but I got something functional. And I was pretty happy with the sturdiness.

Moving onto the main piece.

Not too bad. Felt good in the hand. The next step was figuring out how to mount onto the surface effectively. Keep in mind, the inside of the cuff has to be smooth in order to accommodate the mic.

In order to make it as light and as uncluttered as possible, I decided to move the board off of the cuff. Wires will go to a separate controller, which will then plug into the computer via USB.

Moving onto my button mounting concept

I want to use cork as a way to mount the buttons. The protoboard, while providing a space and security for soldering, can also serve as a kind of “slot” system for inserting the buttons. Cork provides a kind of tacticle/ergonomic “landing pad” for fingers and finger tips to grip the mic.

I played around with a few different approaches.

Cracking can potentially be an issue, but I found a solution that so far has felt fairly secure.

Then I further sculpted the cork to create more of a natural grip.

As of right now, I have mounted the pieces with double sided tape. Fairly secure but seems to be remove-able and not irrevocably permanent.

The grip feels good in the hand, and I am actually surprised this works as well as it does. I’ve ordered some terminal blocks off of Adafruit so I can attach the wires to the Teensy without having to permanently attach them (in case I need to trouble shoot) but keeping it sturdy and secure (so I hopefully don’t have to trouble shoot).

Feeling good about the physical progress. I’m going to continue refining the software while I wait for my order to come in.

Intro to Fab 6: Mounting Motors

Our last assignment for class was working with motors: attaching them to things, and attaching things to them. There is a class at ITP called “If it moves, it breaks”, and I certainly now know why. Getting things to move can be a huge pain, even in my simple attempt.

In our second assignment, we were asked to make multiples of something. I had originally set out to make a kind of a musical instrument involving motors. At the end, I had everything I needed… besides something attached to the motors that could spin around and hit things to make sound. I knew we would be returning to motors later on in the semester, so I was happy to come back to the concept.

A different previous project also inspired me, which lead me to use cork again as a material. I went out and bought some materials and got started.

First was the screw block terminal, in order to get out the inner shaft couplers. I didn’t realize how easy it would be to damage the inner components by sanding too much.

However, learning my lesson, I got into a groove and took out multiple shaft couplers. They fit on my DC motor nicely.

I wanted to have some kind of soft, flexible percussive element screwed in securely to the other side of the shaft coupler. I clipped the metal pin to one side of a jumper cable and then secured the other end into the coupler.

Then I turned on the motor:

And it immediately twisted up on itself. I thought maybe it was spinning too fast, so I tried to slow down the speed, straightened it out, and then hit it again.

It spun so fast that the plastic contact wore out in a second and sent the cable flying. Not the type of percussive effect I originally had in mind.

Maybe come back to that later, and moved on to mounting the motor.

I still my cork from the fastening project, so I decided I would use the pipes from that and mount the motor to the cork. I hit it with the band saw…

And then tried the other cut, and realized there were space issues on the band saw. I really wanted to keep the band saw fence so I could get somewhat straight results, so I flipped the cork and cut.

I had gone to the hardware store and gotten the metal bands with circles in them. We didn’t have any snips in the shop that could cut them, but the nibbler came through to save the day.

Then shaping the form of the band. Not as easy as it looks. The stiffness didn’t allow for very elegant shaping, but it did feel extremely sturdy.

Now back to my original issue:

I created a 90 degree hinge for the couplers. (The straight male to male headers had more distance than the pre-bent 90 degree headers). Then I mounted both, thinking that perhaps giving the thread it’s own straight direction away from the axle, it might not get tangled on itself.

Plus a little bit of weight at the end. But nothing too heavy, so it was less of a weapon and more of an instrument. I had kept these foam balls I had bought for the original multiples project. Hot glue and some tape to prevent the foam from chipping and making dust everywhere (it’s a kind of… crispy? foam that flakes off).

And the test:

Yeesh. It looks to me like I will either need more distance from the shaft or a mounted wheel to achieve the effect I am going for. I’ve also considered flexible, but a bit more firm material than something like string. Maybe fishing line or a semi-rigid wire? I’m going to continue to hunt for the appropriate materials and try a few things out. But for now, some photos of what I have:

12-14-16, I’m adding some extra progress

I went to the hardware store and got some thin but stiff wire. Something light, but could stay somewhat rigid.

It was stiff enough to hold a (somewhat) 90 degree angle. It was actually nice to be able to bend it, as it gives the ultimate purpose of making noise a little more flexibility. I reused the ball and hot glued to secure.

But in order to give it a somewhat more finished look, I used the denim I had to create a soft type of mallet head. This is temporarily secured with a rubber band. I imagine in the future being able to wrap different materials around the ball in order to create different sound qualities.

The first arm length seemed to be a little too long, so took out the wire and shortened it.

Better, but not quite cutting it.

Starting to get a good effect, but running out of room.

I think I’m going to continue experimenting with different tips. The speed of the motor on it’s own seems so powerful, but when you put any kind of load on it, it slows down dramatically. I’m thinking of perhaps doing a simple rubber coating of the tip to make it as light as possible without leaving a sharp edge. There are a few other avenues I could take, and I’m excited to come up with more ideas.

Intro to Fab 4: Materials and Fasteners

We have been pushed away from acrylic and plywood and towards… well, anything else! This week’s assignment is materials and fasteners. Exploring the broader world of materials and their benefits (and drawbacks) is important. And knowing how to attach them together wouldn’t be bad, either. But I was also really happy to see Ben’s emphasis on how fasteners can be emphasized rather than hidden. Sometimes you want to “celebrate the fastener”. Really inspired by this assignment.

img_20161201_182541

I wanted to work with cork. Cork boards are one of my design go-to’s for personal spaces, as it allows me to attach and remove things at will. I like that re-usable and flexible aspect of being able to stick things into cork. Being able to change what is on my wall quickly, trying something out but take it down if I want without damaging a wall, it is great. That might be justifying my commitment issues, but that is the topic for a different blog post.

And despite it’s re-useability, it doesn’t (have to) look cheap. The raw texture of it appeals to me, and the cork I got was a yoga block from REI so it looked better than some of the flat cork board material I have gotten from Staples in the past.

I wanted to use brass elements, as I haven’t really done much with metal yet. At the hardware store I got the last four of 5/8″ pieces of brass plumping pipes, and some decorative fasteners I thought might match.

 

img_20161201_182859img_20161201_182903

I wanted to see how soft the cork was to work with. It was light, but felt dense at the same time. Pressing one of the brass pipes into it, with a decent amount of force, left a mark. If I really pushed or used a hammer, I could probably force it into the cork.

img_20161201_183841

But I wasn’t going to take that approach. The 5/8″ spade drill bit was perfectly just smaller than the tube, which I wanted to mount into the cork. The sizing made me optimistic for a friction fit.

img_20161201_185800 img_20161201_190013 img_20161201_190150 img_20161201_190334 img_20161201_190344

With some generous advice from the shop staff (again! I’m growing to love them), I setup the cork block to be cut down the middle. We weren’t sure what to expect, but it cut like butter. Very minimal amount of dust, and you would barely be able to tell that it was cut at all from a distance if they were held together. And then they smoothly rub together with no friction. They were fun to just kind of touch and play around with like a toy. I’m looking for excuses to use cork in the future because it is so fun to cut in this way.

img_20161201_190433 img_20161201_190725 img_20161201_193128 img_20161201_193321 img_20161201_193443 img_20161201_193529

I figured out where I wanted to mount my pipes, marked both pieces appropriately, and setup the drill press. I figured out how much depth I wanted to achieve, and made a placement jig so that I could quickly drill the holes in a consistent manner. Coring out the cork was easy, but very messy. LOTS of cork dust. I tested the pipe, and at first was afraid the hole might be a little *too* small. But that turned out to be perfect. Great enough for a friction fit, but I could pull it back out if I needed.

img_20161201_194901 img_20161201_195029 img_20161201_195727 img_20161201_200306 img_20161201_201628

Grrrrrr! The stickers on the pipes could not be any stickier or harder to peel off. Once I finally did get them off by using the shop sink, there was still adhesive on the surface. I took some isopropyl alcohol and rubbed the gunk off of them, which took longer than I thought.

img_20161201_201638 img_20161201_202109 img_20161201_202117

Fasteners. I had gotten little golden colored hooks that had threaded screws as their base. Marking out the middle of one of the cork pieces, I put the hook in the center of it.

img_20161201_202131 img_20161201_202143 img_20161201_202155

After that, I hung the metal carabiner/key holder on the hook. That this point, I thought that I had something compelling looking. I have consciously not told you what I was making up until this point, because I wasn’t entirely sure what it was that I had in my head when I started. I’m imagining many potential uses for this basic form. Maybe hanging a small succulent to dangle in the middle, or a small light source. Or maybe it is just a really fancy key holder.

The real point of it is that you can tack things into it and change it’s appearance and functionality. With this in mind, I moved onto my next step and next material.img_20161201_202921 img_20161201_203209 img_20161201_204130

I had a pair of jeans that had been torn very badly at the knees and were only getting worse. I decided to gift myself a new pair of jean shorts and get some material for my homework at the same time. I used a seam ripper from the soft lab and flattened out both pieces.

img_20161201_204332 img_20161201_204337 img_20161201_204440 img_20161201_204443 img_20161201_204708 img_20161201_205721

I wasn’t sure which fastener I wanted to use to connect the denim to the cork. But thats ok, because its cork! I can always take them out and put in something else. I opted for the smaller nails, and left them a little elevated in order to make them more prominent and noticeable.

img_20161201_205727 img_20161201_210337 img_20161201_210316 img_20161201_210239

img_20161201_210212 img_20161201_210156

While I could push them in for the most part, at some points it was way more difficult so I used a mallet. I also decided I liked the inside of the denim better than the outside, so I undid everything and flipped it (thanks again, cork!).

img_20161201_210623 img_20161201_210640 img_20161201_210647 img_20161201_210726

I then hung a portable speaker in the middle. It can open up and hold a phone inside of it. This was one of my initial ideas, that I could kind of create a speaker enclosure that looks pretty by making something that wraps around a pre-existing speaker.

When first looking for materials, I had the “cork + warm metal” material palette going. But I was afraid it was going to get a little too steampunk-y. But again, the fact that you can stick anything into the cork helps with the potential versatility a lot. Yuèpíng was nice enough to let me borrow a couple of her pins to test out my customization theory.

 

img_20161201_212511 img_20161201_212521 img_20161201_212626 img_20161201_212639

Adding and removing the pins left minimal wear. I’m sure if this was used constantly, eventually the materials would look a little worse for wear. But for the time being, the use case worked. I’m happy with the end result… even if I’m not entirely sure what this thing is. It is sturdy enough to stand on it’s own and hold certain amounts of weight, but also temporary, flexible and re-useable. Really glad I got a chance to play with cork, and will be using it again in the future. Going to try and hunt down some affordable sources to stock up.

IDS Final: ACM Digital Library reseach

For the IDS final, I’m going to be combining with my Physical Computation final to build an interface for my Mic Cuff Controller. When searching around the ACM Digital Library site for articles, I found some interesting similar takes on what I’m trying to physically build.

The E-Mic seems to be in a similar realm. Incidentally, one of the prototype proposals on page 5 is the closest thing I’ve found to what I’ve been physically trying to implement (fig 22):

http://dl.acm.org/citation.cfm?id=1085743

And off the site, a prototype called “Project Tahoe”:

http://vhosts.eecs.umich.edu/nime2012/Proceedings/papers/202_Final_Manuscript.pdf

Though, I believe one of the more valuable discoveries in my ACM DL searches was discovered when I started broadening my search terms. I’m starting to commit more to the physical shape of my device, and I know that at least for now I want to stay somewhat focused on musical applications for it. I want an engaging, fun user testing scenario and I think musical activity will bring that about.

However, because I want to focus on a more casual user for the time being, I need to be careful about the software that is being used to demonstrate what the Mic Cuff Controller can do. With a certain amount of functional prototyping done, I am starting to focus on making user friendly software that allows for customization.

Being able to let users who may not be musically inclined play with a musical device offers certain design challenges. We want the users to be able to make changes to the state of playback, and be able to clearly implement decisions. However, an overly technical UI could be overwhelming, and may cause users to disengage. Though in trying to course correct for that, you could wind up taking away too many options and wind up with functionality that is too limited. This could also lead to users disengaging if they can’t do much with the program. My IDS final will be focusing on how to strike this balance.

I found a really interesting paper related to these ideas:

User Customization of a Word Processor

http://dl.acm.org/citation.cfm?id=238541&CFID=870155206&CFTOKEN=24252391

This is a 1996 paper from Stanley R. Page, Todd J. Johnsgard, Uhl Albert, and C. Dennis Allen about “identifying the customization changes users typically make to their word processors.” The main takeaways are that many people customize their software when given the chance, and that the researchers didn’t expect these findings. “A surprising 92% of the participants in this study did some form of customization of the software.” Additionally, the only predictive indicator of a person to customize was how often they used the program. Five categories of customization activities were identified: 1) customizing functionality, 2) customizing access to interface tools, 3) customizing access to functionality, 4) customizing visual appearance, and 5) setting general preferences. (“Findings”, p342-343)

I could go on with everything that has inspired me about this paper, but then I would just wind up re-typing the entire paper all out on this blog. Even though this research was done 20 years ago, I think this paper is a great reminder of fundamental UI principles and customization best practices. I’ve been meditating on this research and thinking of the best ways to heed it’s advice or push the ideas forward. One lead I’ve been pursuing pertains to the last paragraph:

“Finally, studying the patterns of users’ customization should help us move our adaptable systems to self adapting ones; systems that will accurately anticipate the customers’ work and respond by providing the appropriate tools and services. To this end we need to continue to study when, why, and how users tailor software to accomplish their work.” (p345)

A concept to integrate the desire for more study would be to integrate the study into the interface itself. A program that could analyze its own usage could be able to change its interface based off of that data. Or, more precisely, prompt the user with an opportunity to change the interface themselves. A simple example would be to track the amount of times the user has opened the program. If after, say, 50 times using the program, the user has still not entered an “options” menu to customize the program, then upon the 51st time opening the program prompts the user with information about the options menu and what it can do for them.

These ideas became a little more tangential from what I was originally trying to achieve with my project, but I am going to see if I can integrate them into the work. The danger would be basically re-creating Microsoft’s much hated “Clippy” word processing assistant feature. But I think there can be something between that and a complete lack of communicated information about a given interface. Given the prevalence of customization among all users, and the knowledge that more frequent users are more likely to customize, I think that some form of this approach could be interesting to try.

Even though if that particular experiment is more appropriate for another project, the paper has been a great source of research. It is good to know that users *will* customize, why they customize, and how they customize.