s
Projects
Bandwidth

Watch on Vimeo.

This synaesthetic, interactive musical experience provides six original modes in which the player may produce music. Kick off your shoes and get lost in a world of delicious sounding abstract geometry.

INSTRUCTIONS
Turn up your sound volume.
Click and drag to explore.
Press bubbles to change scenes.
Press Escape to quit.

Android and iOS versions coming soon.

Advanced configuration is possible by editing the contents of the bundled settings.yml. By changing the values of that file, you can control the window properties, startup behavior, user interface detail, and OSC network parameters. When set up on multiple machines, Bandwidth's 'grid' mode will broadcast OSC messages and most other modes will recieve.

Made possible by The Public. UK

Special Thanks:

The OpenFrameworks project,
Graham Peet
Joss Widdowson
Rebecca Shostak
the oooShiny group
Sandwell Arts Trust
Fran McHugh


Creative Commons License
Bandwidth by Josh Nimoy is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

withDRAWal

Watch on Vimeo.

withDRAWal is a series of ten interactive scribble styles commissioned by Graham Peet of The Public in West Bromwich, England. I programmed and tested the whole thing at Beit T'Shuvah. Each mode responds to the scribbling input in a different way. In writing this software using C++ and openGL in OpenFrameworks, I found myself using a maze solving algorithm, the Box2D physics engine, and Perlin noise. Instructions: draw with your mouse, and press the circle at the bottom left corner to move on. If you have a keyboard, press spacebar to clear the screen without moving on. Each number key corresponds to one of the ten modes. Press D to enable the debug overlay. Press - and + to adjust speed or fatness - use at your own risk. Press S to render one screenshot to the data folder. F toggles fullscreen. Q brings back the beginning instruction screen. If you're on a Mac portable that has a sudden motion sensor, mode 5 will let you tilt the screen.

Map for the Sefton Park Walk

A map of Liverpool's Sefton Park for the Sefton Walk Park project, the sound player will follow the trail and even catch its place when the user jumps around in the playback controller. I'm also particularly proud of the black and white version of the map. The map had to be drawn from several references since neither Google, nor conceptual maps have needed to show such a detailed level of trails in the area.

www.hci-fun.org.uk

I redid the website for the human computer interaction organisation in Liverpool England that was commissioning me to do a couple pieces at FACT. I used an open source flavour of flash - mtasc and swfmill with a lot of xml and a makefile. The source was made available on the site. We used the blog engine, b2evolution, to manage the content, and the flash file just pulled the content from the mysql database and presented it this multi-windowed way. Now that I look back on it, Drupal would probably have been more appropriate. The user interface was a bunch of UIComponent windows that popped up inside the browser. Small green indicators and trails were used to point out where the user had clicked and dragged the mouse in an artistic overlay. I subclassed the window component to add a resize control at the bottom right corner.

Icon==Function

This series of seven interactive musical compositions is intended for serious play and wonderment about tools. Each piece is an abstract visual sound scoring language capable of playing itself back as the user gives input. The series is several attempted embodiments of my conceptual response to a recent HCI conclusion, as written in CWC 2004:

"Icon interpretation is inherently ambiguous because the relationship between icon and function is not determined by a set of well-defined syntactic and phonological rules."

Icon==Function is a science fiction, showing a hypothetical dystopia / utopia. What would an interface be like if its iconography were determined by a well-defined ruleset?

The software art was conceptualised by JT in a series of long-winded email discussions with David England and curator Michael Connor. The sketches were then interpreted by Josh, Matthew Phillips, and David Lu in a number of programming languages. The code is hosted by Sourceforge.

hexball coded by jt using lingo

interactive piece

gears coded by jt using p5

interactive piece

nesting circles, coded by jt using p5

interactive piece

triangle music coded by jt using p5

interactive piece

bouncing boxes coded by jt using p5

interactive piece

bouncing boxes coded by Matthew Phillips using as2

interactive piece

colored squares coded by David Lu (aka Forkinsocket) using as2

interactive piece

Each piece in the series is a simplified programming language used to instruct the computer's functional output. A whimsical code editor is exposed to the user as a visual toy that interacts with a mouse and keyboard. No other labeling or indirect tagging is used to mark the mini-programs written with each mini-language. The languages are made of simple geometries - capable of representing themselves as their own icon. The function's code can be seen as its own graphical icon. The well-defined syntax of iconography in each representation is precisely the instruction language controlling the sound output. When a user browses a small set of thumbnails looking for a state to load and play, the user is not seeing a textual name given to the work, nor a proxy graphic that attempts to explain the function by some cultural mapping. Instead, the user sees the code, itself. In the name "Icon==Function," a double equality operator is taken from modern text-based programming to communication that the icon and the function are the same as one another. If a single equality sign were used, it would signify undesirably that the function in these pieces are somehow being assigned to the icon, and that the icon is a transparent address for the function. The art series aims to collapse the two ideas into one just for the sake of both conceptual argument and exploratory scientific research.

Just Language Enough

In computer programming, the word "function" is used to describe a collection of instructions that perform a coherent task. A function hides the code inside of it from attention, and allows its code to be invoked only by calling the function's name. In creating the seven visual languages in this series, my biggest aim was to produce the beginnings of language-ness from my interactive geometries. The aim of adding the modern features of 'real' programming code (such as memory and logical flow of control) was somewhat irrelevant. In most cases, a simple linear list of instructions sufficed in order to make the artistic point, and to provide the software user with an experience that was close enough to programming that they would gain some sense of authorship with their constructions. A stronger focus was also placed on making the experience playful in a game-like way, rather than providing the user with an industrially competent solution.

Usability of the Pie are unfamiliar components that interface designers hope to channel their users to embrace through play or necessity. This is the part of the experience that I am calling the exploration.

In the pieces of Icon==Function, the interfaces I am recycling come from CAD and desktop graphics software applications like a paint program or paste-up application. The colored square game is a slight variation of a pixel editor. The triangle music interface is a modified vector-line drawing tool. "Gear Sound" and "Nested Circles" both rely on circle-drawing tools. Although HexBall heavily resembles a video game, the most basic interface is a pixel editor -- and so on. That is the intuition aspect. The exploration aspect is what happens to those graphics creation interfaces that result in something more than simply the circle that was drawn, or the pixel that was toggled. No one originally expects music to come from their rectangle dragging tool, nor would they expect the element they drew to bounce around on the screen and obey gravity after they drew it. However, this expectation should also become intuitiho are already familiar with painting and graphics applications. Even if draggable rectangles exist as part of widespread commercial GUI OSes as a 2D spatial "multi-select" for file icons, it is questionable what percentage of the user population considers that an integral part of their computing lifestyles and would recognize it in a new program, or know to check for it. My mother (age 59, health care manager) has had considerable difficulty getting into each piece as she must be told to actually drag the mouse instead of just clicking it. After being reminded to drag, she spends most of her time struggling with the touch pad on my laptop, and eventually concludes that the piece is visually beautiful, without delving deeply into trying new combinations. It is interesting for me to see her interact with the pieces because she doesn't seem to understand the language, nor is she searching for one. On the other hand, my brother (musician in his early 20s) is better familiar not only with graphics programs, but video games and software development as well. He has witnessed my software interface experiments since the mid-1990s and already knows that I expect him to explore, given little or no instruction. My brother will start by doing "the sensor dance." The sensor dance is what happens when a human is trying all the different entry points into the interaction that he or she can think of, in hopes to get any sort of response out of the unrecognized interface. While an ambiguous physical installation will cause a person to wave their arms in the air, move close and far, shout, clap, and touch, a desktop computer based work (restricted to a mouse and keyboard) evokes a different kind of sensor dance. The person will click around on everything, dragging the mouse in different places and pressing different keys. In my own observations, people have also been known to re-launch the application to see if it does something new or to check to see if it's broken. My brother is quick to engage in a neactive language, creating new instances and enjoying the outcomes as his own achievements. Since he is a musician, he's familiar with the idea of embracing a system and recombining its parts to produce a new expression. One could say that my interfaces are more intuitive for him. He is familiar on more levels than just see-and-remember. I think these outer, more anthropological scopes of understanding usability could make the problem clearer than simply analyzing the language that translates between icon and function, or icon-system and function-system.

Artist and coder: Josh Nimoy, coder: David Lu, coder: Matthew Phillips, Commission from FACT (www.fact.co.uk) and JMU HCI (www.hci-fun.org.uk) curated by Michael Connor & Marta Ruperez. HCI research headed by David England.

EAZ AR/MR Workshop

Watch on Vimeo.

Report: FACT HCI Augmented Reality Workshop, Success.

In early November 2005, a forty minute educational workshop experience for 300 pupils age 10-11 was given for the Liverpool FOCUS EAZ (Education Action Zone) for schools in the Fazakerley and Walton areas. The technology introduced was addressed at various times as "Augmented Reality," "Camera Vision," "Gestural Interface," and "Illuminating Lamp." Pupils were invited to step into a silhouette-driven environment and interact with six demonstration programs in a game play mode. Additionally, a five minute slideshow was given by the team (Karen Hickling, Josh Nimoy, Marta Ruperez) giving brief backgrounds on FACT (Foundation for Art and Creative Technology) and HCI (Human Computer Interaction group at John Moores University). In preparation for the workshop, Josh had authored (or massaged previously authored) software into an automatic shuffle of six abstract games. Some of the games were more multiplayer than others. Those which were not multiplayer presented limited resource for interactive satisfaction - making great exercises in collaboration, teamwork, and turn-taking for the pupils. The workshop was given a total of ten times. This essay reports on things we learned about the workshop and the HCI as we refined the workshop's structure.

In the planning stages of the workshop, Josh was initially unsure that the camera interface would be appropriate for the age group but proposed it to a few people at FACT. Artist, Carlos "Caen" Botto came back with a strongly positive response, saying, "This kind of activity is very appropriate. The most important needs at this age is that the system have a very evident reactivity (think that at this age the abstract thinking is in early formation), letting the children experience a sensation of control in a very direct way. The other important need is for high physical activity. I think this proposal is right in both aspects. The group size is not a problem. It is important not only to interact, but also to see others interacting with the system. A big group can be broken into three shifts, When one group is interacting, the other groups can observe. On the pedagogic side, I think the educative aspects are centred in psychomotricity, creativity development, cooperation and intuitive problem solving, et cetera." Just before software development, a local toy store was visited in curiosity about which products were targeted for this age range - in hopes to keep the experience from being boring, embarrassing, or too complex for the students. Similar conclusions were reached as Caen's mentions of creative development, psychomotricity, and high physical activity. These children were going to be more energetic than us!

The order of the workshop was originally planned to begin with a 35 minutes of play, with a five minute presentation at the end. The rational for this was so the students would be engaged first, generating questions in their heads. In retrospective analysis, there had been too much preparation to entertain than was actually needed. We soon realised in implementation that workshops lasted for unpredictable durations, sometimes cutting the presentation completely off from the experience. Since the presentation was a relatively important component, it was moved ahead to precede the play session. This change turned out to be beneficial as well. Ideas were put into the heads of the students pre-play, so the play sessions could be more than just play. It was informed play. This was not the only response to chaos. Besides the duration of workshops varying, it was also hard to position ten children in the projection -- it would completely block the projection and although the students were having fun, we wanted them to experience the software in a way that would allow them to comprehend the interactivities. We began to break the groups into three or four, and segmenting each software program into 1-2 minute turns, calling out each group and keeping time. Before we bothered to do this, the turn taking emerged naturally from the group behaviours. It was just faster to impose this early on for punctuality.

We gave very little instruction, and just let the students do what they did. Each group's behaviour evolved in similar ways. As the students discovered the systems, one student would back up so far that the projection was blocked completely by his (it was usually male) body, preventing the rest of the children from interacting with the systems. In the more self-governed clusters, this would result in that person being yelled at by the spectators to "move forward." On the second day, a clear barrier was drawn so that no one could back up too far. This architectural restriction was somewhat of a solution for the "block all" personality type - although this impulse was still observed in students within the remaining space. Students would also pound on the wall where the image was projected, treating the virtual objects as if they were buttons, or as if there were sensors in the walls. The only reason we introduced a rule against the wall pounding was in respect to the workshop happening just in the next room. Other than disturbance, there didn't seem to be any problem with pounding the wall. While on the subject of violence, it was interesting to note the playful violence between boys. Often, when in the shadow space, boys would pretend-fight, as in karate or boxing. This was probably not just due to the high energy levels of the activity, but the expectations that come along with any game-play mentality. The boys felt as though they were "inside the videogame" and were acting accordingly - creating the same scenes they had seen in Street Fighter and Mortal Kombat. Violence was one kind of cheap laugh among the several categories of cheap laughs discovered by the students while in play. On the first day, there was no teacher accompanying the groups. This caused them to be less obedient. However, even if on the second day, there was at least one teacher governing the students, the "play violence" persisted with unusual strength.

A special learning group came in - a smaller group with a wide range of conditions. In this case, the most immediate software worked (the sparkling star trails). The initial concept presentation was cut very short and more time was given to play. This group reacted more or less the same as the other classes of children. It is entertaining to note the democratising power of Augmented Reality: everyone seems to act like a ten-year-old child when they try it out. By the time we got to the last group, students already knew what to expect. Rumours were being passed between friends during breaks and lunchtime. The technology turns out to be not as new and shocking as predicted. Each group was asked to raise hands if someone had an "EyeToy" at home. Three to ten children would always raise their hands, completely aware of what was about to come. On the other hand, slides showing Tom Cruise in Minority Report doing gestures in front of his pre-crime computer-cave went virtually unrecognised. These children were too young to be allowed into the cinema to view this movie with that rating, despite its being responsible for disseminating the idea of AR so widely. The popular question asked was "how does this work?" The next most popular question was "Is this an EyeToy?" Josh's favourite question came from a special learning student: "Are you from America?"

Previously, an emerged personality type was mentioned - the boy who blocked the entire screen. Other such personality types reoccurred during the workshops. The "geek" type would get bored of the demonstration and sit in the back of the room with Josh and the computers, talking about more advanced topics like programming (example question: "Did you do code this in HTML using Notepad?"). The non-participant would sit in the chair, or on the floor and refuse to get up and join the play, even after being prodded by Karen or the teacher. Conversely, and more frequent was the over-active participant -- a child who did not have enough patience to wait for his/her group's turn, would have trouble leaving the game space when the turn was over, and would be found sticking arms and legs into the projector from the perimeters during other groups' turns. Somewhat related to this over-active participant was the shouting director, trying to verbally control who ever was in the ring - telling them to try different positions or interactions. For the most part, children listened to the directors. In the end, we fully realise that it really did not matter what software was running. The most entertaining part of the experience was students being allowed (for once) to let loose in a projection-obstructing frenzy.

Teachers sitting and watching the experience seemed overwhelmed. While only a few of them stepped in to try the interface, everyone had something positive to say. "It crosses the whole curriculum" said one teacher, in reference to the joining of arts and sciences that is the augmented reality field. "You have got a mixed group of people co-operating and working together. Boys mixing with girls and children who do not know each other" said another teacher, amazed at how the experience seemed to break down barriers between different kinds of students.

slow fade grid

slow fade grid

screen capture of a pen-trail

screen capture of a pen-trail

stars

stars

stars

stars

stars

stars

stars

stars

squishy satsumas

squishy satsumas

slow fade grid

slow fade grid

stars

stars

Josh coding on laptop

Josh coding on laptop

exploding watermelons

exploding watermelons

scribble pen

scribble pen

more scribble pen

more scribble pen

scribble pen again

scribble pen again

multi scribble

multi scribble

fade grid

fade grid

squashy satsumas

squashy satsumas

exploding melons

exploding melons