In thinking of physical objects that would work well in AR space, I came across sheet music and couldn’t really shake the idea of how nicely it would work in AR. As someone who’s kept sheet music around for various instruments, one frustration was that sometimes my brain just couldn’t wrap my head around how something was supposed to sound. If I could point my phone at the sheet and have a snippet played for me, it would be able to pick up where I left off right away, just a quick boost to keep my progress up. Here’s a video mockup of what I had in mind.

This was just a quick mockup to get a look going. After looking at it a few times, I think that a further iteration would have individual notes changing color as opposed to the whole bar for more specificity. Maybe some kind of playhead on a timeline would be helpful if the user wanted to go over a specific part.

This is a project that I started back in the beginning of Quarantine but never got back to. I did one quarantine journal entry from my kitchen, and filmed a few more but never built them out in AR. Here I used a generic image tracker on my kitchen island to trigger a video of me cooking in the kitchen. The idea with this project was that someday, maybe long after I passed away, someone could visit my apartment, and if the trackers were placed properly, this visitor could see what it was like for me during the Covid-19 Quarantine.

For my Final Project I’m working with Nok Jangkamolkulchai, (full transparency, this is also my final for Electronic Rituals as well as for Video Sculpture) I am creating a collective Quarantine Journal that will live in a physical installation. This installation will make use of the Pepper’s Ghost effect which simulates a hologram floating in space with the use of a projector, a pane of glass and some tricky reflections.

(Sidenote: The final version of the project relies on me being able to access some equipment that was left on campus at NYU, so this is just a working prototype)

Initially I wanted to use AR to create walking tours of neighborhoods from people who were displaced due to gentrification. When Covid hit, it made it dangerous to think about being out in public to do anything, so the scope of the project had to shift. I made a few image-target based AR experiences to show what my quarantine had been like and decided I wanted to open the project up to community contributions.

Distanced is a physical installation that is made up of a collection of quarantine experiences. By projecting a short loop of someone in quarantine, accompanied by a clip of that person speaking about their experience, the goal is that somehow the shared experience is able to make sense of the time. If not make sense of it, at least describe it.

rough sketch of how the projection is displayed using Pepper’s Ghost

This period of time is going to be a weird thing to explain to people who aren’t here living through it with us. I want this to be a tool to understand at least a bit of what it was like and to go through it by assigning them a role in it.

For this prototype, I’m running everything in a Max/Jitter patch. As a placeholder for the copper boards being touched, I instead just run things from a single button press on my keyboard. Programming the patch to respond to the copper will be simple once I have access to the equipment. The prototype is made from cardboard, but I would like to have it made out wood for the actual piece. Inside of the cardboard is a mini projector that is pointed at a small mirror. That mirror reflects the projection through an angled glass pane onto a solid white surface. It is the reflection from that white surface, viewed through the angled glass that causes the projection to look as if it is floating in space, The Pepper’s Ghost Effect.

*For the randomization of the assets, I wanted to play with the number 19, but I was unable to get it working. This is something that I am still exploring and will be working on after I present this in class. For now I am just using the random function within Max, but figuring out my own system is a priority.

I would love to create something that allows people to easily create and place AR versions of themselves. I initially came to this idea and as way to agitate gentrification. The thinking is that when people are priced out of a neighborhood, after the immediate concerns of finding housing and rebuilding their lives, there’s got to be a fear that everything that they did to contribute to the neighborhood would also be erased. There are memories and stories that gentrification erases forever. Even someone who dies in a neighborhood at least has the chance of being remembered as a ghost. That’s it, AR Ghosts!

If you have a a resident who has been, or is about to be, priced out, what if that person gave a walking tour of the neighborhood as it is/used to be. This would serve multiple purposes:

1. Celebrates the culture that exists/existed in the space before the gentrifiers

2. Informs conscious gentrifiers of the culture that they should learn to honor instead of replace

3. Serves as a “Fuck You” to gentrifiers who don’t care about the damage they do

Showing these 3 images to a classmate was somewhat helpful. They understood what was happening in each frame. They understood that the first frame was the start of an experience, but didn’t quite recognize that the street sign was being scanned to get it going. The next two frames, it was clear to them that the person in the frame of the phone was added in AR and not really on the street with the person holding the phone.

They liked the concept of taking people from neighborhoods that have been, or in danger of being gentrified and having someone native to that neighborhood sharing a narrative that captures the culture of the place as it was (before it’s replaced.).

For my first case study, I played Toko Toko - A journey to augmented creativity. In searching for examples of augmented reality to study, I wanted to find something that went beyond “oh that looks so cool!” Mainly, I was looking for something with a narrative to it, whether it was a movie or a game. Toko Toko is a game where you’re helping an aspiring artist, Hako, find her creative inspiration. You do this by sharing your own real-life drawings with her. Based on what you draw for her, she’ll have guesses as to what they look like and she’ll even find creative uses for them. My first task was to draw hats for her, and while none of her guesses were completely accurate, I could see how my doodles looked like what she saw.

The game has you choose a flat plane to work on, I think I initially made the mistake of choosing an essentially textureless white table that made it hard to track. Because of this, my play circle started drifting as I was playing. I also think that my iPhone 7 might’ve had something to do with it, maybe it’s time for me to upgrade.

As to whether this piece was a sensible use of AR, I think it worked really well. It wasn’t a distraction from the narrative and, aside from the drifting (which I think is more my fault [but hey! learn to teach your users how to use your product!]) it was a really pleasant experience that fit my criteria of something that was more than just cool-looking.

There’s something about playing the game in a public space that still has me feeling a bit self-conscious. There was a point where the game had me pointing my phone in the direction of someone I didn’t know, so I was worried that she thought I was filming her. I explained that I was playing a game before she asked anything, but I would’ve rather not been in that situation. I imagine as AR becomes more prevalent, people would have less reason to feel awkward about it, but I’m definitely not at that point yet.

The overall design of the game made it easy to feel a connection to the plot and the characters almost immediately. They’re cute little characters looking to you for help. The way that the game takes your drawn assets and incorporates them into the game is a seamless transaction that is highly satisfying. The extra touch of having Hako guess that my sorry excuse for a Basquiat-esque crown was blades of grass, and then put it on her head anyway was a huge highlight. I definitely plan to spend more time playing this game, just maybe from the comfort of my home instead.