Week 6
During week 6, I finished preparations for files to submit for the MFA review. I uploaded narrative and process book files to my presentation site at http://www.duanecash-unleashed.com.
I have also been continuing research into more texts on BCI technology.
Week 5 Updates
This week I continued make updates to the Presentation website.
I made adjustments to the Past Work page by adding another portfolio item. I am working the narrative and process books to ready them for the review. These items will be included as links in the Past Work area when they are ready.
Next week I will continue work on the written component of thesis and additional prototype items.
This week, I have made some updates to the M.F.A. Presentation website.
I created a new section for Contributions to my field. I also updated my Gap Analysis/Timeline to reflect the current status of my thesis.
In the Prototype section, I have provided some artwork for the prototype system architecture.
I have also uploaded a working Bibliography in the Writings section of the presentation site.
by Duane Cash
Thesis topic:
Mind-controlled systems for deeper cognitive connections with virtual assistants
The technology of virtual assistants has become more popular in recent years with the advent of smartphone voice recognition technologies such as Siri on the Apple iPhone and Google Now on Android-based devices. These virtual assistants primarily use voice for input into the system and a speech synthesis controller for audio feedback to the user. Although the reliability of this technology is improving over time, there are situations that may require the necessity to use an alternate form of input when voice is either not available or the environment is such that a voice recognition system would fail.
Another emerging is that of brainwave-reading EEG headsets. The Emotiv EPOC headset and Neurosky’s MindWave Mobile are examples of EEG headsets which have become available to consumers, developers and researchers for projects related to using the power of the mind to control smartphones and computers in a variety of ways. Currently, the technology has been used in a limited fashion to control helicopters, run phone applications and a play games. However, the use of brainwave analyzers with virtual assistants such as Siri has not been successful before now. Although projects have surfaced in recent years claiming to have the ability, such as the Black Mirror Project in 2011, this project and others have been eventually found to be fakes in their implementation.
Also the ability to use Siri in a third-party iPhone application has not been successful because Apple has not released an API to developers to make use of the voice recognition functions and speech recognition. The ability to use any API to control Siri on the iPhone has largely been undiscovered except for project that resorts to creating a ruby-enabled Siri proxy or the requirement to jailbreak the device. My custom API for using Siri in an iOS application does not require this type of activity.
This project contributes to the field by creating the first mind-controlled virtual assistant on a smartphone, and thus, making a virtual assistant that no longer depends upon voice as form of input. By providing a mind-controlled interface, users can communicate with their mobile device when voice is not an option. With multiple individuals using such as system, this technology becomes a form of pseudo-telepathy where individuals can communicate with one another with only mental commands. In these situations, the virtual assistant becomes a translator for the brainwave EEG patterns to speech from one mobile device to another. This idea is a breakthrough that will greatly change the way we interact with virtual assistants in the future as it becomes adopted in mainstream consumer use.
This document reveals some of the artistic and technology influences related my thesis topic of mind-controlled systems for human-to-machine interaction with voice assistants. Some of my influences originated from movies and others came from historical figures that left behind some of the seeds of inspiration for many of my interests in this subject matter.
(2001: Space Odyssey. Dir. Stanley Kubrick, 1968. Film.)
The Stanley Kubrick film 2001: A Space Odyssey influenced my fascination with talking machine and intelligent computer systems. The concept of interacting with a highly intelligent computer system such as HAL 9000 was appealing and led me to explore ways to program computer systems to converse in a similar manner by using voice recognition and speech synthesis technologies. As this technology became more readily available through PC and mobile SDKs, I sought ways to use the technology in inspiring ways both in my school projects and some of my research projects.
(Knight Rider. By Glen A. Larsen. NBC, 1982-86. Television.)
Having an interest in vehicles that use speech recognition, it was also natural for the television series “Knight Rider” to play an influential role in steering my interest in the field of virtual assistants and their use in real-world situations. The intelligent car called KITT influenced me to examine how such technology could be applied to our daily lives today. The conversational nature of the KITT car influenced me to keep the human factor in mind in my virtual assistant technology endeavors.
(Back To The Future. Dir. Robert Zemeckis, 1985. Film.)
The character Dr. Emmett Brown, while fictional, influenced my own perceptions of technology and how we might envision the design of future interfaces. While Dr. Brown’s ideas in the Robert Zemeckis film Back To The Future presented the character as “mad scientist” in certain ways, as time progressed and new technologies became feasible in the 21st century, I began to mentally revisit some of these ideas to find ways to make them possible today. The emerging technology brainwave-reading headsets such as the Emotiv EPOC and the Neurosky MindWave Mobile made the idea of harnessing brainwave pattern closer to reality.
(Neurosky MindWave Mobile EEG headset. Neurosky.com. April 2013. Web.)
(Nikola Tesla. Wikipedia.com. April 2013. Web.)
Nikola Tesla, along with his inventions, has always fascinated me ever since reading a biography I borrowed from the local library when I was in my youth. I was amazed at the number of inventions that Tesla had and how influential his inventions have been in today’s world in the field of electricity and radio communication. Now as an inventor myself, I have begun to understand the importance of getting one’s ideas quickly to patent office. Reading about Tesla’s inventions has made me realize that great inventions can emerge from a variety of circumstances and that one should be vigilant in recognizing the opportunity for inventions.
(Steve Jobs. Wikipedia.com. April 2013. Web.)
By far, the most influential part of my pursuit of innovative interfaces comes from the advent of the Apple iPhone development and the visions stemming from the character of Steve Jobs. After his death in 2011, I viewed a video interview that offered great advice for me as a designer, developer and innovator in all my works. In a 1994 interview at the Silicon Valley Historical Association, Steve Jobs offered the following profound advice:
“When you grow up, you tend to get told that the world is the way it is and your life is just to live your life inside the world, try not to bash into the walls too much, try to have a nice family, have fun, save a little money. That’s a very limited life. Life can be much broader, once you discover one simple fact, and that is that everything around you that you call life was made up by people that were no smarter than you. And you can change it, you can influence it, you can build your own things that other people can use. Once you learn that, you’ll never be the same again.
“The minute that you understand that you can poke life and actually something will, you know if you push in, something will pop out the other side, that you can change it, you can mold it. That’s maybe the most important thing. It’s to shake off this erroneous notion that life is there and you’re just gonna live in it, versus embrace it, change it, improve it, make your mark upon it.
“I think that’s very important and however you learn that, once you learn it, you’ll want to change life and make it better, cause it’s kind of messed up, in a lot of ways. Once you learn that, you’ll never be the same again.”
These inspiring words made a drastic impact on my thinking ever since the moment I heard them. Since then, my projects have aimed to reach higher and have largely been successful in improving the interactive experience and finding innovation from everyday sources. My current project on mind-controlled virtual assistants is the first step towards other interactive innovations I have in mind to imprint as my own mark upon the world.
Below is a quick descriptive list of some items I have uploaded into a “Past Work” section of my M.F.A. Thesis Presentation website at http://www.duanecash-unleashed.com/past-work.html
You will find examples and video at the link above.
Item #1:
Duane Cash Portfolio Website
Link: http://www.duanecash.com
Tools: Dreamweaver and jQuery
This website was created as part of the Interactive Design Game Development Portfolio Class. The goal was to create an interactive portfolio as a website that exhibited our skills, resume, and examples. Since my skill set comprised iOS development, the portfolio site was created using visual and interactive elements related to this type of work.
Created for ITGM 749 – Interactive Design Game Development Portfolio
Item #2:
“Pennies From Heaven” – ActionScript game
Link to Flash Game: Pennies From Heaven
Tools: Flash CS5 Actionscript
This project was completed to demonstrate the ability to outline, prototype, and finalize a small-scale game that showed mastery of customs functions, event handling, collision detection, and custom functions. The outcome was a catcher-style game in which the player catches coins falling from the sky in a metal cup in a quest for the highest score.
Created for ITGM 719 – Scripting for Interactivity
Item #3:
“Max Raven” - 3D Character Development
Tools: Maya 2010
This project involved the development of a 3D model starting from the story for the game world and resulting in a 3D character model to be present in a 3D character turntable for a movie reel. Using Maya 2010 and Z-Brush, Max Raven was designed as a main character for a fictional world and story called Mechroids.
Created for ITGM 710 - Character Development
Item #4:
Mechroid Manufacturing Factory on the Moon of Jarmic
Environment and Level Design
Tools: Maya 2010
This project consisted of the development of 3D environment with 3D modeled structures and a fly-through animation. Developing the back-story from the Max Raven Character further, the goal of this 3D environment idea was to create the mining facility that was populated by the Mechroid robots would were plotting to start an intelligent robot attack of the planet Sorna and launch an inter-planetary war.
Created for ITGM 352 – Environment and Level Design
Item #5:
“Scorpious” – 3D Ship Model and Animation
Tools: Maya 2010
The Scorpious space ship was designed as part of a game scene animation that explores the creation of 3D models in a character-based environment that exhibits textures, materials and lighting effects. The Scorpious ship belongs to the fictional Max Raven character developed in conjunction with the Mechroid back-story. Model sheets and a final fly-through animation was the outcome of the project.
Created for ITGM 240 – Modeling Materials and Lighting
Item #6:
Fall of the Giants – UDK game level
Tools: UDK, Maya, Photoshop, RealDraw Pro
This project was Unreal Development Kit level that allowed players to explore the remnants of a battle between two colossal robots. The scenery was a grassy valley littered with the mechanical artifacts of the robots. The level features the development of terrains, skyDome, particle effects and weapon pickups.
Created for ITGM 721 – Environments for Games
This prototype video shows the mind-controlled virtual assistant using the MindWave Mobile EEG headset and a custom Siri API. The prototype uses no jailbroken devices or Ruby-enabled proxies to interact with Siri. The user thinks of the intended action and the activity is performed on the device.
This prototype shows how the user can use mental commands to interact with the virtual assistant, open a menu, select icon items, play videos and music, make phone calls and open a map.
Mind-Controlled Apple iPhone Siri Virtual Assistant Prototype 2 from March 20, 2013
–Duane Cash
The purpose of this update is to provide a rough timeline and outline some important thesis milestones for my thesis project. I have also provided the topic, statement, and abstract here before the timeline.
Mind-controlled systems for Human-to-Machine Interaction with Virtual Assistants
Current intelligent virtual assistants have the limitation of activation primarily through voice input. Implementing mind-controlled systems with smartphone applications provide an alternate method of human-to-machine interaction with voice assistants in use cases where speech input is not available.
Modern usage of virtual assistants such as the iPhone Siri have increased consumer interest in using speech recognition and text-to-speech to perform a variety of tasks on the smartphone. While this technology has generated appeal as an alternative approach to human-to-machine interaction, a growing number of individuals have found these speech recognition systems to be frustrating to use, especially systems that fully require the user to initiate commands through voice only. Users with speech challenges may find it especially difficult to benefit from these voice assistants. This thesis proposes that speech recognition systems can extend their usability reach by providing alternate navigation methods for initiation of actions and communication through mind-controlled brainwave systems. As a proof of concept, a prototype system will be created which allows users to control a limited number of actions by processing EEG brainwave signals from the user through a wearable sensor headset paired with the user’s smartphone. This system will allow users to use their thoughts to initiate actions on their phone without the use of hands or voice.
A mind-controlled virtual assistant that leverages brainwave mapping to mobile devices interfaces will be enable individuals to perform a number of basic functions which would normally require touch methods or voice to initiate. Opening new views, maps, playing a video and making a phone calls using brainwave commands will change the way users interact with virtual assistants by providing users with an alternative method of interaction. By harnessing the power of the mind to control smartphone devices, users will discover deeper cognitive integration with virtual assistants and benefit from the personal communication with these machines.
Mind Controlled iPhone Siri Prototype Test
As I mentioned in my previous blog, I performed some initial tests using the MindWave EEG headset to control Siri with my thoughts. The initial tests were successful in showing how the speech recognition from Siri can be coupled with a brainwave-analyzing device to perform some basic tasks.
Here’s the link to the video of the prototype in action:
This is a one of my initial tests using a brainwave-reading device to control some custom Siri functions on a iPhone. For the first speech recognition segment I wave my hand over the device to activate Siri. During the next portions, I use mental commands from the EEG signals to commands the iPhone to open a map, open a menu and close a menu with voice feedback from Siri.
Today I performed some initial tests using the Neurosky MindWave Mobile brainwave-reading device along with some of its accompanying SDK tools. My first tests with the device were successful. After the initial setup process, I found that the software development kit might provide the structure necessary to couple with the speech recognition application on the iPhone. The sample project provided compiled successfully and after a review of the API reference, this hardware seems to provide access to the raw data necessary to process into commands in later prototype phases. In comparison to the Emotiv EPOC headset, which seems to have issues even with the base sample projects, I found the Neurosky MindWave Mobile to be less complicated in both setup and execution the brainwave triggers.
Using the initial voice recognition prototype I created earlier, I decided to see if I could link the MindWave Mobile headset with my current iOS prototype app. The goal was to use my brainwave patterns to trigger two different tasks – opening and closing a menu screen. Implementing the MindWave libraries into my Siri voice application, I was able to a set up the project so that I could open the menu and close the menu screen on the iOS device using the brainwave patterns from the MindWave headset. With some additional adjustments to the code I was able to calibrate it so the triggering of the menu was more natural.
For now, the initial tests have been successful in coupling the mind-controlled hardware with the iOS app with the virtual assistant. Soon I will make a video showing more of this system.
