Neurosky MindWave Mobile Hardware Tests

Today I performed some initial tests using the Neurosky MindWave Mobile brainwave-reading device along with some of its accompanying SDK tools. My first tests with the device were successful. After the initial setup process, I found that the software development kit might provide the structure necessary to couple with the speech recognition application on the iPhone. The sample project provided compiled successfully and after a review of the API reference, this hardware seems to provide access to the raw data necessary to process into commands in later prototype phases. In comparison to the Emotiv EPOC headset, which seems to have issues even with the base sample projects, I found the Neurosky MindWave Mobile to be less complicated in both setup and execution the brainwave triggers.

Using the initial voice recognition prototype I created earlier, I decided to see if I could link the MindWave Mobile headset with my current iOS prototype app. The goal was to use my brainwave patterns to trigger two different tasks – opening and closing a menu screen. Implementing the MindWave libraries into my Siri voice application, I was able to a set up the project so that I could open the menu and close the menu screen on the iOS device using the brainwave patterns from the MindWave headset. With some additional adjustments to the code I was able to calibrate it so the triggering of the menu was more natural.

For now, the initial tests have been successful in coupling the mind-controlled hardware with the iOS app with the virtual assistant. Soon I will make a video showing more of this system.

Below, you will find my Thesis Paper Outline. This outline provides structure to the thesis paper writing process and allows me to categorize major areas of discussion and concerns in the project and prototype.

Thesis Paper Outline

Mind-Controlled Systems for Deeper Cognitive Connections with Virtual Assistants

1. Introduction
   1. Thesis Statement – Current intelligent virtual assistants have the limitation of activation primarily through voice input. Implementing mind-controlled systems with smartphone applications provide an alternate method of human-to-machine interaction with voice assistants in use cases where speech input is not available.
   2. Thesis Abstract – Modern usage of virtual assistants such as the iPhone Siri have increased consumer interest in using speech recognition and text-to-speech to perform a variety of tasks on the smartphone. While this technology has generated appeal as an alternative approach to human-to-machine interaction, a growing number of individuals have found these speech recognition systems to be frustrating to use, especially systems that fully require the user to initiate commands through voice only. Users with speech challenges may find it especially difficult to benefit from these voice assistants. This thesis proposes that speech recognition systems can extend their usability reach by providing alternate navigation methods for initiation of actions and communication through mind-controlled brainwave systems. As a proof of concept, a prototype system will be created which allows users to control a limited number of actions by processing EEG brainwave signals from the user through a wearable sensor headset paired with the user’s smartphone. This system will allow users to use their thoughts to initiate actions on their phone without the use of hands or voice. A mind-controlled virtual assistant that leverages brainwave mapping to mobile devices interfaces will be enable individuals to perform a number of basic functions which would normally require touch methods or voice to initiate. Opening new views, maps, playing a video and making a phone calls using brainwave commands will change the way users interact with virtual assistants by providing users with an alternative method of interaction. By harnessing the power of the mind to control smartphone devices, users will discover deeper cognitive integration with virtual assistants and benefit from the personal communication with these machines.
   3. Problem Statement – Even though mind-controlled systems require large equipment and may be problematic to setup. Current intelligent voice assistance have some issues picking up speech from individual who suffer from voice-related disabilities.
   4. Proposed Solution – By harnessing the power of the mind to control smartphone devices, users will discover deeper cognitive integration with virtual assistants and benefit from the personal communication with these machines. Since voice input may not be available in all circumstances, a mind-controlled system provides an alternate form of input. Current trends in the emerging technologies of brainwave analyzers and intelligent virtual assistants now allow developers to create mind-controlled software for mobile devices that will help in use cases where voice input is not available
   5. Project Overview – Plan for design of prototype
      1. Overall Process Plan
      2. Plan for Intelligent Virtual Assistant Prototype
      3. Plan for Mind-Controlled system Prototype
      4. Process for coupling Virtual Assistant and Mind-Controlled System together
2. Historical Context of Intelligent Virtual Assistants
   1. Influence of Intelligent Computers in Media and Games
      HAL 9000KITT from Knight RiderGLaDOS from Portal
   2. Current Intelligent Software Assistants and Projects
      1. Siri in iOS devices
      2. Google Now
      3. S Voice
      4. CALO Project
   3. How my project fits into the context and moves the technology forward
3. Historical Context of Current Mind-Controlled Systems
   1. Approaches in retrieving raw data through EEG brainwave analyzers
      1. Emotiv EPOC Research Edition
      2. Neurosky – MindSet and MindWave Mobile
      3. MyndPlay – BrainBand and Research Toolkit
      4. Interaxon – Muse
   2. Hardware Investigation Conclusions
   3. Use cases for mind-controlled solutions
4. Project Details for a Mind-Controlled Virtual Assistant
   1. Overview of Design Goals
   2. Transition details for replacing “Voice” with mental control
   3. System architecture for the mind-controlled system
   4. Prototype hardware and software decisions
      1. Overview of hardware and software investigation
      2. Discussion of software design choices
      3. Discussion of hardware implementation choices
   5. Emergence of a new intelligent agent model for the future
      1. The V.A.L. (Virtual Assistant Linguistic) assistant is “born” as a MCE (Mind-Controlled Entity)
      2. How an MCE will create deeper cognitive connections and interactions
      3. Benefits for those suffering from voice-related disabilities
      4. The importance of this research for future emerging technologies
5. Project Reception and User Tests
   1. User evaluation feedback results and comparison with voice only systems
   2. Important lessons learned from creating the system
   3. Possible improvements for future versions of the MCE (Mind-Controlled Entity)
6. Conclusion
   1. Review of thesis goals and the importance of the emerging technology of MCE (mind-controlled entities)
   2. Verification of how the prototype satisfied the thesis goals as a solution
   3. Areas of future growth for mind-controlled virtual assistants

Here’s a link to my vimeo hosted video of  my initial prototype for the speech recognition side of the project:

http://vimeo.com/60359236

I made this iPhone app in Xcode and the interactivity indicator using with my graphic design tools. Currently there is no public API available to developers for Siri, yet I found a way to activate it using public frameworks available within the standard iOS SDK. I have recently been amazing how Apple his managed to hide powerful capabilities almost out in the open. One just needs to pour over the header files closely for clues on how to activate features like Siri. Instead of activating Siri with the Home button (which has been difficult for me so far to activate programmatically), I am using the proximity sensor to activate the voice recognition by wave my hand over the front facing camera.

I have setup the hosting and domain for my thesis presentation site here:

http://www.duanecash-unleashed.com

The website (for now) shows the basic wireframe structure showing the main categories. Content will be filled in over the course of the next week.

Here’ s a screenshot from the first page to show the overall appearance. I feel that the design should be simple to provide easy navigation to the different areas.

Thesis Statement

Current intelligent virtual assistants have the limitation of activation primarily through voice input. Implementing mind-controlled systems with smartphone applications provide an alternate method of human-to-machine interaction with voice assistants in use cases where speech input is not available.

Thesis Abstract

Modern usage of virtual assistants such as the iPhone Siri have increased consumer interest in using speech recognition and text-to-speech to perform a variety of tasks on the smartphone. While this technology has generated appeal as an alternative approach to human-to-machine interaction, a growing number of individuals have found these speech recognition systems to be frustrating to use, especially systems that fully require the user to initiate commands through voice only. Users with speech challenges may find it especially difficult to benefit from these voice assistants. This thesis proposes that speech recognition systems can extend their usability reach by providing alternate navigation methods for initiation of actions and communication through mind-controlled brainwave systems. As a proof of concept, a prototype system will be created which allows users to control a limited number of actions by processing EEG brainwave signals from the user through a wearable sensor headset paired with the user’s smartphone. This system will allow users to use their thoughts to initiate actions on their phone without the use of hands or voice.

A mind-controlled virtual assistant that leverages brainwave mapping to mobile devices interfaces will be enable individuals to perform a number of basic functions which would normally require touch methods or voice to initiate. Opening new views, maps, playing a video and making a phone calls using brainwave commands will change the way users interact with virtual assistants by providing users with an alternative method of interaction. By harnessing the power of the mind to control smartphone devices, users will discover deeper cognitive integration with virtual assistants and benefit from the personal communication with these machines.

Siri Epoc Prototype (in Process)

This video demonstrates a prototype for the virtual assistant side of the thesis project. The brainwave controller will be implemented at a later date. For now, the user can activate the speech recognition on the iPhone with a hand over the front camera. Once the speech recognition is in active listening mode it will listen for speech from the user. Once the user speech, the voice assistant starts listening for silence to determine when to stop listening and process the speech. After processing the speech, the virtual assistant will display the recognized speech and response text along with the action.

Here’s the video below (in .flv format due to size limitations):

SiriEpocTest2

And below I have included some screen captures from the iPhone app:

Thesis Statement

Current intelligent virtual assistants have the limitation of activation primarily through voice input. Implementing mind-controlled systems with smartphone applications enhances the human-to-machine interaction with voice assistants in use cases where speech input is not available.

Thesis Abstract

Modern usage of virtual assistants such as the iPhone Siri have increased consumer interest in using speech recognition and text-to-speech to perform a variety of tasks on the smartphone. While this technology has generated appeal as an alternative approach to human-to-machine interaction, a growing number of individuals have found these speech recognition systems to be frustrating to use, especially systems that fully require the user to initiate commands through voice only. Users with speech challenges may find it especially difficult to benefit from these voice assistants. This thesis proposes that speech recognition systems can be enhanced by implementing methods for initiation of actions and communication through mind-controlled brainwave systems. As a proof of concept, a prototype system will be created which allows users to control a limited number of actions by processing EEG brainwave signals from the user through a wearable sensor headset paired with the user’s smartphone. This system will allow users to use their thoughts to initiate actions on their phone without the use of hands or voice.

A mind-controlled virtual assistant will be enable individuals to perform a number of basic functions on the device which would normally require touch methods or voice to perform. Opening new views, maps, playing a video and making a phone calls using brainwave commands will enhance the way users interact with virtual assistants by providing users with an alternative method of interaction. By harnessing the power of the mind to control smartphone devices, users will discover deeper cognitive integration with virtual assistants and possibly benefit from the personal communication with these machines.

Below is a revision to my thesis statement.  I found that it was necessary to describe the main issue with the present virtual assistant technology first.

Thesis Statement

Current intelligent virtual assistants have the limitation of activation primarily through voice input. Implementing mind-controlled systems with smartphone applications enhances the human-to-machine interaction with voice assistants in use cases where speech input is not available.

Thesis Topic

Mind-Controlled Systems for Human-to-Machine Interaction with Voice Assistants

Thesis Statement

Implementing mind-controlled systems with smartphone applications enhances the human-to-machine interaction with voice assistants.

Visual Component Prototype Description

In order to demonstrate the thesis statement for this project, I will create an interactive iOS application that is controlled by a brainwave controller (i.e., the Emotiv EPOC EEG, Neurosky MindWave, etc.) which will allow the user to send a limited set of mental commands to the iPhone.  The custom application will consist of Objective-C algorithms that will listen for input from brainwave controller, a custom NLU (natural language understanding) feature to process the brainwave information, and methods for executing the mental commands on the smartphone device.

The prototype will also allow the user to use voice commands by waving the hand over a sensor on the front of the device to activate the voice recognition to listen for commands without touching the iPhone. The inclusion of this feature will speed up the prototype development for the applications side for testing before the brainwave controller link to the device is established. The application will use Siri speech recognition and will speak with Siri voice responses based upon the processing of the limited NLU feature. The limitations established for the NLU feature will allow the user to execute a number of functions on the iPhone such as, but not limited to the list below:

• Open a new view or menu
• Activating a button action
• Opening the Maps app for an address
• Playing a video
• Making a phone call
• Move a virtual ball

Visual Component Action Plan

The full visual component for the prototype will consist of a mockup for the iPhone application and components, and the working prototype (video recorded). A schedule of major milestones for the visual component is listed below:

ITGM 755 – By the end of ITGM 755, a mockup and a working prototype for the iPhone Siri portion will be completed minus the connection with the brainwave controller. Investigation of the Emotiv Epoc brainwave controller/SDK and other hardware will take place during this time to verify the use of this particular hardware or if other hardware will be used. Some limited brain-to-machine interaction may take place to evaluate the system. The iPhone virtual assistant will be voice operated for this term to verify and test the base working system. The brainwave controller will be added in the next phase during the next term. I will also implement the visual design elements for the project using available graphic design software for the user interface.

ITGM 765/Thesis Review – During this stage the working iPhone application will be coupled with the communication from the brainwave controller and the mental command processing will be implemented. The mind-controlled application will be demonstrated to show the enhancements of mind-control versus voice control on the iPhone application. The findings will be video documented to show the working visual component.

Artistic Influences Explained

Overview

The purpose of this document is to reveal the source of artistic influence and inspiration from my own portfolio work and chosen discipline. This paper will examine my general discipline of work in the area of integration of innovative technologies into the iOS platform and how several outside artistic influences have established a foundation for future work and innovation in this area.  Having chosen to pursue the thesis topic of mind-controlled systems for human-to-machine interaction with voice assistants, the original artistic influence can be explained by looking at these sources in closer detail.

Voice Assistants and Talking Machines

While working as an iOS Developer with Toyota and Honda, the nature of the work has often lead to the exploration of new technologies to implement into future vehicles. One of these technologies relates to the use of voice recognition and virtual assistants in the vehicle. Before working at Honda R&D, the fascination of talking to devices was prevalent in my own personal projects and experiences with iOS development. Besides voice recognition in the vehicle, a number of my own projects explored the area human-to-machine interfaces. The idea of communicating with machines was influenced by artistic and conceptual influences from watching movies such as the 1968 Stanley Kubrick film 2001: A Space Odyssey. Specifically, the scenes relating to the interaction between HAL 9000 and David Bowman pushed my imagination to new levels (Kubrick, “2001: A Space Odyssey”).

(2001: Space Odyssey. Dir. Stanley Kubrick, 1968. Film.)

The deeply conversational manner in which HAL 9000 conversed so intelligently with humans led me to engage in projects that explored the area of voice recognition and text-to-speech. Other areas of artistic and design influence came from television shows.

Being interested in vehicles and speech recognition, one can also see how the popular 1982 television series “Knight Rider” played an influential role in defining my discipline of study and work. The intelligent car called KITT (Knight Industries Two Thousand) became an influential part of my ideas for the future vehicle (Larson , “Knight Rider”).

(Knight Rider. By Glen A. Larsen. NBC, 1982-86. Television.)

The concept of speaking to a vehicle in an intelligent manner, just as the character Michael Knight did in the television series was an intriguing idea that may have eventually steered me into the direction of using virtual assistants in more meaningful ways inside mobile and future vehicle applications.

In 2009, I created a voice recognition program in C# that used voice input from the user to interact with a virtual R2-D2 model on the PC. The program was able to understand commands to perform movements on the screen such as moving forward, back, and turning left and right. Moving into recent years, I have been developing iOS applications that use the Siri voice recognition to interact with the user or initiate interactions on the screen without the necessity of touching the screen. Additionally in the design of interfaces, such as the one found on my own portfolio website at www.duanecash.com, one can see how earlier experiences with the HAL 9000, KITT, and the iPhone’s Siri have influenced the patterning of its visual aesthetics.

(Screenshot from http://www.duanecash.com. January 2013. Web)

Mind-Controlled Machines

While voice recognition can be an excited area to explore, the area of using brainwave patterns to produce mind-controlled machines is emerging as a viable method of human-to-machine interaction. While still in its infancy, the field of mind-controlled applications is becoming more real and attainable with SDKs and tools currently available to consumers. My own work in iOS application and game development and the exploration of virtual assistants has naturally led me to consider the idea of using the mind as a method of control for virtual assistants. My development experiences and knowledge in the area have now led me to believe that some of the ideas found in some popular movies, such as Back To The Future, can now be attainable with the right tools and technical implementations (Zemeckis, “Back To The Future”).

(Back To The Future. Dir. Robert Zemeckis, 1985. Film.)

Even seeing the character Dr. Emmett Brown wearing this giant headgear influenced my own perceptions of technology and visions for future interactive interfaces. In 1985, the notion of reading one’s mind in any manner was largely science fiction. Now over 25 years later, with the technology of brainwave recognition and analysis emerging into consumer products such as the Emotiv Epoc, the foundation necessary for using brainwave patterns to control other devices has arrived.

(Image of Emotiv Epoc headset. Emotiv.com. January 2013. Web.)

As my own work on a mind-controlled virtual assistant progresses, one can see how movies like Back To The Future and 2001: A Space Odyssey became an influential part of the design process. Each of these influences has shaped my perceptions of what a future mind-controlled virtual assistant could include and how best to implement some of the features.

By far, the most influential part of my pursuit of innovative interfaces comes from the advent of the Apple iPhone development and the visions stemming from the character of Steve Jobs. After his death in 2011, I viewed a video interview that offered great advice for me as a designer, developer and innovator in all my works. In a 1994 interview at the Silicon Valley Historical Association, Steve Jobs offered the following profound advice:

“When you grow up, you tend to get told that the world is the way it is and your life is just to live your life inside the world, try not to bash into the walls too much, try to have a nice family, have fun, save a little money. That’s a very limited life. Life can be much broader, once you discover one simple fact, and that is that everything around you that you call life was made up by people that were no smarter than you. And you can change it, you can influence it, you can build your own things that other people can use. Once you learn that, you’ll never be the same again.

“The minute that you understand that you can poke life and actually something will, you know if you push in, something will pop out the other side, that you can change it, you can mold it. That’s maybe the most important thing. It’s to shake off this erroneous notion that life is there and you’re just gonna live in it, versus embrace it, change it, improve it, make your mark upon it.

“I think that’s very important and however you learn that, once you learn it, you’ll want to change life and make it better, cause it’s kind of messed up, in a lot of ways. Once you learn that, you’ll never be the same again.” 

These inspiring words made a drastic impact on my thinking ever since the moment I heard them. Since then, my projects have aimed to reach higher and have largely been successful in improving the interactive experience and finding innovation from everyday sources. My current project on mind-controlled virtual assistants is the first step towards other interactive innovations I have in mind to imprint as my own mark upon the world.

Works Cited

Cash, Duane. duanecash.com. January 2013. Web.

Emotiv Epoc headset. emotiv.com. January 2013. Web.)

Jobs, Steve. Interview by the Silicon Valley Historical Association.  “Visionary Entrepreneur.” 1994. DVD.

2001: Space Odyssey. Dir. Stanley Kubrick. MGM, 1968. Film.

Knight Rider.  By Glen A. Larson. NBC, 1982-86. Television.

Back To The Future. Dir. Robert Zemeckis. Universal Studios, 1985. Film.