The current setup gets in the mid 80%'s on easy and can beat medium songs, but there are a few things that could be improved:
1) Webcam - the webcam used here is a Dlink camera from 1999. A newer camera would have vastly better image quality and hopefully a higher frame rate. I recently got a D90, and I think with some work I could use this as the camera instead of my crappy old webcam.
2) Code - The current code could probably be sped up by using a different method to get the webcam image. Currently it uses some code I found online and improved upon that uses WIA drivers. I'm also getting frame rate dropouts, which is why it is missing notes. It hits 100% of the notes that it see's, but misses notes when it gets no new frames.
3) Relays - I built this using 15A 12V relays. They are an order of magnitude slower than a reed relay which would have been much easier to use, but I wanted to be able to use this setup to run as a light controller in the future and a reed relay can't handle that kind of juice. Stepping up to reed relays or even some sort of IC would speed up response time a lot.
Am I going to try to improve upon the current setup? Probably not, but maybe if I was really bored. I did this to prove it could be done with a webcam, now I know it can.
Wednesday, April 29, 2009
Wednesday, January 28, 2009
You ROCK! - Beat GHIII on easy
After letting it run for a few hours, it has beat the game on EASY. I have to start each song and beat the battle modes, but other than that, it completed every song.
I was also able to beat the first song on Medium, and Should be able to let it run to beat most of them. Might have to do a little work to speed up the video to defeat every song.
Here's a video:
http://www.youtube.com/watch?v=ItqbqsQAonc
I was also able to beat the first song on Medium, and Should be able to let it run to beat most of them. Might have to do a little work to speed up the video to defeat every song.
Here's a video:
http://www.youtube.com/watch?v=ItqbqsQAonc
It works!
A lot has happened since my last post.
I got an Arduino, and a relay board.
The arduino is overkill, i could have gotten a parallel port driven relay board and been done with it, but I wanted to be able to use it for other things in the future. Same with the relay board. It's complete overkill. Relay's can handle 120V at 15amp, I only need something like 3V and a few mili-amps. I might want to use this for other things in the future though.
So, here's what I've got:
Custom C# code to analyze pictures from the webcam, and send the notes out over a Serial Connection.
Arduino, programmed to read from the Serial port, and output that to the relay board, turning on and off the relays.
Relay board wired up to the buttons in the Guitar Hero controller.
While testing it out on the super easy mode, I noticed that the code is faster than the screen, so it would triple play each note. I made some adjustments so that it will ignore a note if it's the same as the last note. This seems to work so far on easy mode, since the notes come so slow.
For the past 2 days, I've been letting it run on easy mode. It's awesome! It seems to do about 3 stars on every song, at around 80% accuracy!
The reason it's not at 100% is due to the webcam. I'm not sure if it's my code, or the webcam, but The framerate drops very low every couple of seconds. This may be my code slowing down, possibly some garbage collection or bad drivers. I don't know yet, but when it happens, it's very likely that it'll miss a note.
Since I changed the code to ignore notes that are the same as the last note, it's become an expert at holding notes. It won't let go of a note until it sees a new one. It's also fairly good at playing double notes.
It just got 75%, 16 note streak, 3 stars on Cherub Rock.
Right now, it's unable to beat the Guitar Battle challenges because of how the camera is setup. The screen moves when you play battle mode, and I haven't adjusted the position of the camera so it can see the notes. Also, since there is no attack, I'd have to manually tell it to send the attack. I beat them myself and left it to play all the songs in between. I should have all of the easy mode finished soon.
More pictures coming on how the relay's are wired up, and hopefully a YouTube video of the action.
I got an Arduino, and a relay board.
The arduino is overkill, i could have gotten a parallel port driven relay board and been done with it, but I wanted to be able to use it for other things in the future. Same with the relay board. It's complete overkill. Relay's can handle 120V at 15amp, I only need something like 3V and a few mili-amps. I might want to use this for other things in the future though.
So, here's what I've got:
Custom C# code to analyze pictures from the webcam, and send the notes out over a Serial Connection.
Arduino, programmed to read from the Serial port, and output that to the relay board, turning on and off the relays.
Relay board wired up to the buttons in the Guitar Hero controller.
While testing it out on the super easy mode, I noticed that the code is faster than the screen, so it would triple play each note. I made some adjustments so that it will ignore a note if it's the same as the last note. This seems to work so far on easy mode, since the notes come so slow.
For the past 2 days, I've been letting it run on easy mode. It's awesome! It seems to do about 3 stars on every song, at around 80% accuracy!
The reason it's not at 100% is due to the webcam. I'm not sure if it's my code, or the webcam, but The framerate drops very low every couple of seconds. This may be my code slowing down, possibly some garbage collection or bad drivers. I don't know yet, but when it happens, it's very likely that it'll miss a note.
Since I changed the code to ignore notes that are the same as the last note, it's become an expert at holding notes. It won't let go of a note until it sees a new one. It's also fairly good at playing double notes.
It just got 75%, 16 note streak, 3 stars on Cherub Rock.
Right now, it's unable to beat the Guitar Battle challenges because of how the camera is setup. The screen moves when you play battle mode, and I haven't adjusted the position of the camera so it can see the notes. Also, since there is no attack, I'd have to manually tell it to send the attack. I beat them myself and left it to play all the songs in between. I should have all of the easy mode finished soon.
More pictures coming on how the relay's are wired up, and hopefully a YouTube video of the action.
Friday, January 2, 2009
Welcome
A while back I was playing Rock Band, and decided to see if i could use a webcam to read the screen and call out the notes. I got it to work, and put a video of it up on YouTube.
http://www.youtube.com/watch?v=9jz_2tCt_qg&feature=channel_page
and
http://www.youtube.com/watch?v=GGFbVxJlue4&feature=channel_page
FIGURING OUT WHAT NOTES TO PLAY:
Using an OLD(circa 1999) webcam that I had in a drawer, and some custom C# code, i was able to analyze each picture, and watch for the notes that need to be played. It's not that complicated, it simply looks at a pre-defined area of the screen and watches it to get 'bright'. When it sees the certain area of the screen get bright, it plays a wav file of me saying the color. This was useful when I wanted to watch the screen and hear the note it thought it should play.
Since I put together the note-reader part of the system, I've seen a bunch of people put together similar systems. I haven't seen any use a webcam though. Many use direct video input, which gives you a perfect signal, but I don't have a video input card. I've also seen people attach light sensors to the screen, but I don't want to do anything like that. I'm using a D-Link DSB-C300 webcam that I got in the fall of 1999. It's Old. Focus is pretty much impossible, and the frame rate isn't all that great. It's good enough to get up to about the HARD level at 320x240 resolution. Once I complete the system, and get all the bugs worked out, I'll look into replacing it if it becomes the limiting factor.
PLAYING THE NOTES BACK TO THE GAME
The next step was to get the notes to play back into the game. The easiest way is probably to use a Parallel port to drive some relays. I found a few parallel port driven relay's on ebay, but who's got a computer with a parallel port anymore? I decided to go with an Arduino. It's something that's programmable, and has a LOT more power than a simple USB or parallel port driven relay board. Once this project is done, I can use it to do something else.
This Christmas season, I saw a good deal on an XBOX 360 GHIII controller, so I decided to resurrect this project. I also purchased an Arduino Duemilanove, the latest version of the Arduino board. So far, very cool.
The Arduino connects to your PC with a USB cable and appears as a regular serial COM Port. Using System.IO.Ports in C#, I'm able to send commands to the Arduino. The Arduino reads the serial commands and then 'strums' a note.
The second part of this of course, is actually sending a command to the game. I opened up the GHIII controller and soldered a wire to each of the inputs of the buttons. I'm and idiot and forgot to take pictures, but I may have to open it up again, so expect pictures later. For now, check out:
http://egyokeo.com/gh2mod/
.
I used a GHIII controller, which, for our purposes is the same. I didn't cut the wire though, I soldered into where the button board connects to the wire. It's small, and I've never really done much soldering before(another fun part of this project :) ), but I made it work.
It's a little strange how the wires work. There's 8 wires for 5 buttons. It seems like you could use 6 wires to get the work done, but that's not how it works. It looks like there is either .3V across the button, or .6, depending on the button. I've come from a Computer Science background, not an electrical engineering one so I don't really understand why it is done this way, but since I plan to just use a relay to connect the wires, it should be OK.
I'm currently waiting for a 8 channel relay board from ebay to show up at my door. Right now, I can read the screen and light up an LED for each button to play and one more LED for the Strum bar. Once I get the relay, I'll wire it up, and trip the relay instead of an LED. I am using relay's that are powerful enough to drive a 15amp 120V signal. They are TOTALLY overkill, but I want to be able to use them for something else in the future, like driving Christmas lights or something like that. Their response time is around 10ms max, which may become a problem on the faster notes. I've started looking into reed relay's, which are around 1ms, or using some sort of MOSFET or something else, that are around 1ns response time. I'll cross that bridge when I come to it.
Hopefully it'll all work, but I foresee a lot of debugging and getting the timing right. It should be interesting, if at least to boost up my XBOX Live scorecard :P.
http://www.youtube.com/watch?v=9jz_2tCt_qg&feature=channel_page
and
http://www.youtube.com/watch?v=GGFbVxJlue4&feature=channel_page
FIGURING OUT WHAT NOTES TO PLAY:
Using an OLD(circa 1999) webcam that I had in a drawer, and some custom C# code, i was able to analyze each picture, and watch for the notes that need to be played. It's not that complicated, it simply looks at a pre-defined area of the screen and watches it to get 'bright'. When it sees the certain area of the screen get bright, it plays a wav file of me saying the color. This was useful when I wanted to watch the screen and hear the note it thought it should play.
Since I put together the note-reader part of the system, I've seen a bunch of people put together similar systems. I haven't seen any use a webcam though. Many use direct video input, which gives you a perfect signal, but I don't have a video input card. I've also seen people attach light sensors to the screen, but I don't want to do anything like that. I'm using a D-Link DSB-C300 webcam that I got in the fall of 1999. It's Old. Focus is pretty much impossible, and the frame rate isn't all that great. It's good enough to get up to about the HARD level at 320x240 resolution. Once I complete the system, and get all the bugs worked out, I'll look into replacing it if it becomes the limiting factor.
PLAYING THE NOTES BACK TO THE GAME
The next step was to get the notes to play back into the game. The easiest way is probably to use a Parallel port to drive some relays. I found a few parallel port driven relay's on ebay, but who's got a computer with a parallel port anymore? I decided to go with an Arduino. It's something that's programmable, and has a LOT more power than a simple USB or parallel port driven relay board. Once this project is done, I can use it to do something else.
This Christmas season, I saw a good deal on an XBOX 360 GHIII controller, so I decided to resurrect this project. I also purchased an Arduino Duemilanove, the latest version of the Arduino board. So far, very cool.
The Arduino connects to your PC with a USB cable and appears as a regular serial COM Port. Using System.IO.Ports in C#, I'm able to send commands to the Arduino. The Arduino reads the serial commands and then 'strums' a note.
The second part of this of course, is actually sending a command to the game. I opened up the GHIII controller and soldered a wire to each of the inputs of the buttons. I'm and idiot and forgot to take pictures, but I may have to open it up again, so expect pictures later. For now, check out:
http://egyokeo.com/gh2mod/
.
I used a GHIII controller, which, for our purposes is the same. I didn't cut the wire though, I soldered into where the button board connects to the wire. It's small, and I've never really done much soldering before(another fun part of this project :) ), but I made it work.
It's a little strange how the wires work. There's 8 wires for 5 buttons. It seems like you could use 6 wires to get the work done, but that's not how it works. It looks like there is either .3V across the button, or .6, depending on the button. I've come from a Computer Science background, not an electrical engineering one so I don't really understand why it is done this way, but since I plan to just use a relay to connect the wires, it should be OK.
I'm currently waiting for a 8 channel relay board from ebay to show up at my door. Right now, I can read the screen and light up an LED for each button to play and one more LED for the Strum bar. Once I get the relay, I'll wire it up, and trip the relay instead of an LED. I am using relay's that are powerful enough to drive a 15amp 120V signal. They are TOTALLY overkill, but I want to be able to use them for something else in the future, like driving Christmas lights or something like that. Their response time is around 10ms max, which may become a problem on the faster notes. I've started looking into reed relay's, which are around 1ms, or using some sort of MOSFET or something else, that are around 1ns response time. I'll cross that bridge when I come to it.
Hopefully it'll all work, but I foresee a lot of debugging and getting the timing right. It should be interesting, if at least to boost up my XBOX Live scorecard :P.
Subscribe to:
Posts (Atom)