Neural Network Ratatouille

by polegar
Overall★★★★★★★★★★3.56
Fun★★★★★★★★★★3.25
Innovation★★★★★★★★★★4.57
Theme★★★★★★★★★★3.57
Graphics★★★★★★★★★★2.69
Humor★★★★★★★★★★2.95
Mood★★★★★★★★★★2.81
Incompatible Genres ? Hmmm is it combatible to have a rat in a kitchen or to control your game with your webcam ?

For each ludum dare in which I participate, I try to learn something new. This time it's https://js.tensorflow.org. Because it's incredible to be able to do deep learning programming in JS !

I've used this tutorial as a starting point : https://js.tensorflow.org/tutorials/webcam-transfer-learning.html and then develop the game with pure JS and HTML5.

Surprinsingly, I've had more trouble with my rusty JS than with tensorflow ... I probably shoud have use a lib like Phaser Js instead of writing everything on my own.

/!\ I just added a post comp version with some minor bux fixes (you can now restart by pressing R and you can no longer launch the game without having trained your neural network), I suggest you to play this version for a better experience /!\

HERE'S HOW YOU PLAY THE GAME (with your webcam) :

1- For each control you need to provide some pictures of you holding an object(a fruit or a vegetable for example :)) in a specific direction. For IDLE I suggest you to hide the object your holding. (hint : you can hold clic to capture a lot of picture fast)

2- Then you have to clic on "Train ratatouille" and wait a little for the number to go down.

3- After that clic on "Let's cook" and be ready to play !

4- Show the object on the direction you want to move the character. You need to catch the cheese and avoid the mouse trap !

screenshot
Tools: Phaser

Comments31

nander24 Apr 2018, 12:59
It actually works!! The game itself isn't that special, but the technique actually works, nice!
Faloon24 Apr 2018, 13:32
Yes, true. Astonishing.
Ananace
Ananace24 Apr 2018, 15:24
Well, that actually worked much better than I expected it to, though the gameplay is quite simplistic.

Can't say I've played many games with webcam controls.
legavroche
legavroche24 Apr 2018, 15:31
This webcam control game is fun and innovative! The technicality of this game is astonishing.
deep neural network programming(for machine learning?) + webcam + tensor flow and building the game in Javascript is a plus plus plus!
microwerx
microwerx24 Apr 2018, 15:36
That was pretty cool. I thought it was innovative to use your webcam for training a neural net.
incobalt
incobalt24 Apr 2018, 22:52
It took me far too long to realize I had to click the train button even though it was written right here >.
aplite24 Apr 2018, 22:56
What a unique mechanic! Really impressive and it works well given the limited time constraint. The movement was hard to get precise so avoiding the traps was difficult at times.

To help players get used to the mechanic you could start them off in a room with just a piece of cheese, and then move them into the room with traps and cheese. I killed myself a couple of times at the very start and had to redo the setup process each time.
polegar
polegarauthor25 Apr 2018, 09:24
Thank you everyone for the kind comments :) . @incobalt @aplite I just posted a post comp version with some fixes for the issues you encountered (add a restart with R, and the button "let's cook" is no longer visible if you don't train the model)
Orteil
Orteil25 Apr 2018, 11:13
oh dang, that actually works great ! good stuff
GastricSparrow
GastricSparrow25 Apr 2018, 13:18
Wow, as a machine learning novice I love what you've done here. I've been looking into how to incorporate machine learning into games, and here you are making this on a whim! Great innovation happening, love it.
jfraser314
jfraser31425 Apr 2018, 20:19
Difficult but hilarious. Had my class playing this for a while, here is one of my students trying (made sure to use one that wasn't too embarrassing). Thanks for the great idea (though it made me dizzy)
sndr
sndr25 Apr 2018, 23:30
Whoa, this is impressive! Unexpected entry with a lot of creativity, both on the technical side and the design one. Controls worked surprisingly smooth (I was expecting it to be a bit more fiddly), it creates a really cool challenge, and the theme is most certainly super cute. :) Loved this! Really good job!
candlesan26 Apr 2018, 04:49
Well that was really different! How fun. Thanks for giving this piece of tech a try and I also applaud your desire to learn something new every jam - I think that's a great idea!!!
DigNZ
DigNZ26 Apr 2018, 07:39
What an amazing idea! It didn't quite work for me but was close enough to see that it should be possible. I probably should've chosen more different images for the actions. Impressive and innovative technical effort and a simple little game.
Weston Walker26 Apr 2018, 15:26
Extremely innovative idea here. Props to you for breaking out of the mold. Everything seemed to work as it should. Great job!
Team Luminous
Team Luminous26 Apr 2018, 17:24
What an exciting idea! It worked — tho holding the object both left and right ended up being hard. :D I'd love a way to remove samples, too, as I added bad ones by accident.

-Rao
John Darrington
John Darrington27 Apr 2018, 02:14
Holy sh. This was amazing - and one of my favorite ideas here this year. I was impressed with the hard work in finding a good mechanic.
sebbernery
sebbernery27 Apr 2018, 11:25
Awesome, that's a very good usage of NN in JS. And it works perfectly (for me) and gives very fun game moments (I played with my head).
designernap
designernap27 Apr 2018, 16:30
Lovely bit of tech and game here! wonderful work!
morrilet
morrilet29 Apr 2018, 21:52
The tech behind this was amazing, but the game itself was lacking. Stick this on a more interesting game or a series of minigames and it could be pretty cool. Full points for innovation, though, as this is a really interesting concept. Great work!
bloodyaugust
bloodyaugust30 Apr 2018, 07:22
Tried to play this on stream, but alas, cannot use OBS and have the webcam free at the same time. :frowning2: I'll be back to play off-stream and give further feedback.
polegar
polegarauthor30 Apr 2018, 11:20
@bloodyaugust oh that's a pity it could have been fun on stream, it's good to know though.
Paul Avallone30 Apr 2018, 22:49
This is really cool. I would love to see the game expanded to operating in a kitchen like in the movie. I am impressed how well the controls worked. Great job.
JackLeHamster
JackLeHamster1 May 2018, 15:14
Le concept est pas mal. Faudrait pousser l'idée pour avoir un gameplay plus fun, mais sinon l'idée est originale.

Voilà ma video sur le jeu (parmis d'autres):

polegar
polegarauthor2 May 2018, 12:04
@JackLeHamster Hehe je viens de voir la vidéo c'est cool que tu y ai joué :). Et oui effectivement le jeu est plutôt une expérience technologique qu'autre chose. C'était dur en 48h d'arriver à faire un gameplay sympa tout en apprenant à utiliser tensorflow JS pour détecter les mouvements par webcam.
Par contre le jeu semble assez lent sur la vidéo c'est étonnant, tu as quoi comme CPU (ca doit être lié au fait que tu streamais en meme temps) ?
JackLeHamster
JackLeHamster2 May 2018, 14:42
Ouais, tout est un peu plus lent que j'enregistre. C'était une bonne idée. Je pense peut être tu peux utiliser ta technologie pour faire un jeu de course, tu te mets à la place du cockpit, et suivant que tu te penches à droite ou à gauche, ça bouge la voiture dans cette direction. Et tu voix ta reflection dans le parbrise ou queq chose comme ça.
GastricSparrow
GastricSparrow19 May 2018, 04:57
@polegar Congrats! I barely got beat out by you in innovation, shouldn't have rated you a 5... just kidding, you deserve it!

Other games by @polegar

Found a bug?
Tell us on Discord