melissa – a fun AIR app
A few weeks ago we had a webjam at Adobe Pacific and I built this AIR app in Flash for my demo. It is totally useless but it is a bit of fun.
It uses your microphone input to animate melissa. There are also a range of other animations triggered by keyboard commands.
Instructions:
1) Download, install and open the melissa.air app.
2) choose your mic input from the popup (best to mute your speakers)
3) hit ‘i’ for in to show Melissa
4) talk into the mic and adjust the level with the slider to get a good effect
5) hit ‘w’ for wink
6) hit ‘b’ or blink
7) hit ‘n’ for nose
8) hit ‘h’ for hand
9) hit ‘o’ for out to hide Melissa
If you would like the source files, they are here.
Have fun! (BTW I didn’t win the webjam!)
June 5th, 2008 at 10:18 am
Thankyou 4 Melissa
June 21st, 2008 at 10:18 pm
[...] Worldwide Evangelist for Adobe! What a great job title! Just for fun he showed us his Adobe Air app Melissa that he made. Turn the sound down and Melissa sits on your desktop and lip-synchs with you when you [...]
July 2nd, 2008 at 11:39 am
[...] to control a native window and also add system menus to your AIR app. The other file I showed was Melissa that is already available with the source files from this blog and also shows an implementation of [...]
September 16th, 2009 at 10:13 am
Hey paul, just wondering if you have some code to animate mouth movement from a audio track rather then through the microphone. If you don’t do you have some advice on how to approach it.
Regards,
Dan
September 17th, 2009 at 12:41 am
Hey Dan, yes you can do this quite easily. If you check out the source file of the dancing dudes you can see how I am tracking the amplitude of a music track to change the properties of the pixel bender filter. You can use the same process to get the amplitude of a spoken audio track and get Melissa’s mouth to go to the specific frame based on the amplitude. Do some tests with the track to see what range you are getting in the amplitude and adjust it to use all 10 mouth shapes. Hope this helps
September 23rd, 2009 at 8:51 am
Hi Paul,
Thanks for the feedback on the lip syncing. I have tried this out and it doesn’t give me the desired effect. It does do what it is suppose to, but i am now realising its not going to be what we want.
The company i work for are using quite realistic vector characters and would like to automatically make their face and lips sync to the speaking. They will need realistic mouth shapes for specific sounds. The character facial animation length will be quite long hence why we are trying to avoid keyframing each movement.
I know Toon Boom does this but have had trouble importing illustrator files into this program. The vector gets imported in a very simplistic way. Gradients and transparency don’t seem to come out the way they should, and shapes don’t always import exactly right either. Is there a way to get Toon Boom to import these files properly?
Have you heard of any other program that works well with illustrator and flash that can do this type of lip syncing or add ons to flash that could do this?
My last question is about the transcribing of audio in soundbooth. It doesn’t seem to be very accurate. If this was more accurate i was thinking of maybe using the xmp metafile that is generated and reading this in flash to make the lip sync work. Is this idea viable?
Regards,
Dan
October 27th, 2009 at 5:53 am
Hi Dan
sorry for the slow reply. You are looking for phonetic lip sync. I know that Toon Boom has some support for this but as you say it doesn’t have very good support for AI files. If you could output Toon Boom files to .fla there would be a possible workflow. There was also another product – LipSync MX, but i don’t know if it is still supported or not.
As for the speech to text, the quality is very dependent on the source audio. Users are reporting up to 95% accuracy on some files while as low as 50% on others. These files can of course be edited and the benefit of speech to text is that it gives you the precise timings. However there is currently no workflow for converting the meta data to phonetics or in fact phonetic support within Flash. I agree that this would be very useful and I will send feedback to the team. Who knows!