Comments on: melissa – a fun AIR app

By: paul

paul — Mon, 26 Oct 2009 19:53:00 +0000

Hi Dan
sorry for the slow reply. You are looking for phonetic lip sync. I know that Toon Boom has some support for this but as you say it doesn’t have very good support for AI files. If you could output Toon Boom files to .fla there would be a possible workflow. There was also another product – LipSync MX, but i don’t know if it is still supported or not.
As for the speech to text, the quality is very dependent on the source audio. Users are reporting up to 95% accuracy on some files while as low as 50% on others. These files can of course be edited and the benefit of speech to text is that it gives you the precise timings. However there is currently no workflow for converting the meta data to phonetics or in fact phonetic support within Flash. I agree that this would be very useful and I will send feedback to the team. Who knows!

By: Dan

Dan — Tue, 22 Sep 2009 22:51:43 +0000

Hi Paul,

Thanks for the feedback on the lip syncing. I have tried this out and it doesn’t give me the desired effect. It does do what it is suppose to, but i am now realising its not going to be what we want.

The company i work for are using quite realistic vector characters and would like to automatically make their face and lips sync to the speaking. They will need realistic mouth shapes for specific sounds. The character facial animation length will be quite long hence why we are trying to avoid keyframing each movement.

I know Toon Boom does this but have had trouble importing illustrator files into this program. The vector gets imported in a very simplistic way. Gradients and transparency don’t seem to come out the way they should, and shapes don’t always import exactly right either. Is there a way to get Toon Boom to import these files properly?

Have you heard of any other program that works well with illustrator and flash that can do this type of lip syncing or add ons to flash that could do this?

My last question is about the transcribing of audio in soundbooth. It doesn’t seem to be very accurate. If this was more accurate i was thinking of maybe using the xmp metafile that is generated and reading this in flash to make the lip sync work. Is this idea viable?

Regards,

Dan

By: paul

paul — Wed, 16 Sep 2009 14:41:17 +0000

Hey Dan, yes you can do this quite easily. If you check out the source file of the dancing dudes you can see how I am tracking the amplitude of a music track to change the properties of the pixel bender filter. You can use the same process to get the amplitude of a spoken audio track and get Melissa's mouth to go to the specific frame based on the amplitude. Do some tests with the track to see what range you are getting in the amplitude and adjust it to use all 10 mouth shapes. Hope this helps

By: Dan

Dan — Wed, 16 Sep 2009 00:13:29 +0000

Hey paul, just wondering if you have some code to animate mouth movement from a audio track rather then through the microphone. If you don’t do you have some advice on how to approach it.

Regards,

Dan

By: madblog » Blog Archive » Adobe Online Creative Festival presentation source files

Wed, 02 Jul 2008 01:39:51 +0000

[...] to control a native window and also add system menus to your AIR app. The other file I showed was Melissa that is already available with the source files from this blog and also shows an implementation of [...]

By: Life is not a race to be first finished » Blog Archive » Adobe Educational Leadership

Sat, 21 Jun 2008 12:18:20 +0000

[...] Worldwide Evangelist for Adobe! What a great job title! Just for fun he showed us his Adobe Air app Melissa that he made. Turn the sound down and Melissa sits on your desktop and lip-synchs with you when you [...]

By: Graeme Doman

Graeme Doman — Thu, 05 Jun 2008 00:18:52 +0000

Thankyou 4 Melissa