Spine Pro Vtuber Working Prototype

SilverStraw

I spent over a week ( working on this after my full-time work ) researching and developing a working Vtuber web solution. It's time for Spine Pro to level the playing field with Live2D in the Vtube realm. I expect to upload this application after I have enough features.

Edit ( 7-4-2022 ) Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released.

https://silverstraw.itch.io/spine-vtuber-prototype

I also include the model that I have been using for testing.

https://silverstraw.itch.io/spine-vtube-test-model

Nate

Very cool! We have plans to do this officially. It's still early days, but it's super cool!

SilverStraw

I beat you to the punch, Nate. :grinteeth:

Nate wrote
Very cool! We have plans to do this officially. It's still early days, but it's super cool!

Official support would be awesome! :heart:
In the meantime, I will spearhead the frontier. I bet your plate is full with Spine's future development. Maybe we can have some discussions in the future. 😉

Nate

You did! :punch:

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!

Erika

Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<

Mario

Super cool! From the debug render on the left, it seems we are using the same tensorflow light model 🙂

SilverStraw

Nate wrote
You did! :punch:

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!

Yay! Physics!
It's been slow for me as well. Now, I am trying to figure out how to allow user to load their Spine model that would work with spine-ts runtime.

Erika wrote
Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<

Hype! :fiesta:
I can't wait to create Spine models for streaming. Live2D Cubism has a completely different approach to model rigging.

Mario wrote
Super cool! From the debug render on the left, it seems we are using the same tensorflow light model 🙂

Are we really using the same? :bigeyed:
Now, tell me more. :smirk:

UPDATE:
I finally got the web browser file reader to work properly with Spine WebGL runtime. My file browser isn't shown in this video but I am clicking the asset files to be read and loaded into Spine runtime. This is an important step towards Vtubing with Spine rigs.

Nate

Cool! It can also be done via drag and drop. Working with browsers is such pain!

SilverStraw

Nate wrote
Cool! It can also be done via drag and drop. Working with browsers is such pain!

Nate is telling me something.
Okay, I found the example of the drag and drop from Spine Player generator example.

I am updating on my progress with my application. The last couple months has been hectic with work and my other project failures.

Mario

Neat!

Misaki

Great, I’m so excited! 😃

SilverStraw

Hello Mario and Misaki.

This is another step closer. I tried several approaches to operate the mouth on the model.

Misaki

OMG, it looks very easy to use! The mouth behavior looks natural 😃 Although the character has no eyelid yet, just being able to open and close the mouth conveys face expression quite well.

SilverStraw

The character is evolving. I will get to the eyelids . . . eventually.

This should be ready for the next phase of testing.

Misaki

Great evolving! 😃 :yes:

SilverStraw

I used all my rare candies, Misaki. 😉

Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released. I also include the model that I have been using for testing.

https://silverstraw.itch.io/

Misaki

Congrats on the release! 😃
I tried with a simple character which was created based on your Spine project file and it looks good! :heart:
Here's what I tried:

Sorry about the lack of blink animations since I drew and rigged it in a hurry and about the closed mouth not being clean enough! I will make various modifications later.

It may depend on the environment to do this, but it seems that eye pupil movement tends to be a bit jumpy. However, the response for facial tilt, direction, and mouth movements are very smooth! Also, the setup was easy to understand, so creating my own project was easy for me. It is a lot of fun to try out the movements with this tool 🙂

SilverStraw

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.

Another factor is the small size of the eye pupils relative to the face. They were twitching a lot so I had to smooth the incoming signal.That solution isn't perfect. It's hard to distinguish jitter from eye pupil movements. In the future I will allow users to adjust these incoming signals.

Misaki

I have added some modifications to my skeleton:

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.

Unfortunately, I tried it without eye glasses last time and this time, but it seems the eye pupil tracking response is not very good :tear:

Currently, the right eye and the left eye can register separate animations, but since it often happens that only one eye moves and looks weird, it would be good to make both eyes share the same animation once. Or, as in Owl's sample, it would be good to combine the animations for the face and eye direction together for now.