Esoteric Software

[SilverStraw]

I spent over a week ( working on this after my full-time work ) researching and developing a working Vtuber web solution. It's time for Spine Pro to level the playing field with Live2D in the Vtube realm. I expect to upload this application after I have enough features.


------------------------------------------------
Edit ( 7-4-2022 ) Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released.

https://silverstraw.itch.io/spine-vtuber-prototype

I also include the model that I have been using for testing.

https://silverstraw.itch.io/spine-vtube-test-model

[Nate]

Very cool! We have plans to do this officially. It's still early days, but it's super cool!

[SilverStraw]

I beat you to the punch, Nate.

Very cool! We have plans to do this officially. It's still early days, but it's super cool!

Official support would be awesome!
In the meantime, I will spearhead the frontier. I bet your plate is full with Spine's future development. Maybe we can have some discussions in the future.

[Nate]

You did!

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!

[Erika]

Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<

[Mario]

Super cool! From the debug render on the left, it seems we are using the same tensorflow light model

[SilverStraw]

You did!

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!

Yay! Physics!
It's been slow for me as well. Now, I am trying to figure out how to allow user to load their Spine model that would work with spine-ts runtime.

Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<

Hype!
I can't wait to create Spine models for streaming. Live2D Cubism has a completely different approach to model rigging.

Super cool! From the debug render on the left, it seems we are using the same tensorflow light model

Are we really using the same?
Now, tell me more.

---

UPDATE:
I finally got the web browser file reader to work properly with Spine WebGL runtime. My file browser isn't shown in this video but I am clicking the asset files to be read and loaded into Spine runtime. This is an important step towards Vtubing with Spine rigs.

[Nate]

Cool! It can also be done via drag and drop. Working with browsers is such pain!

[SilverStraw]

Cool! It can also be done via drag and drop. Working with browsers is such pain!

Nate is telling me something.
Okay, I found the example of the drag and drop from Spine Player generator example.

---

I am updating on my progress with my application. The last couple months has been hectic with work and my other project failures.

[Mario]

Neat!

[Misaki]

Great, I’m so excited!

[SilverStraw]

Hello Mario and Misaki.

This is another step closer. I tried several approaches to operate the mouth on the model.

[Misaki]

OMG, it looks very easy to use! The mouth behavior looks natural Although the character has no eyelid yet, just being able to open and close the mouth conveys face expression quite well.

[SilverStraw]

The character is evolving. I will get to the eyelids . . . eventually.
---
This should be ready for the next phase of testing.

[Misaki]

Great evolving!

[SilverStraw]

I used all my rare candies, Misaki.

Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released. I also include the model that I have been using for testing.

https://silverstraw.itch.io/

[Misaki]

Congrats on the release!
I tried with a simple character which was created based on your Spine project file and it looks good!
Here's what I tried:

Sorry about the lack of blink animations since I drew and rigged it in a hurry and about the closed mouth not being clean enough! I will make various modifications later.

It may depend on the environment to do this, but it seems that eye pupil movement tends to be a bit jumpy. However, the response for facial tilt, direction, and mouth movements are very smooth! Also, the setup was easy to understand, so creating my own project was easy for me. It is a lot of fun to try out the movements with this tool

[SilverStraw]

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.

Another factor is the small size of the eye pupils relative to the face. They were twitching a lot so I had to smooth the incoming signal.That solution isn't perfect. It's hard to distinguish jitter from eye pupil movements. In the future I will allow users to adjust these incoming signals.

[Misaki]

I have added some modifications to my skeleton:

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.

Unfortunately, I tried it without eye glasses last time and this time, but it seems the eye pupil tracking response is not very good

Currently, the right eye and the left eye can register separate animations, but since it often happens that only one eye moves and looks weird, it would be good to make both eyes share the same animation once. Or, as in Owl's sample, it would be good to combine the animations for the face and eye direction together for now.

[SilverStraw]

Another issue could be certain cameras mirror the image such as left and right are flipped. Are you familiar with Open Broadcasting Studio ( OBS )? It has functionality to create a virtual camera that you can flip horizontally. If you have another web camera that doesn't mirror, you can test it out. I noticed this issue as well when I switched over to my laptop camera which mirrors the image. Currently the prototype only cater to non-mirror images but I could give that an option in the future.

Your feedback has been helpful!

[Misaki]

I have OBS installed, but even if I started the virtual camera, I could not use it because the option does not appear in Google Chrome's camera settings. To be precise, in the general settings of Chrome, I can set like the following:

Screen Shot 2022-07-08 at 13.46.46.png

However, on the actual web page, it was automatically changed to FaceTime HD Camera and could not be freely changed here.

Screen Shot 2022-07-08 at 13.47.00.png

Incidentally, Google Meet allowed me to choose the virtual camera.
I have looked for other ways to flip the camera, but unfortunately have not found any.
I have attached the project file above so you can see if the problem does not occur in your environment.

[SilverStraw]

I think my eye pupil jitter filter is too strong for your model, Misaki. I will lower the strength and update the change as soon as I get the chance. On my test model I though the filter was the right strength. It's great to see another model and how model rigs are affected differently.
_____________________________________
I released a small patch, version 1.0.2.

* Changed canvas resize functionality.
* Decrease the strength of jitter filter for the eye pupils.

Hello Misaki. I hope this weaker filter works better for your model.

[Misaki]

Thank you for your quick fixing, it looks much better! The movement has become smooth, and weird movements such as only one eye moving are less than before.


In the video above, I tried to roll my eyes 360 degrees, but unfortunately the up and down did not seem to respond very well. The distance between up and down is shorter than left and right, so it seems to be hard to catch the movement. I'm not sure if it would be optimal to have the filter adjusted based on my model, so I think it would be nice if the settings could be adjusted by the user.

By the way, I will modify my model later as the rigging of the eyes and mouth is still not very good. I'll share it again when the improvements are done.

[SilverStraw]

So the jitter filter is for the model to stop twitching so much. Much like the .gif below when your eyes are not moving but the model is twitching excessively.

I had the filter strong enough to cause the eye pupils to jump. I thought it was favorable effect to have jumpy eye pupils.
I lowered it to the same level as the other filters as the face and mouth. There should not be too much bias of the change.

Letting the user adjust the face tracking setting is for future updates. When you allow those adjustments, users would want a way to save the settings so they do not have to adjust every setting each time.

[Misaki]

So the jitter filter is for the model to stop twitching so much. Much like the .gif below when your eyes are not moving but the model is twitching excessively.

Ah, I see.

When you allow those adjustments, users would want a way to save the settings so they do not have to adjust every setting each time.

Exactly. I think it would be great if you can download the settings as a file such as a JSON format and load it with the skeleton data when you want to apply the settings again.