SilverStraw

I spent over a week ( working on this after my full-time work ) researching and developing a working Vtuber web solution. It's time for Spine Pro to level the playing field with Live2D in the Vtube realm. I expect to upload this application after I have enough features.


------------------------------------------------
Edit ( 7-4-2022 ) Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released.

https://silverstraw.itch.io/spine-vtuber-prototype

I also include the model that I have been using for testing.

https://silverstraw.itch.io/spine-vtube-test-model

SilverStraw
  • Posts: 55

Nate

Very cool! We have plans to do this officially. It's still early days, but it's super cool!
User avatar
Nate

Nate
  • Posts: 11939

SilverStraw

I beat you to the punch, Nate. :grinteeth:
Nate wrote:Very cool! We have plans to do this officially. It's still early days, but it's super cool!
Official support would be awesome! :heart:
In the meantime, I will spearhead the frontier. I bet your plate is full with Spine's future development. Maybe we can have some discussions in the future. ;)
SilverStraw
  • Posts: 55

Nate

You did! :punch:

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!
User avatar
Nate

Nate
  • Posts: 11939

Erika

Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<
User avatar
Erika

Erikari
  • Posts: 3082

Mario

Super cool! From the debug render on the left, it seems we are using the same tensorflow light model :)
User avatar
Mario

Mario
  • Posts: 3082

SilverStraw

Nate wrote:You did! :punch:

Besides the camera/vtubing itself, we also have some supporting features in the works that makes vtubing super cool: physics! We're in a bit of a rut though, as we started these things and then switched gears to improve the new graph. Lately there's been a lot of distractions that eat our time and/or fracture our focus, plus the holidays, so even that's going much more slowly than we want. We do keep plodding along though, so we'll get there in time!
Yay! Physics!
It's been slow for me as well. Now, I am trying to figure out how to allow user to load their Spine model that would work with spine-ts runtime.
Erika wrote:Oooh now that looks very promising as well, SilverStraw!
I'm so excited about being able to use Spine rigs for streams >.<
Hype! :fiesta:
I can't wait to create Spine models for streaming. Live2D Cubism has a completely different approach to model rigging.
Mario wrote:Super cool! From the debug render on the left, it seems we are using the same tensorflow light model :)
Are we really using the same? :bigeyed:
Now, tell me more. :smirk:

---

UPDATE:
I finally got the web browser file reader to work properly with Spine WebGL runtime. My file browser isn't shown in this video but I am clicking the asset files to be read and loaded into Spine runtime. This is an important step towards Vtubing with Spine rigs.
SilverStraw
  • Posts: 55

Nate

Cool! It can also be done via drag and drop. Working with browsers is such pain!
User avatar
Nate

Nate
  • Posts: 11939

SilverStraw

Nate wrote:Cool! It can also be done via drag and drop. Working with browsers is such pain!
Nate is telling me something.
Okay, I found the example of the drag and drop from Spine Player generator example.

---

I am updating on my progress with my application. The last couple months has been hectic with work and my other project failures.

SilverStraw
  • Posts: 55

Mario

Neat!
User avatar
Mario

Mario
  • Posts: 3082

Misaki

Great, I’m so excited! :D
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

Hello Mario and Misaki.

This is another step closer. I tried several approaches to operate the mouth on the model.

SilverStraw
  • Posts: 55

Misaki

OMG, it looks very easy to use! The mouth behavior looks natural :D Although the character has no eyelid yet, just being able to open and close the mouth conveys face expression quite well.
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

The character is evolving. I will get to the eyelids . . . eventually.
---
This should be ready for the next phase of testing.
SilverStraw
  • Posts: 55

Misaki

Great evolving! :D :yes:
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

I used all my rare candies, Misaki. ;)

Spine Vtuber Prototype itch.io release

The Vtuber Prototype using Esoteric Software Spine models is released. I also include the model that I have been using for testing.

https://silverstraw.itch.io/

SilverStraw
  • Posts: 55

Misaki

Congrats on the release! :D
I tried with a simple character which was created based on your Spine project file and it looks good! :heart:
Here's what I tried:

Sorry about the lack of blink animations since I drew and rigged it in a hurry and about the closed mouth not being clean enough! I will make various modifications later.

It may depend on the environment to do this, but it seems that eye pupil movement tends to be a bit jumpy. However, the response for facial tilt, direction, and mouth movements are very smooth! Also, the setup was easy to understand, so creating my own project was easy for me. It is a lot of fun to try out the movements with this tool :)
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.

Another factor is the small size of the eye pupils relative to the face. They were twitching a lot so I had to smooth the incoming signal.That solution isn't perfect. It's hard to distinguish jitter from eye pupil movements. In the future I will allow users to adjust these incoming signals.
SilverStraw
  • Posts: 55

Misaki

I have added some modifications to my skeleton:

Eye glasses affect the reliability of eye pupil tracking.I think it's because of the glare on the eye glasses. I used another software called Vtube Studio and it also has this issue. Eye glasses are a bane to eye pupil tracking.
Unfortunately, I tried it without eye glasses last time and this time, but it seems the eye pupil tracking response is not very good :tear:

Currently, the right eye and the left eye can register separate animations, but since it often happens that only one eye moves and looks weird, it would be good to make both eyes share the same animation once. Or, as in Owl's sample, it would be good to combine the animations for the face and eye direction together for now.
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

Another issue could be certain cameras mirror the image such as left and right are flipped. Are you familiar with Open Broadcasting Studio ( OBS )? It has functionality to create a virtual camera that you can flip horizontally. If you have another web camera that doesn't mirror, you can test it out. I noticed this issue as well when I switched over to my laptop camera which mirrors the image. Currently the prototype only cater to non-mirror images but I could give that an option in the future.

Your feedback has been helpful!
SilverStraw
  • Posts: 55

Misaki

I have OBS installed, but even if I started the virtual camera, I could not use it because the option does not appear in Google Chrome's camera settings. To be precise, in the general settings of Chrome, I can set like the following:
Screen Shot 2022-07-08 at 13.46.46.png

However, on the actual web page, it was automatically changed to FaceTime HD Camera and could not be freely changed here.
Screen Shot 2022-07-08 at 13.47.00.png

Incidentally, Google Meet allowed me to choose the virtual camera.
I have looked for other ways to flip the camera, but unfortunately have not found any.
I have attached the project file above so you can see if the problem does not occur in your environment.
You do not have the required permissions to view the files attached to this post.
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

I think my eye pupil jitter filter is too strong for your model, Misaki. I will lower the strength and update the change as soon as I get the chance. On my test model I though the filter was the right strength. It's great to see another model and how model rigs are affected differently. :think:
_____________________________________
I released a small patch, version 1.0.2.

* Changed canvas resize functionality.
* Decrease the strength of jitter filter for the eye pupils.

Hello Misaki. I hope this weaker filter works better for your model.
SilverStraw
  • Posts: 55

Misaki

Thank you for your quick fixing, it looks much better! :D The movement has become smooth, and weird movements such as only one eye moving are less than before.


In the video above, I tried to roll my eyes 360 degrees, but unfortunately the up and down did not seem to respond very well. The distance between up and down is shorter than left and right, so it seems to be hard to catch the movement. I'm not sure if it would be optimal to have the filter adjusted based on my model, so I think it would be nice if the settings could be adjusted by the user.

By the way, I will modify my model later as the rigging of the eyes and mouth is still not very good. I'll share it again when the improvements are done.
User avatar
Misaki

Misaki
  • Posts: 760

SilverStraw

So the jitter filter is for the model to stop twitching so much. Much like the .gif below when your eyes are not moving but the model is twitching excessively. :bigeye:

I had the filter strong enough to cause the eye pupils to jump. I thought it was favorable effect to have jumpy eye pupils. :upsidedown:
I lowered it to the same level as the other filters as the face and mouth. There should not be too much bias of the change.

Letting the user adjust the face tracking setting is for future updates. When you allow those adjustments, users would want a way to save the settings so they do not have to adjust every setting each time.
SilverStraw
  • Posts: 55

Misaki

So the jitter filter is for the model to stop twitching so much. Much like the .gif below when your eyes are not moving but the model is twitching excessively.
Ah, I see.
When you allow those adjustments, users would want a way to save the settings so they do not have to adjust every setting each time.
Exactly. I think it would be great if you can download the settings as a file such as a JSON format and load it with the skeleton data when you want to apply the settings again.
User avatar
Misaki

Misaki
  • Posts: 760


Return to Showcase