Unity Severe Freezes After Last Spine Update

thesnake91

Hey!
We've been patiently waiting for the multi-threading update as animations have been our main performance bottleneck - Now that it's here we've updated to the latest version and we are seeing significant performance gains! We're running into a couple of pretty fraustrating issues though:

1- Unity now constantly freezes, the pattern is a mini freeze that lasts about 8 seconds then a minute or two after we get one that lasts long enough that we usually force quit Unity.
2- Building the game went from ₃-5 mins to now 15-20 mins, Unity just gets stuck on importing assets for the majority of the time and only really builds in the last few mins
I haven't seen these issues before and the only thing that has changed is updating spine

Really not sure what's going on and can't think of a way so far to be able to debug it - Any ideas

Versions used

Spinebot

It seems like the issues you're experiencing with Unity freezing and extended build times after updating Spine could be related to the integration of the new multi-threading features or other changes in the latest Spine update. To address these issues, ensure that you have updated both the spine-csharp and spine-unity runtimes to their latest versions, as mismatched versions can cause unexpected behavior. Additionally, check if there are any specific changes in your project setup or code that might be interacting poorly with the new Spine update. If possible, try to isolate the problem by creating a minimal reproduction project that still exhibits the issue. This can help in identifying whether the problem is with the Spine update or something specific to your project setup.

thesnake91

BTW using Unity 6000.0.45f1

Misaki

thesnake91 The reason for the freeze is probably because you haven't disabled Automatic Upgrade Checks yet. However, you should only disable this after completing all upgrade steps. If you haven't yet read the upgrade guide, please read it carefully before disabling: Spine-Unity 4.2 to 4.3 Upgrade Guide

thesnake91

Misaki I did complete the upgrade and I had the same theory which is why I disabled the automatic upgrades in the preferences but the issue still persists

Harald

thesnake91 Just to be sure: Disabling the automatic updates via the Spine Preferences sets a define in your project. Could you please check whether the file ProjectSettings/ProjectSettings.assets contains scriptingDefineSymbols: { } with many platforms listing SPINE_AUTO_UPGRADE_COMPONENTS_OFF? I assume that you're building on your local device as well, or otherwise have the ProjectSettings.assets committed to your version control system and thus available in the same form to your build environment. Another check would be to see whether the Spine Preferences show only the Enable button next to Split Component Upgrade (if so, it properly has the define set).

Harald

@thesnake91 Very sorry to hear you're having issues, and thanks for reporting! Unfortunately this did not occur on our end so far. Did you see any warnings or errors logged in the Console window (e.g. do you see an error message "Internal threading logic error: exited Update loop after timeout!" or anything of this kind)?

We would be very happy if you could send us a minimal Unity reproduction project which still shows this issue. You can send it to contact@esotericsoftware.com, briefly mentioning this forum thread URL so that we know the context. If a minimal project can't be destilled down, we would also be happy to receive a larger project if you can share one.

If that's not possible: To help diagnose the problem in the meantime: does the issue occur only when under Edit - Preferences - Spine section Threading Defaults you have Threaded Mesh Generation or Threaded Animation enabled, does disabling either of the two or disabling both settings fix the situation (assuming you have not set these parameters explicitly on each component like SkeletonRenderer and SkeletonAnimation instead of using Use Global Setting)?

Also if that's possible, could you perhaps attach the debugger to the Unity project and hit Pause in the debugger as soon as it freezes for a longer period of time? Then in the Threads window in the debugging view select the main thread. This should allow us to see where exactly it's stuck.

Harald

@thesnake91 Also, could you please share a complete screenshot of your Spine Preferences settings? And in addition to that, what Enter Play Mode Settings are you using in Edit - Project Settings - Editor (Domain and Scene Reload settings)?

Harald

@thesnake91 If you're having Fix Prefab Overr. MeshFilter enabled in Edit - Preferences - Spine, we've just pushed a bugfix related to unnecessary updates and revert operations (issue ticket here). Could you please have a try whether the latest package update improves the situation on your end?

thesnake91

Hey!! So to update on this I was able to fix most of the issues! Here are some learnings and what I have left

SPINE_AUTO_UPGRADE_COMPONENTS_OFF was present in ProjectSettings.assets as you outlined so I think that's properly turned off

1) Freezes:
I had a really hard time determining the cause since attaching a debugger and pausing (which has helped me identify infinite loops and such in the past) was not yielding anything useful. What I found out is that with the new multi-threaded implementation, callbacks were now causing issues as they were not being triggered from the main thread. What I did is when unity froze, I just left it aside to see if after a while it would unfreeze I would see some indicator somewhere of what's going on, turned out to be our custom Day/Night cycle which causes some units in the game to sleep when night hits which plays their sleep animation triggering call backs and such, specifically the failure was showing up in Dotween for some reason, I haven't took the time to fully understand why it showed up in Dotween but disabling Dotween in this flow fixed the issue. That callbacks pattern though was all over the place, in other instances in just threw an internal threading exception. I just implemented a gloabl "MainThreadDispatcher" that queues actions for callbacks instead of performing the actions directly, I had to do that all over the place but it restored functionality and got rid of the errors. I thought this wouldn't happen by default though as I saw "callbacks fencing out to be called from the main thread by default" was listed as one of the notes for the threading implementation, not sure If I changed that behaviour somehow or if the defaulting logic is not working as intended, I tried finding a setting to change this but I couldn't

2) Build times
This one I still don't have an answer for, the game used to build in 3-5 mins, and now it takes around 15-18 mins to build, nothing really changed outside of the spine upgrade so I'm not sure why it trippled. In the editor logs I see the following pattern of logs repeating during the longest phase of the build process (at the beginning)

Unloading 1 Unused Serialized files (Serialized files now loaded: 0)
Unloading 9 unused Assets / (3.1 KB). Loaded Objects now: 103541.
Memory consumption went from 15.53 GB to 15.53 GB.
Total: 117.411100 ms (FindLiveObjects: 6.323100 ms CreateObjectMapping: 2.684000 ms MarkObjects: 108.237900 ms  DeleteObjects: 0.165100 ms)

Really not sure if this is directly related to spine in anyway but I would assume so just based on the timing of it happening after the update, could be also something that resulted from the automated migration process.

Awesome work though on the multi threading! Spine is much more performant now in our project

Harald

thesnake91 1) Freezes: [..]
That callbacks pattern though was all over the place, in other instances in just threw an internal threading exception. I just implemented a gloabl "MainThreadDispatcher" that queues actions for callbacks instead of performing the actions directly, I had to do that all over the place but it restored functionality and got rid of the errors. I thought this wouldn't happen by default though as I saw "callbacks fencing out to be called from the main thread by default" was listed as one of the notes for the threading implementation, not sure If I changed that behaviour somehow or if the defaulting logic is not working as intended, I tried finding a setting to change this but I couldn't

That is very strange, as:

The threaded update system is only instantiated and used when in Play mode, and not used in Edit mode for any updates.
MainThreadUpdateCallbacks defaults to true (I assume you haven't changed that to false) and any callbacks issued from updating would only be issued from a worker thread in Play mode (see (1) above).

Did you perhaps encounter your issues inside of Play mode, or shortly after entering Play mode (in the first frame)?

While investigating potential bugs or oversights, I just noticed that the documentation and notes did not yet explicitly mention that AnimationState callbacks have to be registered differently to be called from the main thread, sorry for forgetting about that! While any other update callbacks are fenced out to the main thread automatically, for AnimationState events like SkeletonAnimation.AnimationState.Complete you have to register to SkeletonAnimation.MainThreadComplete to receive them on the main thread. I assume that your problems were coming from AnimationState events, and not from other callback delegates, is this correct? We'll for now add a remark to the issue ticket and to the changelog and check where in the code documentation we can mention this so that it does not get overlooked. Sorry again for this oversight!

There is one exception of the above events: OnPostProcessVertices, which is deliberately left on worker threads so that parallellization can be utilized here.

thesnake91 2) Build times
This one I still don't have an answer for, the game used to build in 3-5 mins, and now it takes around 15-18 mins to build [..]

This one remains mysterious indeed. Have you tried whether under Edit - Preferences - Spine, section Threading Defaults, disabling Threaded Mesh Generation or Threaded Animation changes the situation in any way?

thesnake91 Awesome work though on the multi threading! Spine is much more performant now in our project

Thanks very much for the feedback, very glad to hear it improved the situation on your end as well!

thesnake91

Hey!

Yes to your first point, the issue doesn't show up if I never enter play mode so I'm pretty sure it's corelated to that.

Complete, Interrupt and also Event all caused the issue. To clarify though here's an example of the pattern we're using

` var track = SetAnimation(0, tertiaryAction, false);

            track.Complete += OnTertiaryCompleteBlend;
            track.Interrupt += OnTertiaryCompleteBlend;

I'm assuming the pattern you're mentioning should apply toTrackEntryas well? And also for performance, do you recommend usingMainThreadCompletevs my custom implementation of a globalMainThreadDispatcher`

My current pattern after the changes
`
var track = SetAnimation(0, tertiaryAction, false);

            track.Complete += (trackEntry) => MainThreadDispatcher.Enqueue(() => OnTertiaryCompleteBlend(trackEntry));
            track.Interrupt += (trackEntry) => MainThreadDispatcher.Enqueue(() => OnTertiaryCompleteBlend(trackEntry));

I have already tried to disable threading to see if any impact on build times but no luck, it's not the end of the world but is certainly inconvenient, I'm sure I'll find the cause at some point though. If it's related to spine I will be sure to share it

OH something else I forgot to mention, In SkeletonAnimationBase, there's a function OnValidate

protected void OnValidate () { if (!Application.isPlaying) { Renderer.OnRebuild -= OnRendererRebuild; Renderer.OnRebuild += OnRendererRebuild; requiresEditorUpdate = true; } else if (Time.frameCount != 0) { // OnValidate is called once when starting play mode in the Editor, don't trigger re-init then. requiresEditorUpdate = true; } }
For some reason, it throws an NPE everytime I build the game, I literally added a return; at the top of the function to bypass it temporarily as I'm not sure what is failing here

No worries at all on the misses in documentation, again it's a big change and I'm appreciative that I can get support and that I'm able to share my feedback directly!

Harald

Yes to your first point, the issue doesn't show up if I never enter play mode so I'm pretty sure it's corelated to that.

Thanks for the info. Which Enter Play Mode Settings are you using in Edit - Project Settings - Editor (Domain and Scene Reload settings)?

thesnake91 And also for performance, do you recommend usingMainThreadCompletevs my custom implementation of a globalMainThreadDispatcher`

The built-in MainThreadComplete handler works by enqueueing a struct in a List<>:
mainThreadEventQueue.Add(new EventQueueEntry(EventType.Complete, entry));
while yours enqueues a lambda. Unfortunately your's will likely cause unnecessary allocations (unless you are building for a platform where this is resolved in a better way). You could run a test in the editor and on the target device whether this is actually the case or not.

Complete, Interrupt and also Event all caused the issue. To clarify though here's an example of the pattern we're using

Thanks for the update, glad to hear it has only been these callbacks, where it is expected.

I'm assuming the pattern you're mentioning should apply toTrackEntryas well?

Yes.

thesnake91 I have already tried to disable threading to see if any impact on build times but no luck, it's not the end of the world but is certainly inconvenient, I'm sure I'll find the cause at some point though. If it's related to spine I will be sure to share it

It's partially good news that it's not related to the threading settings, which would be even more difficult to track down, apart from the fact that during a build and outside of play-mode in general no threaded mesh or animation updates shall happen at all.

If you have a minimal or even larger reproduction project, please also don't hesitate to let us know, we would be very happy to resolve any such issues.

thesnake91 OH something else I forgot to mention, In SkeletonAnimationBase, there's a function OnValidate

protected void OnValidate () {
if (!Application.isPlaying) {
Renderer.OnRebuild -= OnRendererRebuild;
Renderer.OnRebuild += OnRendererRebuild;
requiresEditorUpdate = true;
} else if (Time.frameCount != 0) {
// OnValidate is called once when starting play mode in the Editor, don't trigger re-init then.
requiresEditorUpdate = true;
}
}
For some reason, it throws an NPE everytime I build the game, I literally added a return; at the top of the function to bypass it temporarily as I'm not sure what is failing here

Oh, this is strange indeed, thanks for sharing! Could you please share the stack-trace of the NullReferenceException? It should be written to the build log.

Which platform are you building for, and does that make any difference?

thesnake91 No worries at all on the misses in documentation, again it's a big change and I'm appreciative that I can get support and that I'm able to share my feedback directly!

Thanks for your patience and understanding! We hope that these issues can be resolved soon, also thanks to your feedback and insights.

thesnake91

Hey Harald!

I'm not gonna lie, that build issue was driving me INSANE - There are a few bugs that are only seen in builds and not in editor and attempting to fix those with 15-20 min build times per attempt was not great.

It seems I found the issue though!

In SpineBuildProcessor.cs there are multiple calls to EditorUtility.UnloadUnusedAssetsImmediate(), commenting those out dropped the build times from 15-20 mins to under 3 mins! I don't see any side effects so far from commenting those out either.

Can I get more context on why these calls are made? Am I missing any side effects by commenting them out?

Thank you!

Harald

@thesnake91 Thanks very much for reporting and very sorry for the inconvenience! Oh dear, it looks as if there has been a regression going on, as these UnloadUnusedAssetsImmediate calls have already been moved out of the loops in the past, and some more improvements were removed (also ignoring Preferences setting Optimize Preview Meshes which you can disable to speed up builds at the cost of larger build size). We've fixed this regression, re-adding past improvements back in.

In addition to that, we've now removed the UnloadUnusedAssetsImmediate calls completely, as these shall not be necessary and any harm done in the sense of delaying the build needs to be prevented.

BTW: While frequently building for bugfixing, you might want to consider disabling Optimize Preview Meshes if you haven't already.

A new spine-unity 4.3-beta package has just been released.
https://esotericsoftware.com/spine-unity-download

Issue ticket URL for later reference:
EsotericSoftware/spine-runtimes2960

thesnake91

Thank you so much for the quick responses and fixes!!

I've been fighting with another issue that I'm having a hard time understanding.

To start off the weirdness, here's my temporary fix that worked for a while (called on start)

            skeletonAnimation.enabled = false;
            MonoBehaviourExtensions.CallWithDelay(this, () =>
            {
                skeletonAnimation.enabled = true;
            }, 0.1f);

For some reason, when I edit one of my prefabs and save them, units now suddenly spawn with no animations playing, like SetAnimation is being called on them but they're just frozen... I can see sometime that they slightly change state when I trigger an animation but it's like nothing actually starts playing, the fix for this that works sometimes is to turn off the SkeletonAnimation and turn it back on. Now even that is inconsistent, sometimes it would work for a few days in then seemingly randomly it would stop working for all units. This doesn't seem to be directly corelated with threading, as turning it on or off doesn't change anything.

A fix that I found that seems to work so far is to completely remove skeletonAnimation component from the prefab, exist and come back in and add it. Out of curiosity I did a diff on the .prefab file to see if I can see anything and here's what I'm seeing

Before

--- !u!114 &8215447342615759975
MonoBehaviour:
  m_ObjectHideFlags: 0
  m_CorrespondingSourceObject: {fileID: 0}
  m_PrefabInstance: {fileID: 0}
  m_PrefabAsset: {fileID: 0}
  m_GameObject: {fileID: 7592681743913139351}
  m_Enabled: 1
  m_EditorHideFlags: 0
  m_Script: {fileID: 11500000, guid: d247ba06193faa74d9335f5481b2b56c, type: 3}
  m_Name: 
  m_EditorClassIdentifier: 
  updateTiming: 1
  threadedAnimation: 2
  isUpdatedExternally: 1
  animationName: 
  loop: 0
  timeScale: 1
  unscaledTime: 0
  wasDeprecatedTransferred: 0
  skeletonDataAssetDeprecated: {fileID: 0}
  initialSkinNameDeprecated: 
  editorSkipSkinSyncDeprecated: 0
  initialFlipXDeprecated: 0
  initialFlipYDeprecated: 0
  updateModeDeprecated: 3
  updateWhenInvisibleDeprecated: 3
  separatorSlotNamesDeprecated: []
  zSpacingDeprecated: 0
  useClippingDeprecated: 1
  immutableTrianglesDeprecated: 0
  pmaVertexColorsDeprecated: 1
  clearStateOnDisableDeprecated: 0
  tintBlackDeprecated: 0
  singleSubmeshDeprecated: 0
  addNormalsDeprecated: 0
  calculateTangentsDeprecated: 0
  maskInteractionDeprecated: 0

After

--- !u!114 &8777365795840085696
MonoBehaviour:
  m_ObjectHideFlags: 0
  m_CorrespondingSourceObject: {fileID: 0}
  m_PrefabInstance: {fileID: 0}
  m_PrefabAsset: {fileID: 0}
  m_GameObject: {fileID: 7592681743913139351}
  m_Enabled: 1
  m_EditorHideFlags: 0
  m_Script: {fileID: 11500000, guid: d247ba06193faa74d9335f5481b2b56c, type: 3}
  m_Name: 
  m_EditorClassIdentifier: 
  updateTiming: 1
  threadedAnimation: 2
  isUpdatedExternally: 0
  animationName: 
  loop: 0
  timeScale: 1
  unscaledTime: 0

With limited knowledge of the internals of Spine, my guess is that isUpdatedExternally being turned on here is the issue - Does the auto upgrade process do that? Assuming that this means that it's expected that a script is going to intiate animation updates rather than them happening every frame, this is not something we ever turned on for any skeleton.

Final note, when this happened the auto upgrade option was turned on... After updating spine it seems to turn itself back on automatically

Again, thank you for all your hard work!