We’ve had a production level freepbx distro (all modules up to date and also all yum updates applied) running for about 1 month, no issues. Last week things went crazy.
Now several times per day extensions unregister and can no longer register. No incoming calls work either. The only solution Ive found is either a full system restart or a fwconsole restart.
Things that changed last week.
I activated the “Queues Pro” commercial module, but I have disabled and removed it from the system since. Removing it made no difference.
I played with asternic’s convert to MP3 settings.
I might have updated modules last week. I don’t remember.
So now, a few things I see (though I can’t be sure they weren’t there before)… I get the warnings about task processor being over 3000 all the time. I’ve checked and sometimes tasks have the queue column up to the hundreds of thousands. The system is a Xen Server virtual machine, with 4 cores and 16GB, but only about 100 extensions and Ive never seen more than 40 simultaneous calls.
Things seem to work normally and then the unregistration thing happens. I thought it was the queued up tasks, but Ive now seen the unregistration happen with queues at zero.
What I discovered now is that it seems mixmonitor starts recording and never stops. It also creates dozens of tiny 44byte files for each call.
My latest test is stopping all recordings and see if there are any problems, but I just made that change so nothing to report yet.
While in the CLI Ive also seen this FRACK and some backtracing, though this wasnt folliowed by the unregistration so no clue if this is related.
Any tips on what I should try to look for in order to debug this issue?
thank you!
-amco
PS.- problem just happened even though the call recordings are off, so maybe those 44 byte files are unrelated
some of the tasks that give warnings are:
subm:ast_channel_topic_all-cached-0000057b
subm:ast_channel_topic_all-00000583
subp:ast_channel_topic_all-00000895
subm:cel_aggregation_topic-00000006
this following one happens most often
subm:manager_topic-00000007 5818352 1650025 1663207 2700 3000
here is the frack i caught
[2018-08-17 14:59:11] ERROR[32159][C-00000035]: pbx_functions.c:701 ast_func_write: Function AUDIOHOOK_INHERIT not registered
[2018-08-17 14:59:12] ERROR[32207][C-00000011]: astobj2.c:518 __ao2_ref: Excessive refcount 100000 reached on ao2 object 0x2a189d0
[2018-08-17 14:59:12] ERROR[32207][C-00000011]: astobj2.c:518 __ao2_ref: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x2a189d0 (0)
Got 21 backtrace records
#0: [0x5fcc55] asterisk __ast_assert_failed() (0x5fcbd1+84)
#1: [0x45d4bd] asterisk ()
#2: [0x45d816] asterisk __ao2_ref() (0x45d7e8+2E)
#3: [0x5cf8e6] asterisk ()
#4: [0x5cfa69] asterisk ()
#5: [0x5c7786] asterisk ()
#6: [0x5c8338] asterisk ()
#7: [0x5c8620] asterisk ()
#8: [0x5c868f] asterisk stasis_publish() (0x5c8667+28)
#9: [0x5d4569] asterisk ast_channel_publish_snapshot() (0x5d44b7+B2)
#10: [0x582e74] asterisk pbx_exec() (0x582dda+9A)
#11: [0x570787] asterisk ()
#12: [0x573c6a] asterisk ast_spawn_extension() (0x573c06+64)
#13: [0x574945] asterisk ()
#14: [0x576115] asterisk ()
#15: [0x5f9c08] asterisk ()
[2018-08-17 14:59:13] ERROR[32223][C-00000031]: pbx_functions.c:701 ast_func_write: Function AUDIOHOOK_INHERIT not registered
[2018-08-17 14:59:15] ERROR[32336][C-0000002d]: pbx_functions.c:701 ast_func_write: Function AUDIOHOOK_INHERIT not registered
[2018-08-17 14:59:16] ERROR[32326][C-0000003a]: astobj2.c:518 __ao2_ref: Excessive refcount 100000 reached on ao2 object 0x2a18d20
[2018-08-17 14:59:16] ERROR[32326][C-0000003a]: astobj2.c:518 __ao2_ref: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x2a18d20 (0)
Got 23 backtrace records
#0: [0x5fcc55] asterisk __ast_assert_failed() (0x5fcbd1+84)
#1: [0x45d4bd] asterisk ()
#2: [0x45d816] asterisk __ao2_ref() (0x45d7e8+2E)
#3: [0x5d7ca2] asterisk stasis_message_create_full() (0x5d7c2e+74)
#4: [0x5d7d2a] asterisk stasis_message_create() (0x5d7d02+28)
#5: [0x5d3d4c] asterisk ()
#6: [0x5d3dab] asterisk ast_channel_blob_create_from_cache() (0x5d3d62+49)
#7: [0x5d45bf] asterisk ast_channel_publish_cached_blob() (0x5d457c+43)
#8: [0x5d4716] asterisk ast_channel_publish_varset() (0x5d466c+AA)
#9: [0x58ca5e] asterisk pbx_builtin_setvar_helper() (0x58c6fc+362)
#10: [0x7fd6f5b5912e] app_macro.so ()
#11: [0x7fd6f5b59552] app_macro.so ()
#12: [0x582ef6] asterisk pbx_exec() (0x582dda+11C)
#13: [0x570787] asterisk ()
#14: [0x573c6a] asterisk ast_spawn_extension() (0x573c06+64)
#15: [0x574945] asterisk ()
#16: [0x576115] asterisk ()
#17: [0x5f9c08] asterisk ()
[2018-08-17 14:59:16] ERROR[32338][C-0000003b]: astobj2.c:518 __ao2_ref: Excessive refcount 100000 reached on ao2 object 0x2a18d20
[2018-08-17 14:59:16] ERROR[32338][C-0000003b]: astobj2.c:518 __ao2_ref: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x2a18d20 (0)
Got 23 backtrace records
#0: [0x5fcc55] asterisk __ast_assert_failed() (0x5fcbd1+84)
#1: [0x45d4bd] asterisk ()
#2: [0x45d816] asterisk __ao2_ref() (0x45d7e8+2E)
#3: [0x5d7ca2] asterisk stasis_message_create_full() (0x5d7c2e+74)
#4: [0x5d7d2a] asterisk stasis_message_create() (0x5d7d02+28)
#5: [0x5d3d4c] asterisk ()
#6: [0x5d3dab] asterisk ast_channel_blob_create_from_cache() (0x5d3d62+49)
#7: [0x5d45bf] asterisk ast_channel_publish_cached_blob() (0x5d457c+43)
#8: [0x5d4716] asterisk ast_channel_publish_varset() (0x5d466c+AA)
#9: [0x58ca3e] asterisk pbx_builtin_setvar_helper() (0x58c6fc+342)
#10: [0x7fd6f5b58220] app_macro.so ()
#11: [0x7fd6f5b59552] app_macro.so ()
#12: [0x582ef6] asterisk pbx_exec() (0x582dda+11C)
#13: [0x570787] asterisk ()
#14: [0x573c6a] asterisk ast_spawn_extension() (0x573c06+64)
#15: [0x574945] asterisk ()
#16: [0x576115] asterisk ()
#17: [0x5f9c08] asterisk ()
[2018-08-17 14:59:16] ERROR[32370][C-0000003b]: pbx_functions.c:701 ast_func_write: Function AUDIOHOOK_INHERIT not registered