Chat downloading fails at "Embedding Images" step

[Wicloz]

Info Page:

After updating chat downloads no longer get stuck but instead fail at the end with the following error:

Unhandled exception. System.AggregateException: One or more errors occurred. (A task was canceled.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.Task.GetExceptions(Boolean )
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean )
   at System.Threading.Tasks.Task.Wait(Int32 , CancellationToken )
   at System.Threading.Tasks.Task.Wait()
   at TwitchDownloaderCLI.Modes.DownloadChat.Download(ChatDownloadArgs inputOptions)
   at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult`1 result, Action`1 action)
   at TwitchDownloaderCLI.Program.Main(String[] args)
--- End of stack trace from previous location ---

   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean )
   at System.Threading.Tasks.Task.Wait(Int32 , CancellationToken )
   at System.Threading.Tasks.Task.Wait()
   at TwitchDownloaderCLI.Modes.DownloadChat.Download(ChatDownloadArgs inputOptions)
   at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult`1 result, Action`1 action)
   at TwitchDownloaderCLI.Program.Main(String[] args)

Is there something wrong with my setup or is chat still bugged for now?

Environment for the API container:

      TZ: Europe/Amsterdam
      DB_HOST: host.docker.internal
      DB_PORT: 5432
      DB_USER: REDACTED
      DB_PASS: REDACTED
      DB_NAME: REDACTED
      DB_SSL: disable
      JWT_SECRET: REDACTED
      JWT_REFRESH_SECRET: REDACTED
      FRONTEND_HOST: https://twitch.wicloz.rocks
      COOKIE_DOMAIN: twitch.wicloz.rocks
      TWITCH_CLIENT_ID: REDACTED
      TWITCH_CLIENT_SECRET: REDACTED

[Zibbp]

Twitch made some breaking changes to the chat GraphQL endpoint, it could be that. Can you provide a video ID so I can try on my end?

[Wicloz]

This happens to for example 1811359837 but it happened to every VOD I tried to archive.

[Wicloz]

https://www.twitch.tv/videos/1810500939 is another example I just tried.

[Zibbp]

I was able to download 1811359837 without issue. Can you try pulling the latest docker image and trying again?

[STATUS] - Downloading 94%
[STATUS] - Downloading 95%
[STATUS] - Downloading 96%
[STATUS] - Downloading 97%
[STATUS] - Downloading 98%
[STATUS] - Downloading 99%
[STATUS] - Downloading + Embedding Images
[STATUS] - Backfilling commenter info
[STATUS] - Writing output file

[Wicloz]

I have completely reinstalled Ganymede including a fresh DB and data folders and still get the same error. Is there any way I can get some additional log output to help resolve the issue?

I am only using the API and frontend containers, and handling the database and web server manually.

Could the issue perhaps relate to the database or my Twitch extension?

[Zibbp]

Nothing with your setup should effect the chat download to error out like that. What are you hosting the API container on? Docker? A NAS? What architecture is your CPU? x86 or arm?

[Wicloz]

I created a new Docker network without IPv6 support and remade the API container using that network. With this the embedding step works again... This seems like an issue with TwitchDownloader or my network setup.

[Zibbp]

Not sure why disabling ipv6 for the docker network would resolve the issue. Is that the only difference between the two networks?

[Wicloz]

Default bridge network with IPv6 support:

• https://twitch.wicloz.rocks/channels/channelname works.
• Embedding Images fails.

Separate bridge network with IPv6 support:

• https://twitch.wicloz.rocks/channels/channelname gives 404.
• Embedding Images fails.

Separate bridge network without IPv6 support:

• https://twitch.wicloz.rocks/channels/channelname gives 404.
• Embedding Images succeeds.

In all cases the network is used by both the front end and API. Command used to create networks: docker network create (--ipv6) ganymede.

[Wicloz]

I have resolved the 404 errors, they are unrelated to this issue.

[SzzoekzZx9]

As per this issue at TwitchDownloader, Issue #674, it would appear this is a known issue due to Twitch deprecating the current badges API. I am not sure the timeline for fixing this, however, this is blocking me from being able to archive chats at all.

Is it possible to add an "archive chat (no badges)" options? Or would it be best to just wait for the next TwitchDownloader version?

I just wanted to bring this to everyone's attention, as I only recently started hosting this project (right after Twitch breaks the tools relying on that API) and this issue caused some headaches!

[Zibbp]

I am not sure the timeline for fixing this, however, this is blocking me from being able to archive chats at all.

You're unable to download the chat of a VOD or livestream? I've been able to archive vods recently with no issues.

[Wicloz]

I am able to download and render chat ATM make sure you are using the latest version.

This issue specifically happened because of using a Docket network with IPv6 support.

[SzzoekzZx9]

I am unable to download chat through the frontend. I am able to download chat inside the API container using the TwitchDownloaderCLI program if I do not include the embed badges option. Is it possible that the badges are cached for you, since you have mature installations, and the downloader is not reaching out to the old API, the one causing issues?

I just deleted everything, repulled the docker images, and tested again and it still did not work. This is the version info from the frontend:

Frontend
Version:
c81f1c5de0881ac3d11c40abe6ab9abe0c64fee4-2023-4-30
API
Version:
Build Date:
2023-05-06T02:35:38Z
Git Commit Hash:
79e25a3
Uptime:
3h57m47.171522989s
Package Versions
FFmpeg:
ffmpeg version 5.1.3 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12.2.1 (Alpine 12.2.1_git20220924-r4) 20220924
configuration: --prefix=/usr --enable-avfilter --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libpulse --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-libdav1d --enable-lto --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --enable-librist --enable-libsrt --enable-libssh --enable-libvidstab --disable-stripping --disable-static --disable-librtmp --disable-lzma --enable-libaom --enable-libopus --enable-libsoxr --enable-libwebp --enable-vaapi --enable-vdpau --enable-vulkan --enable-libdrm --enable-libzmq --optflags=-O2 --disable-debug --enable-libsvtav1
libavutil 57. 28.100 / 57. 28.100
libavcodec 59. 37.100 / 59. 37.100
libavformat 59. 27.100 / 59. 27.100
libavdevice 59. 7.100 / 59. 7.100
libavfilter 8. 44.100 / 8. 44.100
libswscale 6. 7.100 / 6. 7.100
libswresample 4. 7.100 / 4. 7.100
libpostproc 56. 6.100 / 56. 6.100
TwitchDownloader:
TwitchDownloaderCLI 1.52.8
Chat Downloader:
0.2.5
Streamlink:
streamlink 5.5.0

Are these the latest versions of everything, as expected?

EDIT: I just built TwitchDownloaderCLI from source and it was able to download chats with embedded badges so it seems the fix is in place. I am trying to overwrite the version in the API with the one I built. I will report back any successes.

EDIT 2: It works now. I cloned the TwitchDownloaderCLI and built TwitchDownloaderCLI locally for LinuxAlpine. I then copied the new build into the container and reran the archive request from the Ganymede front end. It successfully completed.

I am not sure why you both were able to get it to run, concidering the TwitchDownloader repo seems to have had to impliment changes to the getBadgeData function to successfully retrieve this information. My only guess is that your badges are cached or somthing? I won't pretend to know your system better than you though ;)

Thanks for the help! I can give more information regarding my process if needed.

[Zibbp]

I cleared all the files in /tmp in my dev instance and I was able to archive a vod with badges. Perhaps it's something with a specific vod / channel? Anyway, glad you got it to work for now. I'll be on the lookup for any new TwitchDownloader releases.

Another thing to note is that you don't need to the build the package yourself if you don't want to. The TwitchDownloader CI pipelines against master build all the packages. If you scroll down on the latest CI pipeline for master https://github.com/lay295/TwitchDownloader/actions/runs/5063448529 you will see an "Artifacts" section in which you can download various builds.

Maybe in the future I can add a flag that will pull the latest master branch build and use that, rather than a tagged release.

[Zibbp]

v1.3.0 has just been released which uses the recently released v1.52.9 TwitchDownloader version (includes the badge fix).

[soleny894]

Hi,

I got this issue today (running v1.3.0):

[LOG] - Input 1st party emote count: 0. Output count: 15
Unhandled exception. System.AggregateException: One or more errors occurred. (The given key 'bits_charity' was not present in the dictionary.)
 ---> System.Collections.Generic.KeyNotFoundException: The given key 'bits_charity' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey )
   at TwitchDownloaderCore.TwitchHelper.GetChatBadgesData(List`1 comments, Int32 streamerId, CancellationToken cancellationToken)
   at TwitchDownloaderCore.TwitchHelper.GetChatBadges(List`1 comments, Int32 streamerId, String cacheFolder, EmbeddedData embeddedData, Boolean offline, CancellationToken cancellationToken)
   at TwitchDownloaderCore.ChatUpdater.ChatBadgeTask(IProgress`1 progress, CancellationToken cancellationToken)
   at TwitchDownloaderCore.ChatUpdater.UpdateEmbeds(Int32 currentStep, Int32 totalSteps, IProgress`1 progress, CancellationToken cancellationToken)
   at TwitchDownloaderCore.ChatUpdater.UpdateAsync(IProgress`1 progress, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean )
   at System.Threading.Tasks.Task.Wait(Int32 , CancellationToken )
   at System.Threading.Tasks.Task.Wait()
   at TwitchDownloaderCLI.Modes.UpdateChat.Update(ChatUpdateArgs inputOptions)
   at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult`1 result, Action`1 action)
   at TwitchDownloaderCLI.Program.Main(String[] args)

TwitchDownloader: TwitchDownloaderCLI 1.52.9
Chat Downloader: 0.2.7
Streamlink: streamlink 5.5.1 

[Zibbp]

Was this during a live stream chat convert, or a chat render? Can you provide a VOD ID or upload the chat json so I can take a look at it?

It appears the Twitch API is returning empty badges, see lay295/TwitchDownloader#726 and lay295/TwitchDownloader#729

[lay295]

The earlier part of this issue was probably the deprecation of the badge API, and the one from earlier today was a bug in migrating to the GQL API (whoops). Putting out 1.53.0 of TwitchDownloader now that should hopefully fix it.

[Zibbp]

Thanks for the quick update!

I've released v1.3.1 on my end which includes the updated TwitchDownloader version. After updating the container I would retry downloading the chat and rendering. Or if you're unable to re-download the chat (livestream) copy the chat back to the container's /tmp folder and run the chat render task again for the queue entry.