Sonarr/Radarr/Lidar Code base

[ericflix]

Any chance of moving over the Sonarr/Radarr/Lidar Code base? Pretty much all the *arr apps were forked off Sonarr and modified. There is even a fork in the works for non manga comics - Kapowarr.

This would remove the added work of the UI other than customizations and let you work on the backend.

[Thundernerd]

I would really like to yeah. I initially looked at it and saw the massive code base without any proper guide on how to use/modify it without spending a lot of time myself trying to figure things out. It might be something that is actually worth it even though it'll cost a lot of time, but I'll have to look into that.

[ericflix]

Seeing the Kapowarr is a comic arr app, it might be quicker to fork off them.

I do like your approach on pulling from all the manga/manhwa/manhua sites, makes it much easier.

[Thundernerd]

I'll have a look at that one!

I specifically wanted to support all the websites that serve content out there because a lot of people have their own favorite fan translations that they want to read from and not everything gets uploaded to newsgroups or torrents. It would be nice if it can support both of those down the line though, but it isn't the main focus for now!

ETA:

Kapowarr actually seems like a custom implementation in python, so they aren't forked from Sonarr or any of the others. I might just have to dive into the original one

[Thundernerd]

So I finally had a look at Sonarr and was able to make some modifications to get some things going. It isn't fully working yet but made some promising progress!

Added a bunch of random manga/manhua/manhwa to the overview

And was able to properly check one, though the season information is a bit borked

Figured I'd might as well fork Jackett to see if I can use that as a indexer gateway and was able to make a small bit of progress there as well

[ericflix]

That looks great! How are you handling scraping chapters from different sites? Is that what you are modifying Jackett for? Will it be able to scrape sites and even parse/ingest cbr/cbz files?

Since you are using the Sonarr code base I am hoping that you will support postgres too. I'm not a dev, but if you need testing done on a docker/kubernetes environment let me know.

[Thundernerd]

If Docker supports postgres then this will automatically as well, I'm not really making any breaking changes, or at least not planning to!

The way it currently works is:

1. You search for a manga, it uses mangaupdates.com to map those search results
2. One you've selected/added your manga it'll scan the entry on mangaupdates.com for the releases, these are then converted to "episodes"
3. Whenever Sonarr does its refresh, or whenever you manually look for a specific item it'll contact Jackett
4. Whenever there's a match for an episode/chapter Sonarr will ask Jackett for a torrent file, I am basically abusing torrents as regular data containers
5. These torrents contain links to all images belonging to a chapter, which Sonarr will internally download through regular http requests
6. (it doesn't do this yet) It'll then create a ComicInfo.xml file and together with that bundle it all up into a cbz
7. That cbz file can then be automatically processed and imported by Sonarr putting it in the right place!

[ericflix]

Sounds great. I'm looking forward to trying it out when you get it to the point you want to release it.

Postgres is supported on all the *arr apps now, Directions are on the Servarr site. I'll let you know how it works then there is a container available for it.

[alexandru0-dev]

@Thundernerd That sounds great, do u have a repo/branch where i can find the wip?

[Thundernerd]

These are not yet available, I'm currently looking into making builds for both the Sonarr and Jackett forks and then turn them into docker images! Once I've confirmed that that is working as intended I'll be sure to notify both of you (and anyone else that wants to)

[alexandru0-dev]

I don't really need to be in a functional stage, just curious and if I have time I can look into it

Btw for builds I can help you, I'm also going to create nix builds for myself.

I've quickly seen your jackett fork commits but doesn't your changes completely remove the old jackett behavior?

So I would be a fork for only indexing manga sources no?

[Thundernerd]

These are the repo+branch:
https://github.com/tnrd-org/Sonarr/tree/mangarr
https://github.com/tnrd-org/Jackett/tree/mangarr

I expect them both to be completely separate solutions, as in the functionality won't be PR'd into the original repositories as what I'm doing doesn't match with the intended use of the original software.

In fact, the Jackett one abuses torrents as a whole and simply uses them as data containers that will be read by my Sonarr fork and the "Fake Torrent" download client that I've built, which in turn is quite a bit different from how Sonarr usually works

[alexandru0-dev]

Thx, I was looking at your profile instead of the org

Also can I ask why did u choose jackett over prowlarr?

[Thundernerd]

I am not familiar with Prowlarr, so I’ll have to take a look at it! I used Jackett because I know of it and figured it might be a nice base to start from. I’ll definitely have a look at Prowlarr though

[alexandru0-dev]

Sure, I've looked a bit and integrations of other *rr services should be easier and this also could lead to possibly upstream it.

Tho idk how the fact that mangas are mainly DDL and you should manage different sources could impact upstream.

If I have some answers I will ping you.

Thx for the great work again

[Thundernerd]

@alexandru0-dev I'm planning on switching over to Prowlarr. Most of the leg work for the indexers has already been done in Jackett so I "just" have to port them over!

[alexandru0-dev]

@Thundernerd first of all thx again

Is there a way I can contact you directly
Discord or Matrix for example?

I've asked in the prowlarr's discord server and there are some problems to tackle about multiple sources I think or I didn't understand how your implementation would work

So I wanted to chat properly

[Thundernerd]

@alexandru0-dev You can hit me up on Discord. I'm also in the Servarr discord under the same name as here so feel free to send me a message request there!

[markschrik]

Hi!

Nice work :)
Once this is available with a sonarr base, I am really interested in running it. If you want, please keep me posted too!

[Thundernerd]

I have made quite some progress and have successfully been running it for a couple of days on my own server now!

I can post a docker compose along with some setup steps if y’all are interested?

It is far from finished though, especially one major issue is that there’s no one source of truth when it comes to available chapters. I’m currently writing an indexer that periodically scrapes all sources to and then links them all together so I can use this to figure out what chapters have been released.

Either way, still quite some work to do, but it is possible to run already!

[g-nogueira]

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

[Thundernerd]

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

This is the repo for my Prowlarr fork which is where the sources will go:
https://github.com/donderjoekel/Prowlarr

[ericflix]

Have you reached out to the Servarr Wiki? They added the Whisparr project/fork to their list of *arr apps. If you can get Mangarr included to the *arr ecosystem, that would open up a large group in the community of devs that can help get Mangarr to a mature state faster.

It is also a possibility to get your manga scrappers added into the prowlarr project if there is no major changes needed.

[alexandru0-dev]

@ericflix I did, but unfortunately mangarr works only with the jackett/prowlarr forks and are not merge-able with upstream (excluding indexers, the rest is shared)

We can find people that are willing to collaborate on the project there, surely.

The issue about incompatibility is mainly having different sources for DDLs (how different sources need a different way to be downloaded, aggregated etc)
As for now it's not compatible with upstream unfortunately and I hardly think it is ever going to be in the short run especially with the project not having that much of traction rn.
As of now existing indexers are replaced with manga sources.

@Thundernerd correct me if I'm wrong.

[Thundernerd]

Yup that’s correct.

My forks of Sonarr and Prowlarr are not huilt to support torrents and/or newsgroups. Unfortunately from my experience manga isn’t uploaded in agreed-upon standards so it would be very hard to use that.

Instead what my forks do is utilize the fair share of manga hosting websites and turns them into sources that can be used to download the images directly and pack it up into an archive for you to use in your favorite reader.

The core of both my Sonarr and Prowlarr fork have strayed so far from the original intended functionality that these will never be merged upstream.

[jordanfelle]

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

[Thundernerd]

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

Maybe? I'm honestly not too sue actually! I'm trying to (ab)use as much as I can from stock Sonarr and just build on top of it, so it could very well work. That said, there's a high chance that it will not work, or already isn't working. If the upload community can band together and come up with a workable standard then I'm 100% down to make it work though.

[g-nogueira]

Didn't think you would try making it work with torrent. That's awesome! Nonetheless, it seems you're focusing on the scrapping part, which I do believe is the better way of getting manga/comics. Thank you for the hard work!

I'm really looking forward to give a look at the code base but, first, I'm trying to make my fork of the sonarr_youtubedl work well so that I can try merging it (but I don't have high expectations, as there are already some stale PR's there).

[Thundernerd]

Wanted to drop a message here with a little update that unfortunately I haven't been able to spend proper time on the new codebase.

It is still regularly in the back of my mind but due to sickness, being out of the country for work, and just generally being busy with all kinds of things I haven't been able to dedicate time here.

Fret not though, it is all still very much planned, albeit just a bit slow.

[balgerion]

Good morning , hope you are doing well in the new year. This project looks absolutely brilliant is there any chance you can merge /clean or point which repo to look at to create pr ? . I would gladly help here if project is not dead

[hologarden]

Good morning , hope you are doing well in the new year. This project looks absolutely brilliant is there any chance you can merge /clean or point which repo to look at to create pr ? . I would gladly help here if project is not dead

The setup is runnable, however with the fork of Prowlarr, it is impossible to connect Sonarr to any manga sources because it throws a [Warn] Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor: No output formatter was found for content types 'application/json' to write the response..

If you want to run it yourself, at least with Docker Compose, this is my .yaml:

services:
  sonarr:
    container_name: mangarr
    image: donderjoekel/sonarr-docker
    restart: unless-stopped
    ports:
      - "8989:8989" # Match Sonarr ports or otherwise change to prevent conflict
    environment:
      - PUID=1000
      - PGID=1000
      - UMASK=002
      - TZ=America/Toronto
    volumes:
      - ./config:/config
      - ./data:/data
      - ./downloads:/mnt/downloads
    networks:
        - default
  prowlarr:
    container_name: mangarr-prowlarr
    image: donderjoekel/prowlarr-docker
    restart: unless-stopped
    ports:
      - "9696:9696" # Match Prowlarr ports or otherwise change to prevent conflict
    environment:
      - PUID=1000
      - PGID=1000
      - UMASK=002
      - TZ=America/Toronto
    volumes:
      - ./prowlarr/config:/config
    networks:
        - default
networks:  
    default:

Prowlarr connects to manga sources no problem (with the exception of those replaced by WeebCentral), however Prowlarr will not connect to Sonarr. Do with this what you may.

Will add, that if @Thundernerd comes back to work on it, that the full logs for this problem can be found below (my ports differ than the compose .yaml above, and the API key has since been changed):

mangarr-prowlarr  | [Warn] Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor: No output formatter was found for content types 'application/json' to write the response.
mangarr           | [Warn] HttpClient: HTTP Error - Res: HTTP/1.1 [GET] http://mangarr-prowlarr:9696/api?t=caps&apikey=45f33dcd49754ff48bedbbbbc5ff82fd: 406.NotAcceptable (0 bytes)
mangarr-prowlarr  | [Warn] Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor: No output formatter was found for content types 'application/json' to write the response.
mangarr           | [Warn] HttpClient: HTTP Error - Res: HTTP/1.1 [GET] http://mangarr-prowlarr:9696/api?t=caps&apikey=45f33dcd49754ff48bedbbbbc5ff82fd: 406.NotAcceptable (0 bytes)
mangarr           | [Warn] Torznab: Unable to connect to indexer
mangarr           |
mangarr           | [v0.1.0.821] NzbDrone.Common.Http.HttpException: HTTP request failed: [406:NotAcceptable] [GET] at [http://mangarr-prowlarr:9696/api?t=caps&apikey=45f33dcd49754ff48bedbbbbc5ff82fd]
mangarr           |    at NzbDrone.Common.Http.HttpClient.ExecuteAsync(HttpRequest request) in ./Sonarr.Common/Http/HttpClient.cs:line 119
mangarr           |    at NzbDrone.Common.Http.HttpClient.Get(HttpRequest request) in ./Sonarr.Common/Http/HttpClient.cs:line 327
mangarr           |    at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.FetchCapabilities(NewznabSettings indexerSettings) in ./Sonarr.Core/Indexers/Newznab/NewznabCapabilitiesProvider.cs:line 58
mangarr           |    at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.<>c__DisplayClass4_0.<GetCapabilities>b__0() in ./Sonarr.Core/Indexers/Newznab/NewznabCapabilitiesProvider.cs:line 35
mangarr           |    at NzbDrone.Common.Cache.Cached`1.Get(String key, Func`1 function, Nullable`1 lifeTime) in ./Sonarr.Common/Cache/Cached.cs:line 99
mangarr           |    at NzbDrone.Core.Indexers.Newznab.NewznabCapabilitiesProvider.GetCapabilities(NewznabSettings indexerSettings) in ./Sonarr.Core/Indexers/Newznab/NewznabCapabilitiesProvider.cs:line 35
mangarr           |    at NzbDrone.Core.Indexers.Newznab.NewznabRequestGenerator.GetRecentRequests() in ./Sonarr.Core/Indexers/Newznab/NewznabRequestGenerator.cs:line 188
mangarr           |    at NzbDrone.Core.Indexers.HttpIndexerBase`1.TestConnection() in ./Sonarr.Core/Indexers/HttpIndexerBase.cs:line 376
mangarr           |
mangarr           |
mangarr           | [Warn] SonarrErrorPipeline: Invalid request Validation failed:
mangarr           |  -- : Unable to connect to indexer: HTTP request failed: [406:NotAcceptable] [GET] at [http://mangarr-prowlarr:9696/api?t=caps&apikey=45f33dcd49754ff48bedbbbbc5ff82fd]. Check the log surrounding this error for details

Aside from this issue, Mangarr (Sonarr) seems to be in a good enough state to function, but I wish I could do more testing without Prowlarr being a setback.

[dear-clouds]

Hello! Just found out about this project. I would also like to be able to contribute to the Sonarr fork if still worked on.

I don't know how far you are into that part, but wouldn't it be simpler to use sources already available for Mihon, instead of making a specific Prowlarr? Like https://github.com/keiyoushi/extensions-source or any other extension repo.

It would be amazing to also be able to download via torrent as the quality is far better than any website can provide. These are pretty simple to parse, for instead on Nyaa:

Programmed for Heartbreak: Sartain in Love 001-011 (2024-2025) (Digital) (Oak) => Manga chapters
Too Many Losing Heroines! v01-03 [Seven Seas] [Stick] => Light Novel volumes
Villains Are Destined to Die 001-162 {2020-2024} {Digital} {1r0n} => Webtoon chapters

As you can see, manga use (), light novels use [] and webtoons use {}, release names is almost always at the end. Of course there are exceptions like [Unpaid Ferryman] Elden Ring: The Road to the Erdtree v01-05 (2023-2025) (Digital) (LuCaZ), but this would be fairly easy to filter on the user end with Custom Formats.

[alexandru0-dev]

wouldn't it be simpler to use sources already available for Mihon, instead of making a specific Prowlarr? Like https://github.com/keiyoushi/extensions-source or any other extension repo.

Unfortunately Mihon/Tachiyomi extensions are written in Kotlin and as of now it's built only for Android. Idk how feasible it is to change the entire codebase to use kotlin multiplatform and then have fun gluing kotlin compiled binaries with C#.

Another thing that I was looking into was actually based on an open PR on Lidarr, which allowed to have a plugin system
But idk how easy the port would be

[dear-clouds]

Unfortunately Mihon/Tachiyomi extensions are written in Kotlin and as of now it's built only for Android. Idk how feasible it is to change the entire codebase to use kotlin multiplatform and then have fun gluing kotlin compiled binaries with C#.

I think this could be feasible with a Middleware API Proxy, this would allow us to only work on a middleware server that would automatically be compatible with all Mihon repos. Then it could simply be added as an "Indexer" in the Sonarr fork. It would be less work to maintain it this way as the Mihon repos are always actively worked on.

[Thundernerd]

This project is currently still a bit in the freezer unfortunately. However, I have been somewhat off-and-on working on a Jackett-like middleware that would support plugins. Initially I was focusing on having just C# plugins but I would like to explore supporting different languages so that we can leverage existing scrapers out there (like those for Mihon/Tachiyomi, but also those for Paperback).

Once I've figured out a proper way to support all this, and have it set up and running I can start working on this project again. I'm not sure if I'll completely build off of Sonarr as it comes with a lot of extra things that aren't quite necessary for this, but that's something to determine later on.

All that said, thanks for all the interest in here :)

[dear-clouds]

Thanks for updating us ❤️

Yes, it would be great to have a Prowlarr/Jackett-like to handle this, this would add so many more options and simplify future updates! I also like the "Stats" view from Prowlarr, so users can see what sources they use the most to download, etc.

Anyway, let us know whenever you want us to help with either the Sonarr fork or middleware :)

[Thundernerd]

I had a quick look during my lunch break and I think I might be able to use the Suwayomi (formerly Tachidesk) to enable a lot of extensions. I'll try to get some time into this over the weekend to see if I can write some middleware that can interface with this and pass along the appropriate data, but from what I could see it is looking good!

[alexandru0-dev]

@Thundernerd the Suwayomi api it's graphql so it should be pretty straight forward

[dear-clouds]

I had a quick look during my lunch break and I think I might be able to use the Suwayomi (formerly Tachidesk) to enable a lot of extensions. I'll try to get some time into this over the weekend to see if I can write some middleware that can interface with this and pass along the appropriate data, but from what I could see it is looking good!

Awesome news! Excited to see how see how it goes 🎉

[Thundernerd]

Was really hoping I would be able to use Prowlarr as a base for the middleware but it seems that (most if not all of) the indexers are predefined. Does someone here have experience making a "dynamic" indexer? Otherwise I might use some parts from the core C# stuff and have to roll my own thing on top of that, but would be nice if I don't have to

[alexandru0-dev]

@Thundernerd
The only "dynamic" indexer example that I can think of is using a plugin system like on Lidarr
Lidarr/Lidarr#1829

And then having plugins like:
https://github.com/ta264/Lidarr.Plugin.Deemix
And
https://github.com/TrevTV/Lidarr.Plugin.Tidal

So you can have the Suwayomi plugin and then just update the plugin

But idk how easy it would be porting the pr

[Thundernerd]

@Thundernerd The only "dynamic" indexer example that I can think of is using a plugin system like on Lidarr Lidarr/Lidarr#1829

And then having plugins like: ta264/Lidarr.Plugin.Deemix And TrevTV/Lidarr.Plugin.Tidal

So you can have the Suwayomi plugin and then just update the plugin

But idk how easy it would be porting the pr

Yeah I think I might still be able to use Prowlarr, at least I want to properly try to use it. I might have to make an IndexerProvider system where you can make your own thing that wraps around an external service like Suwayomi. Shouldn't be too hard, though those are usually famous last words :D