Drop problem

[dkdirekt]

Description

Hello,

I have a problem with drop in ChannelPointMiner, I have 3 streamers files in streamers folder.

In my config.json, in "defaultStreamerSettings" section, I have this:

"enabled":true,
"index":1,
"participateCampaigns":true,
"priorities": 
[
   {
     "type":"constant",
      "score":0,
   },
   {
      "type":"drops",
      "score":500,
   },
],

for streamer #1 and #2, i have these files:

{
    "priorities": 
    [
        {
            "type":"constant",
            "score":100,
        },
    ],
}

for the streamer #3, I have this file:

{
    "priorities": 
    [
        {
            "type":"constant",
            "score":50,
        },
    ],
}

The streamer #3 has enabled drops on his stream and is online currently... But after 30 minutes, this streamer is not picked by ChannelPointMiner for process drops. However, I set for "defaultStreamerSettings" in config.json the priority for drops to 500, so, ChannelPointMiner should take the streamer #3 in priority despite having a "constant" lower score, or I did something wrong?

Sorry for my bad english, i'm french.

Version / commit

2.2.8 - 56473f0 - HEAD

Relevant log output

No response

[Rakambda]

If I remember well, what you define in the default settings is overriden by what you set in the streamer settings.
So basically when you define a "priorities" for a streamer the "priorities" from the default are lost.

So if you override them in a streamer, you have to re-set the default ones.

I can get that this is annoying but I don't really have a good way to handle the "they stack up".
Because what if you want to actually redefine them all? Then you'd be stuck.

Now from what I understand there you want to apply the same rule for them all but give them an order of priority.
You could actually do it this way:

Default

{
  "enabled":true,
  "participateCampaigns":true,
  "index": 999,
  "priorities": 
  [
     {
        "type":"drops",
        "score":500,
     }
  ]
}

Streamer1/2:

{
    "index": 1
}

Streamer3:

{
    "index": 2
}

This way all streamer have the same priority rule, being that drops gives a boost in "ranking points", but if there's an exaequao (so several drops are available), then the index is used. Streamer 1/2 will therefore be picked before Streamer3. However if Streamer1/2 doesn't have drops, then their "ranking" will have 0 points and Streamer3 will be picked in priority.
(I guess you get the idea just didn't get that the priority is overriden instead of stacked)

[dkdirekt]

I see, I did not know that was overidding all default streamer priorities if I just set one priority in streamer file, good to know.

I just set "priority" by index in streamers file like you show in your example.

So, if I understand right, the priority is like this: Score (Higher) > Index (Lower) > Random?

Thank's.

[Rakambda]

Yeah, I don't think I really mentioned it anywhere in the doc. Will have to add it.

You're right for the priority. Will have to document too I guess 😉

[dkdirekt]

Yep, well, I just try for 1 hour, and it's seems to not work properly... I have now 4 streamers files with index 1, and one streamer file with index 2 (the streamer that have drops activated), so a total of 5 streamers files.

So, to summarize, streamers with index 1 are simple streamers and the streamer with index 2 is the streamer that has enabled drops (because I just need drops from this streamer, points only when there is no other streamers online).

I just replace my default streamer settings with your default streamer settings in my config.json file.

All streamers are online.

When I start ChannePointMiner, 2 streamers are picked with index 1, but the streamer with index 2 is not picked by ChannelPointMiner (however it's the streamer with highest index but higher score because he have drops).
When a streamer with index 1 goes offline, it pick for a couples of minutes the streamer with index 2, but after a couple of minutes, it retake a streamer with index 1 in place of streamer with index 2.

Randomly, sometime, it take the streamer with index 2 but the drop percentage remains unchanged, like the log below.

2023-10-04T00:11:44,886 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Points earned [+10 | WATCH | 221.26K]
2023-10-04T00:12:50,443 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Points earned [+10 | WATCH | 1.23K]
2023-10-04T00:13:48,155 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:14:48,883 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:15:49,975 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:15:50,889 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Points earned [+10 | WATCH | 221.27K]
2023-10-04T00:15:50,896 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Claim available
2023-10-04T00:15:51,127 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Points earned [+50 | CLAIM | 221.32K]
2023-10-04T00:16:50,849 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:17:52,429 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:18:20,329 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_2} - Points earned [+10 | WATCH | 113.31K]
2023-10-04T00:18:53,003 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:19:54,265 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:20:54,956 [INFO] {MY_ACCOUNT_NAME} : {STREAMER_WITH_DROPS} - Drop progress on channel {STREAMER_WITH_DROPS} : 40%
2023-10-04T00:22:56,584 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Points earned [+10 | WATCH | 221.33K]
2023-10-04T00:25:00,264 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_2} - Points earned [+10 | WATCH | 113.32K]
2023-10-04T00:28:02,801 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_1} - Points earned [+10 | WATCH | 221.34K]
2023-10-04T00:30:05,640 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_2} - Points earned [+10 | WATCH | 113.33K]
2023-10-04T00:30:05,642 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_2} - Claim available
2023-10-04T00:30:05,870 [INFO] {MY_ACCOUNT_NAME} : {OTHER_STREAMER_2} - Points earned [+50 | CLAIM | 113.38K]

Did you know why? Thank's.

*Edited because missclick on editing my and streamers informations, sry.

[couchoud-t]

Hmm seems weird. Maybe I'll try to add a log to figure out a bit how it's computing the ordering.

[dkdirekt]

Yep, i have added an account (so 2 accounts on ChannelPointMiner) with the same default streamer configuration and streamers files configurations...

On the first account, when the streamer with drops enabled start his stream, it switch on it, but seems to not process the drop because the percentage keep at 21% and after ~7 minutes, it switching on another streamer. as we can see on this screenshot: https://i.imgur.com/FITYvZ6.png

On the second account, when the streamer with drops enabled start his stream, it switch on it, but seems to not process the drop either because the percentage keep at 55% and is not switching on another streamer because only 2 streamers are online as we can see on this screenshot: https://i.imgur.com/DJ9TKum.png

So, for me, CPM seems to not process drops anymore, and seems to (maybe) ignore the drops score when drops are available on streamer (maybe linked).

Thank's.

[Rakambda]

Honestly I don't really manage to find what's going on.

If you can reproduce it consistently, could you provide some logs in trace mode ?
I don't really know how you start the miner, but i'll assume from the jar for now.
For it just start it like java -Dlog4j.configurationFile=log4j2.xml -jar miner/build/libs/miner-shaded.jar --settings config.json with log4j2.xml having this content :

<?xml version="1.0" encoding="UTF-8"?>
<Configuration name="Default" status="warn">
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout disableAnsi="false" pattern="%highlight{%date{HH:mm:ss,SSS} %level [%thread] %logger{1.} %X - %message}%n"/>
        </Console>
        <RollingFile
                name="file"
                fileName="logs/${project.name}.log"
                filePattern="logs/${project.name}.log.%d{yyyy-MM-dd}.gz"
                ignoreExceptions="false">
            <PatternLayout>
                <Pattern>%date{ISO8601} %level [%thread] %logger{1.} %X - %message%n</Pattern>
            </PatternLayout>
            <Policies>
                <TimeBasedTriggeringPolicy interval="1"/>
            </Policies>
            <DefaultRolloverStrategy max="10"/>
        </RollingFile>
    </Appenders>
    <Loggers>
        <Logger level="trace" name="fr.rakambda.channelpointsminer.miner" additivity="false">
            <AppenderRef ref="console"/>
            <AppenderRef ref="file"/>
        </Logger>
        <Root level="warn" additivity="false">
            <AppenderRef ref="console"/>
            <AppenderRef ref="file"/>
        </Root>
    </Loggers>
</Configuration>

The log file may grow big rather fast so I guess would be betting to start it this way when you know you can reproduce.
Another things, logs may contain logs of sensitive data like ids etc, so I guess it'll be safer if you don't post it directly here and DM me it instead (mail, or discord, or we can figure something out).

[dkdirekt]

Yes I start the miner with this logger, but yeah the log file become very big in a short time...

I was able to see yesterday during testing, the score for "drops" is not working, where "watchStreak" works fine. It looks like ChannelPointMiner doesn't detect drop are available on channels, so the streamer score is not increased (unlike watchStreak works fine for example). I think I can't monitor this problem via logfile, am i right? And maybe you can reproduce this with adding a streamer file who have drops enabled with index like 999 (the last so) and 2 other at least online streamers without drops enabled with less index.

I'm removing all streamers except the streamers that have drops enabled, so, ChannelPointMiner is mining it, but, like before, drop progress is blocked on 21%, I have a log file of ~150kb for ~10 minutes but I can't upload it here because there is a lot a sensitive information in it.

To summarize, ChannelPointMiner seems to not detect drops on streamers channel, so the score in config.json seems to not be increased to streamer file, so, the streamer with drops enabled seems not to be a priority. And in all cases, when ChannelPointMiner mining any streamer that have drops enabled, it seems to be not progress the drop.

How can we do for log file?

[Rakambda]

If you at least find that it is because drops doesn't work that's an help.
Do you perhaps have channels names where this happens reliably ? This way I can just try on my side.

The only way i see this not working is if the drops are setup after the stream starts (as CPM checks only when the stream starts IIRC).

[dkdirekt]

Yes I have a channel name where it's not working: humility (enabled drops for Dofus).

[Rakambda]

How do you run the miner? From compiling or a pre-build method?

[dkdirekt]

I'm on ArchLinux Up-To-Date (Kernel 6.1.55-1-lts)
Java informations:
openjdk 17.0.8.1 2023-08-24
OpenJDK Runtime Environment Temurin-17.0.8.1+1 (build 17.0.8.1+1)
OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (build 17.0.8.1+1, mixed mode, sharing)

I'm running miner from release pre-builded (2.2.8 | 56473f0 | HEAD) on tmux with this command:
java -Dlog4j.configurationFile=logger_verbose.xml -jar miner-shaded.jar --settings config.json

[Rakambda]

If you use this jar, does this fix your issue ?
https://www.dropbox.com/t/cNJg4Bh3jiNK30BR

[dkdirekt]

Yep it's seems to work fine with this.

With 15 streamers and humility with higher index, it's seems to pick humility with success and the drop is processed, the percentage is increasing.

[dkdirekt]

I'm waiting the 100% to confirm it's working fine.

EDIT: Missclick on close issue.

[Rakambda]

You can leave it open, I'll let you run it for a bit longer to see if it holds up.
If it does I'll merge the fix.

Basically CPM was expecting a "Drops" tag on the stream but it seems it doesn't always put it anymore.

[dkdirekt]

After somes hours, it's seems to working well.

I noticed other thing, when CPM is switching to highest score streamer (the streamer with drops enabled), the second mined streamer seems to be taken randomly and not the one with less index or highest score (because the score is same for all streamers excepted for streamer with drops enabled). Hard to explain in english sorry, btw it's not a big problem since when the drops are claimed it's working perfectly anyway.

It was only for notice this 😁

[Rakambda]

Hmm, not sure to get it sorry.
You're French right? If so you can explain it in French.

Though it's already a good thing that it seems to fix the drops detection issue.

[dkdirekt]

Hey, sorry I was at hospital for 2 weeks. Anyway...

Yes I can explain it in french:

En gros, dans ma configuration j'ai un streamer avec l'index 0 et 3 autres avec l'index 1. Je veux constamment que le streamer avec l'index 0 soit miné. Lorsqu'un streamer dans la liste des streamers dans CPM à des drops activés, CPM augmente son score jusqu'à (dans ma configuration) être le plus haut en score et donc être le premier streamer à être miné (devant celui avec l'index 0). En toute logique, le deuxième streamer qui devrait être miné devrait donc être celui avec l'index 0, mais ce n'est pas le cas et c'est un des autres streamer en ligne avec l'index 1 qui est miné jusqu'à ce que le drop soit terminé, ensuite l'ordre refonctionne. Mais ce n'est pas un problème en sois car les drops en général ne durent pas bien longtemps, c'était juste pour remonter l'information.

Cependant je remarque encore un problème cet après midi, il y a actuellement des drops actif sur le jeu Sea of Thieves, et j'ai bel et bien un streamer en ligne avec les drops activés dans ma liste de streamers sur CPM, mais les drop ne semblent pas progresser. Par contre lorsque je regarde le stream du dit streamer, la progression monte (dans l'inventaire twitch) mais n'indique rien dans les logs de CPM.

J'ai pu voir sur mon smartphone hier à l'hôpital que certaines personnes avaient déjà des problèmes liés aux drop sur ce même streamer hier, mais il semble surtout que Twitch ai apporté (encore) des modifications aux drops car hier beaucoup de personnes avaient des bugs qui empêchaient la progression de monter en regardant un streamer avec les drops activés, certains avaient une progression qui montait et d'autres non, personnellement hier en regardant le streamer, sur twitch, ma progression ne montait pas, alors qu'aujourd'hui oui.

Je pense que c'était plus simple d'expliquer le "problème" en Français car mon anglais n'étant pas très approfondi la compréhension peut ne pas être très bonne.

EDIT: Après avoir regardé le streamer qui à les drops activés sur twitch, CPM m'à bien indiqué avoir récupéré le drop mais n'à en aucun cas affiché sa progression et n'à pas miné le drop tant que je ne regardais pas moi même le streamer sur Twitch:

2023-10-21T14:19:32,839 [INFO] my_account :  - Drop available [Ancestral Hull]
2023-10-21T14:19:33,054 [INFO] my_account :  - Drop claimed [Ancestral Hull]

[Rakambda]

Pour le premier point sur les priorités, si je donne un cas :

streamer1 : index=0 & score = 50
streamer2 : index=1 & score = 100

Peu importe la valeur des score mais l'idée c'est le score du 2 est plus grand. Dans ce cas tu t'attends à ce que ce soit le streamer1 qui est choisi car il a un index plus faible. C'est bien ça?

Si oui: C'est voulu, l'index n'est utilisé que quand un score est égal afin de les départager. Si tu veux que le streamer1 soit toujours premier, donne lui une priority constante avec une valeur super élevée. Comme ça son score sera toujours le plus grand.

---

Pour les Drops qui ne bougent pas, je pense que c'est ce qui a été remonté avec #620 , surement twitch qui a modifié des choses de leur coté (y'a eu une PR sur le repo en python pour addresser ça a priori).

Je n'ai pas eu l'occasion de tester pour le moment, mais est-ce que cette version corrige le proglème de drops ? : https://www.dropbox.com/t/IypFESct3ix9odZ3

[dkdirekt]

Notre raisonnement est bon, on ne s'est juste pas compris je pense. En fait ma configuration veut que le score par défaut soit de 0 sauf pour les drops où le score est de 1. Je choisis la priorité à travers l'index, je veux que les streamers avec l'index le plus bas soient minés en permanence sauf quand il y a des drops à miner, ils peuvent être temporairement remplacés si besoin il y a. Dans ce cas il n'y a aucun problème, mais je vais donner un exemple concret avec 4 streamers (qui seraient tous en ligne), imaginons:

Streamer 1: index=1 & score=0
Streamer 2: index=1 & score=0
Streamer 3: index=1 & score=0
Streamer 4: index=0 & score=0

Schématisé: Streamer 4 > [Streamer 1, Streamer 2, Streamer 3]

En temps normal le streamer 4 sera toujours miné, avec en parallèle un des 3 autres (ce qui est normal, ils ont le même score de 0 et le même index). Imaginons maintenant que le streamer 3 à des drops à miner, son score passe à 1, il devient donc prioritaire. En théorie, il passe donc logiquement en 1ère position devant le streamer 4 qui lui passe en deuxième position (streamer 3 à le meilleur score et streamer 4 à le meilleur index).

On se retrouve donc dans cette configuration:

Streamer 1: index=1 & score=0
Streamer 2: index=1 & score=0
Streamer 3: index=1 & score=1
Streamer 4: index=0 & score=0

Schématisé: Streamer 3 > Streamer 4 > [Streamer 1, Streamer 2]

Sauf qu'en pratique, le streamer 4 est remplacé par le streamer 3 (ce qui est normal jusqu'ici), sauf que le streamer 4 qu'on pourrait s'attendre à être miné en parallèle est remplacé par streamer 1 ou 2. C'est le constat que j'ai pu faire, ou c'est peut être moi qui n'ai pas bien compris le système de priorité. C'est long et pas très évident à expliquer, mais le problème même s'il est réellement présent n'est pas un gros problème en sois. On remarque aussi la même chose lorsqu'on regarde un streamer qui n'est pas présent dans la liste (qui est remplacé par UnknownStreamer), ce qui doit être difficile/impossible à gérer, mais qui n'est pas réellement un problème comme je l'ai expliqué, c'était plus pour remonter l'information.

Pour le problème de visionnage de drops, je vais tester cette version et je tiendrais au courant ici même

Merci.

[dkdirekt]

Je reviens aux nouvelles:

La version fournie sur Dropbox semble fonctionner, c'est à dire que:

• Le score est bien incrémenté au streamer avec les drops.
• La progression augmente bien pour les drops dans l'inventaire Twitch.

Par contre, la progression ne s'affiche plus dans la console ni dans les logs discord (webhook) comme c'était le cas auparavant. Mais dans l'ensemble ça semble bien fonctionner.

EDIT: Lorsque la progression du drop est terminée, une erreur se déclenche et le drop n'est pas claim.

Voici l'erreur en question (je peux fournir l'erreur intégrale à la demande s'il le faut mais elle est très longue):

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "count_by_md" (class fr.rakambda.channelpointsminer.miner.api.ws.data.message.subtype.Summary), not marked as ignorable (3 known properties: "last_read_all", "last_seen", "count"])

[Rakambda]

Plutot cool que ça avance.
J'ai essayé avec un drop et il me l'a bien claim. L'erreur que tu as remonté n'est pas liée aux drops. (c'est a priori un truc plus utilisé maintenant)

[Rakambda]

Cette version devrai corriger l'erreur : https://www.dropbox.com/t/CpFTZdFKGleGLOba

[dkdirekt]

Effectivement le drop à bien été claim un peu plus tard, comme ce n'était pas instantané j'ai pensé qu'il ne l'avait pas claim, mea culpa.

J'upload cette version, merci.