Email stuck in /var/spool/slurm-mail

[lurch999]

Centos7 machine, email going out fine from localhost via sendmail and via Slurm via previous config, but after adding the slurm-email scripts and files, and restarting slurmctld, the emails are still stuck in /var/spool/slurm-mail. Don't see any errors in slurm-spool-mail.log

Suggestions? Would love to make this work!

[neilmunday]

Hi @lurch999,

Under /var/log/slurm-mail you should find Slurm-Mail's logs (unless you've changed the location in slurm-mail.conf).

Have you created the cron job to run slurm-send-mail.py periodically? If not, check the README and create the cron job as described.

If there are no logs here but you are sure the cron job is running, then you can try running slurm-send-mail.py from the command line. If there are errors produced at the command line let me know and we'll see what the problem is.

Cheers,

Neil.

[lurch999]

I see, thanks, that helped! Didn't realize the cron job was required. The cron job that was needed for me is:

* * * * * root /bin/python3 /opt/slurm-mail/bin/slurm-send-mail.py

Almost there: emails going out are being sent to [email protected] when they should be going to [email protected]. Where can I tweak that? I would have thought these would go to same address in Slurm wrapper script? Thanks for your help!

[lurch999]

Thanks Neil. The test you suggested shows the servername is appended to the recipient email so I'll need to figure out how to fix that. The tweak to the slurm-send-mail.py you suggestion didn't work though, but I think I'm on track to figure this out somehow now!

[lurch999]

Got it working now, thanks! Had to make minor adjustment to a Postfix file. Working like a charm!

Question: where is your script getting the value of $CLUSTER from? That value is not being added into the email sent out, so I'd like to fix that, as I see it in various .tpl files

[neilmunday]

Glad to hear that you've got Postfix working.

The value of CLUSTER is coming from the output of this command:

sacct -j JOB_ID -p -n --fields=JobId,Partition,JobName,Start,End,State,nnodes,WorkDir,Elapsed,ExitCode,Comment,Cluster,User,NodeList,TimeLimit,TimelimitRaw,JobIdRaw

The 12th field returned is the name of your cluster. What do you get for a job?

You can also see the name of your cluster(s) like so:

sacctmgr show cluster

[lurch999]

Ah, that makes sense. Of course, your script is working exactly as advertised. Now I know what to fix. Thanks again! Consider this resolved.

[neilmunday]

Excellent! Glad you got it all sorted.