Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

contact based routing only accepts 1 email address - same as #55 #58

Open
njomeara opened this issue Sep 11, 2017 · 15 comments
Open

contact based routing only accepts 1 email address - same as #55 #58

njomeara opened this issue Sep 11, 2017 · 15 comments

Comments

@njomeara
Copy link

njomeara commented Sep 11, 2017

Hi,

I have the same problem as reported in ..
contact based routing only accepts 1 email address #55

The email contacts are configured in the check. Emails only get sent to the default address configured in the mymail_handler.json file and not to the contacts configured in the support.json file.

If i only put one address in "contact" field in the check it will send to that address, adding an array of addresses doesn't work.

mymail_handler.json

{
    "mailer" :  {
        "mail_from": "sensu@ieatoskca-sensu3",
        "mail_to": ["[email protected]"],
        "delivery_method": "smtp",
        "smtp_address": "smtp.internal.company.com",
        "smtp_port": "25",
        "smtp_domain": "company.com"
    },
    "handlers": {
        "mailer": {
            "type": "pipe",
            "command": "/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html",
         "filter": "state_change_only",
	          "severities": ["ok","warning","critical","unknown"],
            "handle_silenced": true,
            "handle_flapping": true
	    
        }
    }
}

support.json

{
  "contacts": {
    "support": {
      "email": {
        "to": "[email protected]"
      }
    },
    "team1": {
      "email": {
        "to": "[email protected]"
      }
    },
    "team2": {
      "email": {
        "to": "[email protected]"
      }
    }
  }
}

check_test.json

{
    "checks": {
        "test_ignore": {
            "command": "exit 2",
            "standalone": true,
            "interval": 30,
            "handlers": ["mailer"],
            "contact": [
		"support",
		"team2"
	    ]
         }
     }
}

I turned on debug and attached the output of:

grep -i test_ignore /var/log/sensu/sensu-server.log
[debug.txt](https://github.com/sensu-plugins/sensu-plugins-mailer/files/1291942/debug.txt)

Sensu was running on version 0.29. I updated to latest as below with same result...

[root@ieatoskca-sensu3 tmp]# gem list | grep sensu
sensu (1.0.3)
sensu-cli (0.8.0)
sensu-extension (1.5.1)
sensu-extensions (1.9.0)
sensu-extensions-check-dependencies (1.0.1)
sensu-extensions-debug (1.0.0)
sensu-extensions-json (1.0.0)
sensu-extensions-occurrences (1.2.0)
sensu-extensions-only-check-output (1.0.0)
sensu-extensions-ruby-hash (1.0.0)
sensu-json (2.1.0)
sensu-logger (1.2.1)
sensu-plugin (2.3.0, 1.4.5)
sensu-plugins-cpu-checks (1.1.2)
sensu-plugins-disk-checks (2.5.0)
sensu-plugins-logstash (1.1.1)
sensu-plugins-mailer (1.2.0)
sensu-plugins-memory-checks (3.0.2)
sensu-plugins-network-checks (2.0.1)
sensu-plugins-ntp (1.0.1)
sensu-redis (2.2.0)
sensu-settings (10.3.0)
sensu-spawn (2.2.1)
sensu-transport (7.0.2)

Regards,

Noel

@majormoses
Copy link
Member

@stevenviola another report of this, if we can't get this working we will have to revert the change.

@stevenviola
Copy link
Contributor

Hey @njomeara, in check_test.json, could you try changing contact to contacts? Are you only seeing the issue when adding contacts to checks, but when having contacts listed for clients, it works, or is the same issue happening for both checks and clients.

@majormoses
Copy link
Member

I don't honestly see the value in having contact and contacts I think we should combine them in a breaking change (major versioned) to avoid confusion in the future. You can always still have a single contact in an array.

@stevenviola
Copy link
Contributor

@majormoses I was emulating the syntax in the sensu reference guide for checks and clients. Both contact and contacts is listed. Now that I reread that, I think the issue is, contact is singular (taking one contact), and that like the logic I ported over. @njomeara is passing an array of contacts so the contacts value should be used.

If we want to keep the compatibility with Sensu Enterprise, I could document the difference between contacts and contact more clearly to avoid confusion.

@majormoses
Copy link
Member

majormoses commented Oct 7, 2017

Ya it makes total sense to me and I feel the same way about that with handler vs handlers in check definitions. All it does is add confusion. I think we could catch this scenario and throw an unknown when passing a single one to it. Thoughts?

@majormoses
Copy link
Member

@stevenviola I just opened a PR to help users with this type of misconfiguration as it is a fairly easy to trap. Once it has been reviewed, merged, and released we can have the user test and see if it catches their scenario and tells them how to update it. If that goes well and this works as expected hopefully we can put this to rest.

@majormoses
Copy link
Member

@njomeara can you please try using 2.0.0 which I just released to attempt to catch this misconfiguration?

@njomeara
Copy link
Author

@majormoses @stevenviola, I updated the sensu-plugins-mailer to 2.0.0 and tried again with this config in the check ...
"contact": [
"team1",
"team2"
]
Again it didn't work. I changed to "contacts" in the check:
"contacts": [
"team1",
"team2"
]
and now it works as expected. I tried both "contact" and "contacts" before upgrading the gem and it still didn't work so the new 2.0.0. version has fixed the issue. I agree that you should be able to use contact and contacts interchangeably, difficulty one to spot when you leave the "s" of contacts.

Thank you for your support

@majormoses
Copy link
Member

@njomeara did you see anything in your server logs that would have indicated that there was an issue? The goal was to make it easier to spot the issue rather than merge the two. I added this: https://github.com/sensu-plugins/sensu-plugins-mailer/blob/2.0.0/bin/handler-mailer.rb#L148-L153 we should probably add a "Gotcha" in the readme to warn users about this behavior.

@njomeara
Copy link
Author

njomeara commented Nov 3, 2017

@majormoses. Debug logs below. First run with "contact" and second run with "contacts". don't see anything in server.log (api.log had nothing either). Note the send run with "contacts" sent to two email addresses but the first run with "contact" only sent to one address

[root@ieatoskca-sensu3 scripts]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contact":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-03T16:36:10.492399+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970}}}
{"timestamp":"2017-11-03T16:36:10.492487+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970}}}
{"timestamp":"2017-11-03T16:36:10.492856+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970,"type":"standard"}}
{"timestamp":"2017-11-03T16:36:10.494016+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509726954},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contact":["support","support"],"executed":1509726970,"issued":1509726970,"type":"standard","history":["1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509726970,"id":"2044e967-1809-4684-a54b-bfff721626ed","last_ok":1509726970,"last_state_change":1509726970,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-03T16:36:10.494378+0000","level":"debug","message":"handling event","event_data":"{\"client\":{\"name\":\"ieatoskca-sensu3\",\"address\":\"xxx.xxx.xxx.xxx\",\"environment\":\"production\",\"mail_to\":[\" [email protected] \"],\"subscriptions\":[\"rabbitmq_servers\",\"redis_servers\",\"sensu_masters\",\"client:ieatoskca-sensu3\"],\"version\":\"1.1.1\",\"timestamp\":1509726954},\"check\":{\"name\":\"dodgy_check\",\"output\":\"testing email alerts\",\"status\":1,\"handler\":\"mailer\",\"contact\":[\"support\",\"support\"],\"executed\":1509726970,\"issued\":1509726970,\"type\":\"standard\",\"history\":[\"1\",\"0\",\"1\"],\"total_state_change\":0},\"occurrences\":1,\"occurrences_watermark\":1,\"action\":\"create\",\"timestamp\":1509726970,\"id\":\"2044e967-1809-4684-a54b-bfff721626ed\",\"last_ok\":1509726970,\"last_state_change\":1509726970,\"silenced\":false,\"silenced_by\":[]}","event":{"id":"2044e967-1809-4684-a54b-bfff721626ed"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-03T16:36:11.286134+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"2044e967-1809-4684-a54b-bfff721626ed"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to  [email protected] \n"]}
[root@ieatoskca-sensu3 scripts]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contacts":["support","support"] }' > /dev/tcp/localhost/3030
[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-03T16:39:10.513673+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150}}}
{"timestamp":"2017-11-03T16:39:10.513844+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150}}}
{"timestamp":"2017-11-03T16:39:10.514320+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150,"type":"standard"}}
{"timestamp":"2017-11-03T16:39:10.515456+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509727134},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509727150,"issued":1509727150,"type":"standard","history":["1","0","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509727150,"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e","last_ok":1509727150,"last_state_change":1509727150,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-03T16:39:10.515817+0000","level":"debug","message":"handling event","event_data":"{\"client\":{\"name\":\"ieatoskca-sensu3\",\"address\":\"xxx.xxx.xxx.xxx\",\"environment\":\"production\",\"mail_to\":[\" [email protected] \"],\"subscriptions\":[\"rabbitmq_servers\",\"redis_servers\",\"sensu_masters\",\"client:ieatoskca-sensu3\"],\"version\":\"1.1.1\",\"timestamp\":1509727134},\"check\":{\"name\":\"dodgy_check\",\"output\":\"testing email alerts\",\"status\":1,\"handler\":\"mailer\",\"contacts\":[\"support\",\"support\"],\"executed\":1509727150,\"issued\":1509727150,\"type\":\"standard\",\"history\":[\"1\",\"0\",\"1\",\"1\",\"0\",\"1\"],\"total_state_change\":0},\"occurrences\":1,\"occurrences_watermark\":1,\"action\":\"create\",\"timestamp\":1509727150,\"id\":\"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e\",\"last_ok\":1509727150,\"last_state_change\":1509727150,\"silenced\":false,\"silenced_by\":[]}","event":{"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-03T16:39:11.487711+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"d6e32dc3-01a2-4fa7-8d79-63c9d64d223e"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to  [email protected] , [email protected]\n"]}
[root@ieatoskca-sensu3 scripts]# /opt/sensu/embedded/bin/gem list | grep mailer
sensu-plugins-mailer (2.0.0)
[root@ieatoskca-sensu3 scripts]# rpm -qa | grep sensu
sensu-1.1.1-1.el7.x86_64

@majormoses
Copy link
Member

majormoses commented Nov 3, 2017

So the second one is working just fine, what is confusing me here is that you are not triggering this: https://github.com/sensu-plugins/sensu-plugins-mailer/blob/2.0.0/bin/handler-mailer.rb#L148-L153 on the first one.

@majormoses
Copy link
Member

OK I think I figured it out thanks to @cwjohnston who helped me realize whats going on. As you are specifying it from the check as contact but in your config you have contacts the most reliable check that we can add is to inspect the event data as well to determine if you are overwriting the config in the event. I will try to get that done this weekend.

@majormoses
Copy link
Member

@njomeara can you try pulling that handler in and seeing if we can trigger something to help user to properly configure.

@njomeara
Copy link
Author

njomeara commented Nov 6, 2017

Now it looks like mail is not sent at all when "contact" is used

[root@ieatoskca-sensu3 bin]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 0, "handler": "mailer","contact":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-06T12:39:00.350826+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940}}}
{"timestamp":"2017-11-06T12:39:00.351002+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940}}}
{"timestamp":"2017-11-06T12:39:00.351511+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard"}}
{"timestamp":"2017-11-06T12:39:00.352529+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971936},"check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0"],"total_state_change":0},"occurrences":3,"occurrences_watermark":3,"action":"resolve","timestamp":1509971940,"id":"c86052cb-3b4d-4f04-9237-25610f6fba84","last_state_change":1509971940,"last_ok":1509971940,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-06T12:39:00.352967+0000","level":"debug","message":"handling event","event_data":"{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971936},"check":{"name":"dodgy_check","output":"testing email alerts","status":0,"handler":"mailer","contact":["support","support"],"executed":1509971940,"issued":1509971940,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0"],"total_state_change":0},"occurrences":3,"occurrences_watermark":3,"action":"resolve","timestamp":1509971940,"id":"c86052cb-3b4d-4f04-9237-25610f6fba84","last_state_change":1509971940,"last_ok":1509971940,"silenced":false,"silenced_by":[]}","event":{"id":"c86052cb-3b4d-4f04-9237-25610f6fba84"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}

Changing to "contacts"

[root@ieatoskca-sensu3 bin]# echo '{ "name": "dodgy_check", "output": "testing email alerts", "status": 1, "handler": "mailer","contacts":["support","support"] }' > /dev/tcp/localhost/3030

[root@ieatoskca-sensu3 ~]# tail -f /var/log/sensu/sensu-server.log | grep dodgy
{"timestamp":"2017-11-06T12:40:08.647702+0000","level":"debug","message":"received result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008}}}
{"timestamp":"2017-11-06T12:40:08.647818+0000","level":"debug","message":"processing result","result":{"client":"ieatoskca-sensu3","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008}}}
{"timestamp":"2017-11-06T12:40:08.648562+0000","level":"debug","message":"storing check result","check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard"}}
{"timestamp":"2017-11-06T12:40:08.649511+0000","level":"info","message":"processing event","event":{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971996},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509972008,"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d","last_ok":1509972008,"last_state_change":1509972008,"silenced":false,"silenced_by":[]}}
{"timestamp":"2017-11-06T12:40:08.649858+0000","level":"debug","message":"handling event","event_data":"{"client":{"name":"ieatoskca-sensu3","address":"xxx.xxx.xxx.xxx","environment":"production","mail_to":[" [email protected] "],"subscriptions":["rabbitmq_servers","redis_servers","sensu_masters","client:ieatoskca-sensu3"],"version":"1.1.1","timestamp":1509971996},"check":{"name":"dodgy_check","output":"testing email alerts","status":1,"handler":"mailer","contacts":["support","support"],"executed":1509972008,"issued":1509972008,"type":"standard","history":["1","0","1","1","0","1","0","1","0","1","0","1","0","1","1","1","0","1"],"total_state_change":0},"occurrences":1,"occurrences_watermark":1,"action":"create","timestamp":1509972008,"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d","last_ok":1509972008,"last_state_change":1509972008,"silenced":false,"silenced_by":[]}","event":{"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d"},"handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"}}
{"timestamp":"2017-11-06T12:40:09.603993+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/handler-mailer.rb -t /etc/sensu/conf.d/html_template.erb -s 'Openstack Cloud 3A ' -c html","filter":"state_change_only","severities":["ok","warning","critical","unknown"],"handle_silenced":true,"handle_flapping":true,"name":"mailer"},"event":{"id":"d1d751b6-1b39-4806-be38-c66abedc9a6d"},"output":["mail -- sent alert for ieatoskca-sensu3/dodgy_check to [email protected] , [email protected]\n"]}

i checked my handler-mailer.rb below to make sure it had the code change ..

[root@ieatoskca-sensu3 bin]# grep src /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-mailer-2.0.0/bin/handler-mailer.rb
def raise_contact_is_array(src)
msg = "you passed #{src} which is meant to be a string, "

@majormoses
Copy link
Member

Well that is progress I guess, that means we have successfully trapped the scenario. The thing that does not make sense is that there is not a log messaging indicating what is happening. @cwjohnston any thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants