• J
    jflores13

    Oh well, I'm sorry but I take it back, seems like we can't stop the loop, even with the Log4j approach we are still getting thousands of events. More than 24,000 in a couple of hours. Most of them are like this:

    ERROR 2019-10-29T11:44:19,621 (com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.postMessage:145) - Slack message failed to send java.net.ConnectException: ratelimited at com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connectImpl(SlackWebSocketSessionImpl.java:338) ~[?:?] at com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connect(SlackWebSocketSessionImpl.java:292) ~[?:?] at com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.postMessage(SlackMessageEventHandlerRT.java:122) ~[?:?] at com.infiniteautomation.mango.slack.event.SlackMessageEventHandlerRT.eventRaised(SlackMessageEventHandlerRT.java:67) ~[?:?] at com.serotonin.m2m2.rt.EventManagerImpl.handleRaiseEvent(EventManagerImpl.java:1048) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.EventManagerImpl.raiseEvent(EventManagerImpl.java:243) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.event.detectors.PointEventDetectorRT.raiseEvent(PointEventDetectorRT.java:43) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.event.detectors.PointChangeDetectorRT.pointChanged(PointChangeDetectorRT.java:54) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.dataImage.DataPointRT$EventNotifyWorkItem.execute(DataPointRT.java:817) ~[mango-3.6.4.jar:?] at com.serotonin.m2m2.rt.maint.BackgroundProcessingImpl$RejectableWorkItemRunnable.run(BackgroundProcessingImpl.java:632) ~[mango-3.6.4.jar:?] at com.serotonin.timer.Task.runTask(Task.java:179) ~[mango-3.6.4.jar:?] at com.serotonin.timer.TaskWrapper.run(TaskWrapper.java:23) ~[mango-3.6.4.jar:?] at com.serotonin.timer.OrderedThreadPoolExecutor$OrderedTaskCollection.run(OrderedThreadPoolExecutor.java:336) ~[mango-3.6.4.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_172] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
    

    Couldn't find out what is triggering these changes that raise the event detector.

    Tried turning off the "Error" data point but now the "Warning" one is also sending weird messages, like this one:

    WARN 2019-10-29T12:01:45,429 (com.serotonin.m2m2.web.dwr.MiscDwr.jsError:167) - Javascript error Description: Error: Tried to register widget with id==chartRendererSelect but that id is already registered Page: http://192.168.100.33:8088/resources/dojo/dojo.js?v=1571864830 line 345 > Function line 3 > eval Line: 2 Browser name: Firefox Browser version: 71 osName: Windows location: http://192.168.100.33:8088/data_source_edit.shtm?dsid=498 @ Oct 29, 2019 12:01 PM
    

    We are getting this message when we try to edit the Log4j data points in the old UI (so we can access the event detectors and the purge now button). Explanation below:

    Strangely, when we try to edit the data source in the old UI, the "Data points" tab is not existent inside the data source. So I look for the points in the global "Data points" tab, where all data points appear, I filter them by name to find the Log4j Slack Messages Data Points, but most times, when I click the edit (pencil) button on any of the 4 data points, I only get the above "Warning" message in Slack and this same message on a blue box in the old UI:

    Error: Tried to register widget with id==chartRendererSelect but that id is already registered
    

    It works super randomly; I seldom get to edit the data point. It's a strange behaviour, something I never saw before with any other data source. I did add the line @terrypacker suggested in the log4j2.xml, inside the loggers section, and restarted it btw.

    Is there a quick fix to this? Thanks!

    posted in Mango feedback read more
  • J
    jflores13

    @terrypacker Just finished configuring the Log4j approach and you were absolutely right, it's perfect for what we wanted. Works well too.

    Thanks a lot!

    posted in Mango feedback read more
  • J
    jflores13

    @jared-wiltshire said in AI, Machine Learning, Neural Networks, TensorFlow:

    Yes, no problem running TensorFlow JS in the cloud on NodeJS. However I am not sure what advantage this would afford over the native binaries.

    True, I'm not sure either, just trying to think about all the possibilities, hehe.

    @terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

    @jflores13 said in AI, Machine Learning, Neural Networks, TensorFlow:
    Something to look into in the future for sure, but that approach would have its own problems to overcome. For example getting the data from Mango to the model.

    Yeeeah, I don't visualize how that interaction could be simplified. I guess it's just not as simple as I was hoping.

    I'll keep looking for possible solutions to this and will share anything I can find.

    Thanks!

    posted in Development general discussion read more
  • J
    jflores13

    @terrypacker ahhh, I see! Alright, don't worry, thanks for that quick fix, I will definitely try the Log4J approach.

    @terrypacker said in Slack Log Settings - Only one log level?:

    That would be a good feature for this module to have, a way to queue and then retry messages that have been rejected for rate limiting reasons. This thread will be a good reference if/when we can get enough support to do more development on this module. I could be persuaded to create a public fork of the Slack module if anyone wants to make some changes and submit a pull request...

    I would love to contribute but, to be honest, I don't think my coding skills are enough for that yet. I'll keep learning and hopefully I can persuade you soon.

    Thanks again, Terry! :)

    posted in Mango feedback read more
  • J
    jflores13

    Hello,

    @craigweb said in Slack Log Settings - Only one log level?:

    Hi @jflores13
    This is not possible with the slack log publisher.

    Alright, thanks, @CraigWeb.

    @terrypacker said in Slack Log Settings - Only one log level?:

    After thinking about this more I believe you could use the LOG4J data source with an event detector which would raise events that could be handled by Slack Event Handlers. This would work as follows:

    1. Create Log4J data source with data points for the various LOG levels you want to send to slack
    2. Add an on change event detector to each point.
    3. Add a Slack event handler to each detector to publish the event message to the channel.

    Note that I just released a new version of this module for Mango 3.6 as while I was testing this I found a bug and also realized that it did not post the latest value of the point in the event message (which it does now).

    Wow, @terrypacker, thanks for taking the time to think it through and to investigate. We will definitely try this LOG4J approach.

    We actually noticed strange behaviour before, because when we changed the log level to Information, the messages we started getting were empty, but we thought we were doing something wrong. Today, we updated the module at 13:52. We got the info messages of the startup and modules loading correctly now, but right after, the extreme error messages started coming. This is the error message copied from our Slack channel:

    ERROR 2019-23-10T16:31:51.825  (com.ullink.slack.simpleslackapi.impl.SlackWebSocketSessionImpl.connectImpl:337) - Error during authentication : ratelimited
    

    This same error was sent 4,050 times from 13:53 to 14:17, then it suddenly stopped. We then decided to add a module to test if it happened again, and it did. When Mango restarted, we got 3,626 error messages from 16:08 to 16:31. Apparently, all of them are the same error.

    I looked that up in Slack and found out about the rate limits, and the raw HTTP response that you get when you hit the limits. Apparently, you get an HTTP 429 Too Many Requests error. According to Slack, the raw response could look like this:

    HTTP/1.1 429 Too Many Requests
    Retry-After: 30
    

    I'm thinking that the handling of this Retry-After header probably has something to do with this, am I wrong?

    Thanks again!
    Greetings!

    posted in Mango feedback read more
  • J
    jflores13

    Hello, @terrypacker and @Jared-Wiltshire.

    @terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

    The examples you have seen were my attempts to do some modeling directly in Java which does work nicely but is quite time consuming since many features that are built into ML languages have to be hand coded in Java. I've considered adding support for Python and R but we haven't really enough interest yet to drive the development.

    Interesting, it's pretty cool that you've tried it. Is it possible that something cloud-based, maybe a combo similar to lambda functions + Amazon API Gateway, could make the ML model more available and reduce the need for Python or R support in the same environment?

    I have no idea if this could be done, but if the answer is yes, I think it could be charged as an extra service/module. Still, I would totally understand if the development doesn't start until more people show interest, which I hope happens soon.

    @terrypacker said in AI, Machine Learning, Neural Networks, TensorFlow:

    However I tried to download and add the minified version as a global script and it didn't work.

    I was thinking about trying that, thanks for saving me the test time.

    @jared-wiltshire said in AI, Machine Learning, Neural Networks, TensorFlow:

    It is highly likely that it wont work in Nashorn (the Java JS engine) as it doesn't support the latest ECMAScript features. Not to mention that there may be memory/script size limits.

    Hmmm. I understand. Well, maybe the memory/size limits could also be avoided with the cloud-based approach, right?

    Alright, thanks a lot for your time and answers, guys!

    posted in Development general discussion read more
  • J
    jflores13

    Hello, dear IAS devs.

    We are using Slack for our alerts and, for dev purposes, we wanted sepparate channels for the different log levels.

    I don't know if we're missing out on something, but my question is, what if we wanted to send different log levels to different channels? Is this possible?

    Via the System Settings UI, there doesn't seem to be an option to do this, but through my Mango learning curve I've found out multiple times that I'm just looking for stuff in the wrong place. Hope this is also the case.

    Thanks in advance!
    Greetings.

    posted in Mango feedback read more
  • J
    jflores13

    Hello,

    I would like to start interacting with ML to get ahead of the competition, and I'm curious, is there a way to implement TensorFlow or something similar to the Mango as a module? Is this line of development visualized for the future?

    Last week, I was exploring Github repositories related to Infinite Automation, and found that @terrypacker has worked in an experimental ML module. Am I right? Is the project still in development?

    Thanks!

    posted in Development general discussion read more
  • J
    jflores13

    Hey, Phil!
    I know, many thanks for your lots of answers.

    Yes, They are stored in the same property on the point locator because they can be. If a numeric point, the value format is interpreted as a java.text.DecimalFormat, and if binary it is simply string matched with what is captured in the value regex (capture group 1)

    Perfect! I figured that was the case, just wanted to make sure. I had gotten the last part wrong though, so thanks for explaining that!

    That definitely seems like it would be part of the problem. I'm not sure why the POP3 data source didn't expose the port as an option, it's certainly easy to do, so I have opened a git issue about this in the repository the module is in. Somewhat impassable issue until that is configurable

    Alright then, I'll be waiting.

    Thanks again! Greetings.

    posted in User help read more
  • J
    jflores13

    Hello,

    We are just getting started with pop3 data sources, and initial testing is turning into a nightmare.

    First we tried setting it up with a Gmail account, but we were getting this message:

    'Monitoreo POP': Couldn't connect to host, port: pop.gmail.com, 110; timeout -1
    

    , so I tried with an Outlook account.

    As far as I understand, both Outlook and Gmail require SSL, and both use port 995, not 110. I would expect that, if Gmail didn't let me in, Outlook wouldn't either, but the message I'm getting is completely different now and doesn't really tell me what's going on.

    This is the event I'm getting now:

    'Monitoreo POP': Command is not valid in this state.
    

    Which command? Which state?

    This is my data source:

    {
       "dataSources":[
          {
             "xid":"Monitoreo_POP3_1",
             "name":"Monitoreo POP",
             "enabled":true,
             "type":"POP3",
             "alarmLevels":{
                "INBOX_EXCEPTION":"URGENT",
                "MESSAGE_READ_EXCEPTION":"URGENT",
                "POLL_ABORTED":"URGENT",
                "PARSE_EXCEPTION":"URGENT"
             },
             "purgeType":"YEARS",
             "updatePeriods":5,
             "updatePeriodType":"MINUTES",
             "password":"********",
             "pop3Server":"outlook.office365.com",
             "username":"?????@outlook.com",
             "quantize":false,
             "useCron":false,
             "editPermission":"superadmin",
             "purgeOverride":false,
             "purgePeriod":1
          }
       ],
       "dataPoints":[
          {
             "xid":"Monitoreo_POP3_1_Showroom_BOSS_LowSH",
             "name":"Low SH en Showroom",
             "enabled":true,
             "loggingType":"ON_CHANGE",
             "intervalLoggingPeriodType":"MINUTES",
             "intervalLoggingType":"INSTANT",
             "purgeType":"YEARS",
             "pointLocator":{
                "dataType":"BINARY",
                "findInSubject":false,
                "ignoreIfMissing":true,
                "timeFormat":"",
                "timeRegex":"",
                "useReceivedTime":true,
                "valueFormat":"END -",
                "valueRegex":"(Alarma LowSH)"
             },
             "eventDetectors":[
             ],
             "plotType":"STEP",
             "rollup":"NONE",
             "unit":"",
             "simplifyType":"NONE",
             "chartColour":"#57138d",
             "chartRenderer":{
                "type":"TABLE",
                "limit":10
             },
             "dataSourceXid":"Monitoreo_POP3_1",
             "defaultCacheSize":1,
             "deviceName":"Monitoreo POP",
             "discardExtremeValues":false,
             "discardHighLimit":1.7976931348623157E308,
             "discardLowLimit":-1.7976931348623157E308,
             "intervalLoggingPeriod":1,
             "intervalLoggingSampleWindowSize":0,
             "overrideIntervalLoggingSamples":false,
             "preventSetExtremeValues":false,
             "purgeOverride":false,
             "purgePeriod":1,
             "readPermission":"user",
             "setExtremeHighLimit":1.7976931348623157E308,
             "setExtremeLowLimit":-1.7976931348623157E308,
             "setPermission":"superadmin",
             "tags":{
             },
             "textRenderer":{
                "type":"BINARY",
                "oneColour":"#fe0000",
                "oneLabel":"one",
                "zeroColour":"#0000ff",
                "zeroLabel":"zero"
             },
             "tolerance":0.0
          }
       ]
    }
    

    There's a doubt with that, because when I'm configuring the data source in the 3.6 UI, there is no "valueFormat" text box, but in the exported JSON the "valueFormat" property has the string that I input into "Binary 0 Value". Is this the expected behaviour?

    So far, as you can see I only have one data point, and I'm not sure if I'm doing the right thing with the "Binary 0 value", do I need to put it in parenthesis as well as if I was trying to group it?

    This is the only error in the console, but I'm not sure it's related:

    13:45:41.181 Possibly unhandled rejection: {"data":null,"status":-1,"config":{"method":"GET","transformRequest":[null],"jsonpCallbackParam":"callback","timeout":{},"url":"/rest/v1/events","params":{},"headers":{"Accept":"application/json, text/plain, */*","X-Requested-With":"XMLHttpRequest","X-XSRF-TOKEN":"207c27ad-f014-41a8-bec3-44ba38a2ecd8"},"cached":false},"statusText":"","xhrStatus":"abort","mangoStatusText":"Request aborted","mangoStatusTextShort":"Request aborted","resource":[]} mangoUi~ngMango~ngMangoServices.js:78:78706
        o http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
        c http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:72
        o http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:72
        c http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
        $digest http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
        timeout http://192.168.100.33:8088/ui/mangoUi~ngMango.js?v=4a67ee5ca81497c4ea25:77
        c http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
        completeTask http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
        r http://192.168.100.33:8088/ui/mangoUi~ngMango~ngMangoServices.js?v=d1c698ddb6fe71a70811:78
    

    Just as a note, I did configure both Gmail and Outlook accounts to authorize POP access.

    Do you think SSL / port # is the problem or what else could be wrong?

    Last doubt: what is the RegEx syntax flavor used by Mango? Just to be sure.

    Thanks in advance! :)

    posted in User help read more