Hi Jared, just giving this a bump and wondering if you were able to replicate this issue

Fox

I hope they're helpful! Looking back it looks a little bit like gibberish! Gimme a bell if you need anything

Fox

I've still got the API dropping in and out, can't always log in. Can I get someone to remote in and take a look?

Fox

Good idea with the thread dumps, if I can get through to have them exported during an import session I will post them here.

Thanks for your input here. Perhaps some feedback regarding my server configuration may help here so I can document best practices for running and getting the best out of the mango system.
Seeing how if SSL with http2 is the way to go perhaps I need to rejig how my server is configured.
Let's see how I go

Fox

I know the system drops out when we are updating the values via the mango API every 5 minutes.
I'm looking at

/v1/point-values
Update one or many data point's current value

But am unsure if this will still apply an update event to affected datapoints, that and I'm not sure of the correct format.
thought it might be something like

{
xid:"aasdf",
timestamp:12314556677,
value:10,
type:"NUMERIC"
}

I'll try the max open files later tonight if you feel that could be a bottle neck if I have several 100 different points coming in with a 100 values at a time.

Thanks for coming back to me at this ungodly hour Terry, sorry I'm stressing a little bit here due to being outside my knowledge base...
I'm still on 3.5.6, I don't want to do any upgrades due to the time of year and that our customers are more active during the spring/summer/autumn seasons and rely on our systems being up. Trying to maintain 99.9% uptime is a gripper.. That and I'm afraid of any breaking changes created could really throw a spanner in the development works...

I confirmed it's not the proxy as I loaded mango's pages directly and sure enough the pages dropped out.

#    Copyright (C) 2014 Infinite Automation Systems Inc. All rights reserved.
#    @author Matthew Lohbihler

###############################################################################
# TO OVERRIDE VALUES IN THIS FILE...
#
# Do not change the values in this file, because when you upgrade your core 
# your changes will be overwritten. Instead, create a new file called 
# <MA_HOME>/overrides/properties/env.properties and override properties 
# there. The overrides directory will never be overwritten by an upgrade, so 
# your customizations will be safe.
# 
###############################################################################

# The port at which Mango Automation will listen for browser connections
web.port=8080
# The host interface to which Mango Automation will bind and listen for new connections
#  0.0.0.0 is the special interface that will force a bind to all available interfaces
web.host=0.0.0.0

# Should Mango Automation open (if possible) a browser window when it starts up?
web.openBrowserOnStartup=true

# Web caching settings

# disable caching
web.cache.noStore=false
web.cache.noStore.rest=true
web.cache.noStore.resources=false

# set max age of cached files in seconds, only if noStore=false
# versioned resources are those with ?v=xxx on the query string
web.cache.maxAge=0
web.cache.maxAge.rest=0
web.cache.maxAge.resources=86400
web.cache.maxAge.versionedResources=31536000

#Upload file size limit (bytes) -1 means no limit
web.fileUpload.maxSize=50000000

# Default database settings, NOTE that on windows this must be an absolute path
#db.type=h2
#db.url=jdbc:h2:${ma.home}/databases/mah2
#db.location=${ma.home}/databases/mah2
#db.username=
#db.password=
#For web console
#db.web.start=false
#db.web.port=8091
#to compact the database size at shutdown (may take longer but will free up disk space)
#db.h2.shutdownCompact=false

#db.type=derby
#db.url=${ma.home}/databases/madb
#db.username=
#db.password=

#General Database Settings
db.pool.maxActive=100
db.pool.maxIdle=10
db.update.log.dir=${ma.home}/logs/

# setting to show query times in the logs as INFO
db.useMetrics=false
# if set, will only log slow queries, above this threshold in ms. Will be logged at WARN level instead of INFO
db.metricsThreshold=100

#--The following database properties are for RQL REST queries and can be changed during runtime and will be picked up at most in 5s--
#Force the use of indexes (Experimental and only on MySQL so far)
db.forceUseIndex=true
#Tell the jdbc driver to fetch this many rows at a time, useful over network connected dbs (Not MySQL)
# negative values will force use jdbc driver default
db.fetchSize=-1
#Tell the database to not return the entire result set (or fetch blocks) but to return row by row
# can slow down performance on network systems but reduce memory footprint for large queries
db.stream=false
#-- End auto-reloading Database Properties --

# MySQL database settings. Your MySQL instance must already be running and configured before this can be used.
#db.type=mysql
#db.url=jdbc:mysql://localhost/<your mysql schema name>
#db.username=<your mysql username>
#db.password=<your mysql password>
#db.mysqldump=<location/command for mysqldump executable for backups>
#db.mysql=<location/command for mysql executable for restore>

db.mysqldump=<location/command for mysqldump executable for backups>
db.mysql=<location/command for mysql executable for restore>


# MySQL database settings. Your MySQL instance must already be running and configured before this can be used.
db.type=mysql
db.url=jdbc:mysql://[DB_IP]/mangodb

db.username={USER}
db.password={PASS}
db.mysqldump=/usr/bin/mysqldump
db.mysql=/usr/bin/mysql

# Database settings for conversion. If the db.* settings point to a new database instance, and the convert type setting
# is set, Mango Automation will attempt to convert from the convert.db.* settings to the db.* settings
# Note that database conversions should not be performed in the same step as an upgrade. First upgrade, then convert.
#convert.db.type=h2
#convert.db.url=jdbc:h2:/mnt/disks/sdb/mango/databases/mah2
#convert.db.username=
#convert.db.password=
#convert.db.type=
#convert.db.url=${convert.db.url}
#convert.db.username=${convert.db.username}
#convert.db.password=${convert.db.password}

#Set the base path for where the NoSQL data will be stored
#db.nosql.location=${ma.home}/databases/
db.nosql.location=/mnt/disks/sdb/mango/databases/
#Set the folder name of the point value store
db.nosql.pointValueStoreName=mangoTSDB
#Set the number of files the database can have open at one time
db.nosql.maxOpenFiles=500
#Time after which a shard will be closed
db.nosql.shardStalePeriod=36000000
#Period to check for stale shards
db.nosql.flushInterval=300000
#Query Performance Tuning, File Access Type: Available[INPUT_STREAM,FILE_CHANNEL,RANDOM_ACCESS_FILE,MAPPED_BYTE_BUFFER]
db.nosql.shardStreamType=MAPPED_BYTE_BUFFER
#Setting to speed up NoSQL queries at the expense of a small increase in disk usage
db.nosql.reversible=true
#Setting this will convert your existing point value store [NONE, REVERSIBLE, UNREVERSIBLE]
db.nosql.convert=NONE
#Number of concurrent threads to use to convert the database
db.nosql.convertThreads=4
#Run the corruption scan if the db is marked dirty
db.nosql.runCorruptionOnStartupIfDirty=false

#Password encryption scheme [BCRYPT, SHA-1, NONE]
#Legacy is SHA-1, 2.8+ BCRYPT
#security.hashAlgorithm=BCRYPT

# The location of the Mango Automation store from which to get license files.
store.url=https://store.infiniteautomation.com

# SSL control

# *** NOTE ***
# You can generate a self-signed certificate for testing using the following command
# keytool -genkey -keyalg RSA -alias mango -keystore /location/to/keystore/file.jks -validity 365 -keysize 2048

# Enter keystore password: {type your keystore password <ENTER>}
# Re-enter new password: {type your keystore password <ENTER>}
# What is your first and last name?
#   [Unknown]: {the hostname mango is running on e.g. mymangotest.com OR localhost <ENTER>}
# What is the name of your organizational unit?
#   [Unknown]: {e.g. Mango testing <ENTER>}
# What is the name of your organization?
#   [Unknown]: {e.g. Infinite Automation Systems Inc. <ENTER>}
# What is the name of your City or Locality?
#   [Unknown]: {e.g. Erie <ENTER>}
# What is the name of your State or Province?
#   [Unknown]: {e.g. Colorado <ENTER>}
# What is the two-letter country code for this unit?
#   [Unknown]: {e.g. US <ENTER>}
# Is CN=localhost, OU=Development, O=Infinite Automation Systems Inc., L=Erie, ST=Colorado, C=US correct?
#   [no]:  {type yes <ENTER>}
# 
# Enter key password for <mango>
#         (RETURN if same as keystore password): {type your key password or just press <ENTER>}

# Note: Enabling SSL also turns on HSTS which may not be desirable, see below
ssl.on=false
ssl.port=8443
ssl.keystore.location=/location/to/keystore/file.jks
ssl.keystore.password=freetextpassword
# If they key password is commented out, it is assumed to be the same as the keystore password
#ssl.key.password=
#Time socket can be idle before being closed (ms)
ssl.socketIdleTimeout=70000

#Enable ALPN (Application-Layer Protocol Negotiation) for HTTP/2
# on current browsers HTTP/2 is only available for TLS/SSL connections.
# Note that with this setting you must also have the ALPN script extension enabled for Mango to start.
# (Adds -javaagent:${MA_HOME}/boot/jetty-alpn-agent-x.x.x.jar to the java options)
ssl.alpn.on=true
#Show debug output for alpn connections in log
ssl.alpn.debug=false

# Configure HSTS (HTTP Strict Transport Security)
# Enabled by default when ssl.on=true
# Sets the Strict-Transport-Security header, web browsers will always connect using HTTPS when they
# see this header and they will cache the result for max-age seconds
ssl.hsts.enabled=true
ssl.hsts.maxAge=31536000
ssl.hsts.includeSubDomains=false

# System time zone. Leave blank to use default VM time zone.
timezone=

#Rest API Configuration
rest.enabled=true

#Enable to make JSON More readable
rest.indentJSON=true
#Cross Origin Request Handling
rest.cors.enabled=true
rest.cors.allowedOrigins=*
rest.cors.allowedMethods=PUT,POST,GET,OPTIONS,DELETE
rest.cors.allowedHeaders=content-type,x-requested-with,authorization
rest.cors.exposedHeaders=
rest.cors.allowCredentials=true
rest.cors.maxAge=3600

#For rest API Documentation at /swagger-ui.html
swagger.enabled=true
#path to api-docs for swagger tools
springfox.documentation.swagger.v2.path=/swagger/v2/api-docs
#By default the api-docs endpoint used by the swagger-ui is protected
#  if it is allowed to be public then one can generate tokens in for Users
#  to supply to the swagger-ui and access the api endpoints
#  To use, enter: Bearer <space> <token value> into the Authorize value input in the swagger ui
swagger.apidocs.protected=true
#Regex Pattern to scan for REST API endpoints for Swagger to document/display
swagger.mangoApiVersion=v[12]


#Distributor Settings
distributor=IA

#Jetty Thread Pool Tuning
# Time a thread must be idle before killing to keep pool size at minimum
web.threads.msIdleTimeout=10000
# Number of threads to keep around to handle incoming connections
web.threads.minimum=10
# Number of threads allowed to be created to handle incoming requests as needed
web.threads.maximum=250
# Number of Requests To Queue if all threads are busy
web.requests.queueSize=200
# Ping timeout for response from browser
web.websocket.pingTimeoutMs=20000
#Time socket can be idle before being closed (ms)
web.socketIdleTimeout=30000

#Jetty JSP Configuration
# See here for options: http://www.eclipse.org/jetty/documentation/9.2.10.v20150310/configuring-jsp.html
web.jsp.development=true
web.jsp.genStringsAsCharArray=true
web.jsp.trimSpaces=false
web.jsp.classdebuginfo=false
web.jsp.supressSmap=true
web.jsp.compilerClassName=org.apache.jasper.compiler.JDTCompiler
web.jsp.compiler=modern
web.jsp.fork=false
web.jsp.keepgenerated=true

#iFrame Header Control iFrame Header Control 'X-Frame-Options' (case sensitive options)
# SAMEORIGIN - Only allow Mango to embed i-frames when the requesting page was loaded from the Mango domain
# DENY - Do not allow at all
# ANY - Do not even use the header at all 
# One specific domain name can be supplied so that the header becomes: ALLOW-FROM http://foo.bar.com 
web.security.iFrameAccess=SAMEORIGIN

#Regex used to match serial ports so they show up in the menu
serial.port.linux.regex=((cu|ttyS|ttyUSB|ttyACM|ttyAMA|rfcomm|ttyO|COM)[0-9]{1,3}|rs(232|485)-[0-9])
serial.port.linux.path=/dev/
serial.port.windows.regex=
serial.port.windows.path=
serial.port.osx.path=/dev/
serial.port.osx.regex=(cu|tty)..*

#Start data sources in parallel threads
runtime.datasource.startupThreads=8
#Log startup times for runtime manager
runtime.datasource.logStartupMetrics=true
#Log number of aborted polls for a polling data source this often at a minimum (only logged after next aborted poll past this time)
runtime.datasource.pollAbortedLogFrequency=3600000

#Report Javascript Execution Times at INFO Level logging
# add this to log4j.xml   <category name="org.perf4j.TimingLogger"><level value="info"/></category>
runtime.javascript.metrics=false

#Default task queue size for the Real Time Timer, should multiple tasks of the same type be queued up?
# Tasks are rejected from a full queue, a size of 0 means reject multiple instances of the same task
runtime.realTimeTimer.defaultTaskQueueSize=0
#When a task queue is full should the waiting tasks be discarded and replaced with the most recent
runtime.realTimeTimer.flushTaskQueueOnReject=false
#Delay (in ms) to wait to rate limit task rejection log messages so they don't fill up logs and use too much cpu doing it
runtime.taskRejectionLogPeriod=10000
#Maximum counts to wait to terminate the thread pool's tasks that are running or queued to run
# each count is 1 second.  So the default of 60 = 1 minute.  Note that the medium and low
# timeout happens first and then the remaining time is spent waiting of the high priority tasks.
# So by setting both to the same value will result in waiting only as long as that value.
runtime.shutdown.medLowTimeout=60
runtime.shutdown.highTimeout=60

# Set the location of the file stores
# If not set, the location is MA_HOME/filestore
filestore.location=

# Limits the rate at which an unauthenticated IP address can access the REST API Defaults to an initial 10 request burst then 2 requests per 1 second thereafter
rateLimit.rest.anonymous.enabled=true 
rateLimit.rest.anonymous.burstQuantity=10 
rateLimit.rest.anonymous.quanitity=2 
rateLimit.rest.anonymous.period=1 
rateLimit.rest.anonymous.periodUnit=SECONDS

# Limits the rate at which an authenticated user can access the REST API Disabled by default
rateLimit.rest.user.enabled=false 
rateLimit.rest.user.burstQuantity=20 
rateLimit.rest.user.quanitity=10 
rateLimit.rest.user.period=1 
rateLimit.rest.user.periodUnit=SECONDS

# Limits the rate at which authentication attempts can occur by an IP address Defaults to an initial 5 attempt burst then 1 attempt per 1 minute thereafter
rateLimit.authentication.ip.enabled=true 
rateLimit.authentication.ip.burstQuantity=5 
rateLimit.authentication.ip.quanitity=1 
rateLimit.authentication.ip.period=1 
rateLimit.authentication.ip.periodUnit=MINUTES

# Limits the rate at which authentication attempts can occur against a username Defaults to an initial 5 attempt burst then 1 attempt per 1 minute thereafter
rateLimit.authentication.user.enabled=true 
rateLimit.authentication.user.burstQuantity=5 
rateLimit.authentication.user.quanitity=1 
rateLimit.authentication.user.period=1
rateLimit.authentication.user.periodUnit=MINUTES

Follow symbolic links when serving files from Jetty
web.security.followSymlinks=true

# Content Security Policy settings, please see https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
# The reasons for the default policy are outlined below
# style-src 'unsafe-inline' - inline styles are used by AngularJS Material for the dynamic theming
# script-src 'unsafe-eval' - needed by Fabric.js used in amCharts for drawing on charts, also gives AngularJS a 30% performance boost
# connect-src ws: wss: - necessary as 'self' does not permit connections to websockets on the same origin, this should be configured to restrict it to your server's actual hostname
# img-src data: - allows for small base64 encoded images to be embedded inline into the html
# img-src/script-src https://www.google-analytics.com - allows for enabling Google analytics (not enabled by default, must be manually enabled by admin via UI Settings page)
# img-src/script-src https://maps.google.com https://maps.googleapis.com https://maps.gstatic.com - allows for using the Google maps component
# style-src/font-src https://fonts.googleapis.com https://fonts.gstatic.com - allows for using Google fonts in dashboards
web.security.contentSecurityPolicy.enabled=false
web.security.contentSecurityPolicy.reportOnly=false
web.security.contentSecurityPolicy.defaultSrc='self'
web.security.contentSecurityPolicy.scriptSrc='self' 'unsafe-eval' https://maps.google.com https://maps.googleapis.com https://www.google-analytics.com
web.security.contentSecurityPolicy.styleSrc='self' 'unsafe-inline' https://fonts.googleapis.com
web.security.contentSecurityPolicy.connectSrc='self' ws: wss:
web.security.contentSecurityPolicy.imgSrc='self' data: https://maps.google.com https://maps.gstatic.com https://www.google-analytics.com
web.security.contentSecurityPolicy.fontSrc='self' https://fonts.gstatic.com
web.security.contentSecurityPolicy.mediaSrc=
web.security.contentSecurityPolicy.objectSrc=
web.security.contentSecurityPolicy.frameSrc=
web.security.contentSecurityPolicy.workerSrc=
web.security.contentSecurityPolicy.manifestSrc=
web.security.contentSecurityPolicy.other=

# script-src 'unsafe-inline' - inline scripts are used extensively throughout the Mango legacy UI
# script-src 'unsafe-eval' - The Dojo JS library uses eval()
# style-src 'unsafe-inline' - inline styles are used throughout the Mango legacy UI
# connect-src ws: wss: - necessary as 'self' does not permit connections to websockets on the same origin, this should be configured to restrict it to your server's actual hostname
# img-src data: - allows for small base64 encoded images to be embedded inline into the html
# img-src/script-src https://www.google-analytics.com - allows for enabling Google analytics
web.security.contentSecurityPolicy.legacyUi.enabled=false
web.security.contentSecurityPolicy.legacyUi.reportOnly=false
web.security.contentSecurityPolicy.legacyUi.defaultSrc='self'
web.security.contentSecurityPolicy.legacyUi.scriptSrc='self' 'unsafe-inline' 'unsafe-eval' https://www.google-analytics.com
web.security.contentSecurityPolicy.legacyUi.styleSrc='self' 'unsafe-inline'
web.security.contentSecurityPolicy.legacyUi.connectSrc='self' ws: wss:
web.security.contentSecurityPolicy.legacyUi.imgSrc='self' data: https://www.google-analytics.com
web.security.contentSecurityPolicy.legacyUi.fontSrc=
web.security.contentSecurityPolicy.legacyUi.mediaSrc=
web.security.contentSecurityPolicy.legacyUi.objectSrc=
web.security.contentSecurityPolicy.legacyUi.frameSrc=
web.security.contentSecurityPolicy.legacyUi.workerSrc=
web.security.contentSecurityPolicy.legacyUi.manifestSrc=
web.security.contentSecurityPolicy.legacyUi.other=

# Set the location of the modules data directory, (relative to $MA_HOME if not absolute) If not set, the location is $MA_HOME/data
moduleData.location=data

# HTTP session (authentication) cookie name and domain name settings.
#
# Use the Mango GUID as the session cookie name
sessionCookie.useGuid=true
# name takes precedence over useGuid if set
sessionCookie.name=
# Set the domain name that the cookie is valid for, can be used to make the session login valid for subdomains too. If left blank the session cookie can only be used for the domain that you login at.
sessionCookie.domain=

Let me know if there's anything I can do here to increase performance

Fox

I keep seeing the mangoAPI server coming offline and online, to the point of interrupting user logins. What are your suggestions to stop this? I've got customers complaining about proxy errors (502) due to mango's webservice timing out.
Any suggestions would be great

Fox

I'm consistently getting this popping up after sending data created by a meta data point to be sent to a http server for parsing and forwarding.
Would this be happening now by the sheer number of points I'm firing out over time?
I've got 233 total going out by this publisher atm...
Am I getting these errors because they're not going through? They do return to normal after a while but I'm running low on brain power atm so I'm unable to deduce quickly what the issues could be..

Correct, I'll post my fixes here once I've implemented them if that's helpful

Sorry to storm in but I think that'd be beneficial for a lot of people Jared. Also thanks for your input, I think I've lost sight of a few things lately!

I'm a little confused as to why you'd want to go that far but I believe you'd do something along the lines of..
/opt/mango36/overrides/web/modules/mangoUI/web/dev/controllers/pushCtrl.js

define(['angular', 'require'], function (angular, require) {

  pushCtrl.$inject = ['$scope','$timeout'];

  function pushCtrl($scope,$timeout) 
  {
    const $ctrl = this;
    $ctrl.$onChanges = function (changes) {
      if (changes.points && Array.isArray($ctrl.points)) {

        $ctrl.point = $ctrl.points.find(p => p.xid === 'DP_ind_' + $ctrl.target);
        $ctrl.switchPoint = $ctrl.points.find(p => p.xid === 'DP_sw_' + $ctrl.target);

        $ctrl.ptCSS = { // point css values for color and blink; ex, ng-class=""
          get color(){
            if ($ctrl.point.color < 10) {
              return 0;
            }
            return Math.floor($ctrl.point.value / 10)},
          get blink(){return 'blink' + $ctrl.point.value % 10},
          get colStyle(){return 'color'+this.color}
        };

        $timeout(function() {
          console.log('color:', $ctrl.ptCSS.colStyle, 'blink:', $ctrl.ptCSS.blink );
        },1000);
      }
    };

    $ctrl.pushed = function() {
      console.log($ctrl.switchPoint.value);
        if ($ctrl.switchPoint) {
            $ctrl.switchPoint.toggleValue();
          console.log('switch ' + $ctrl.target + ' was switched.' );
       }
    };
  }

return pushCtrl;
)};//define

then pushLamp.js:

define(['angular', 'require','/modules/web/mangoUI/web/dev/controllers/pushCtrl.js'], function (angular, require,pushCtrl) {

return {
      templateUrl: '/modules/mangoUI/web/dev/push/push.html',
      bindings: {
        points: '<',
        target: '@'
      },
      controller: pushCtrl
    };

});//define

I'm confident you can leverage require in the userModule.js to shorten that controller call into a single word but I've not done a great deal of angularJS lately due to other work commitments....

Fox

I believe that stuff is pre set in the browser cookies. Have a look at that with the maUiApp stuff it remembers the configuration for that there.
There won't be any docs per se, you'll simply have to go through all the source and read how it is all pieced together.

Fixed, just mixed up the full path of the file with the userModule URL after closer inspection.

I know what i've done wrong....

I didn't return the user module!
So sorry man!

define(['angular', 'require','./components/pushLamp.js'],
  function (angular, require, pushLamp) {
    var userModule = angular.module('userModule', ['maUiApp']);

    userModule.filter('unsafe', ['$sce', function($sce) {
       return $sce.trustAsHtml;
    }]);

    try
    {
      userModule.component('pushLamp',pushLamp);
    }
    catch(e)
    {
      console.log('error: ', e);
    }
 return userModule; //MISSING THIS!!! aerghhh I hate mondays...
});

Man I need a swift kick...

ps aux | grep java

This will tell us what mango runs as. Also can you also print

ls -la /opt/mango36/overrides/web/modules/mangoUI/web/dev

nope that's definitely correct, check your directory permissions and the owner.
I'll wager mango itself may not have access to the file.
Whatever the user runs as in mango set that as your modules etc directories for a start.
up to web. everything inside that shouldn't matter.

We'll start with that and work our way up from there...

sudo chown [user]:[user] /opt/mango36/overrides -R

sudo find /opt/mango36/overrides -type d -exec chmod 755 {} \;

if you're in linux, can you navigate to the directory where your usermodule is and enter

pwd

otherwise if in windows, open a command prompt in the location and enter

DIR

to print the path

nope, just wanted to make sure you did have the files inside the mango's installation directory regardless of how it is named.

Did the http requests from your browser work?

If the mango is on localhost it'd be localhost/modules/mangoUI/web/dev/userModule.js

Hang on, your mango install is in /opt/mango36 ?
Not in /opt/mango ?

You've got the right idea, the first thing I suggest is to see if you can even manually load the userModule from your web browser

[mangoURL]/modules/mangoUI/web/dev/userModule.js

if you can pull that, then try with the components directory