Merge pull request #265 from boblemaire/staged

Staged

Docs/dotlocal.rst

@@ -0,0 +1,86 @@
+==============
+iotawatt.local
+==============
+
+How does it work?
+-------------------
+
+The IoTaWatt connects to your local area network via WiFi.
+The procedure for connecting is outlined `here <connectWiFi.html>`_.
+When your router accepts the connection, it assigns an IP address
+and necessary routing information to the IoTaWatt using the 
+Dynamic Host Configuration Protocol (DHCP).
+Depending on your router settings, that IP address will either be
+a fixed local IP address, or a semi-random IP address chosen from
+a pool of available addresses.
+
+When you enter **iotawatt.local** into your browser,
+your computer uses one of several similar
+`zeroconf <https://en.wikipedia.org/wiki/Zero-configuration_networking>`_ 
+protocols to discover the IP address that has been assigned to the IoTaWatt.
+These protocols go under several names: Bonjour (Apple) and LLMNR (Microsoft).
+See the `zeroconf <https://en.wikipedia.org/wiki/Zero-configuration_networking>`_ 
+link for the detailed WiKi.
+
+Essentially, when you type iotawatt.local intro your browser,
+the underlying networking layer in your computer broadcasts a 
+datagram message available to all members of the LAN, asking 
+if there is anyone out there that called iotawatt.
+
+At startup, IoTaWatt creates a process that listens for those datagrams.
+When it hears it's name, it responds to the sender saying "I'm iotawatt
+and my IP address is xx.xx.xx.xx".  The requestor makes a note of
+this address and uses it to send subsequent transactions to iotawatt.local.
+
+How does that *not* work?
+-------------------------
+
+Sounds simple doesn't it?  What could possibly go wrong?
+
+Plenty.  First off, these zeroconf protocols are not part of the
+standardized internet. If your computer doesn't have some version of
+the protocol installed, it won't work.
+
+There are other issues.  Remember that your computer resolves
+the name and then stores the IP for future use?  Well, if the
+IoTaWatt gets assigned an IP address from a pool of addresses,
+it can change without your computer knowing it.  That is a
+common problem.  Your computer can talk to the IoTaWatt for
+days or weeks and then suddenly it stops.  It's possible the
+IoTaWatt restarted or it's DHCP lease expired and it got a
+different IP address.  Some computers will simply never think
+to broadcast a new query, insisting on unsuccessfully retrying the old IP
+address forever.
+
+How to make it work better.
+---------------------------
+
+It is recommended that you to assign a 
+static IP to the IoTaWatt right after installation.  You do this
+in your router and should assign an IP address that is
+not in the DHCP pool, so there is no opportunity for conflict.
+Your router associates the specified IP address with the MAC
+address of the IoTaWatt and always gives it that address during
+the DHCP handshake at startup.
+Write it down.  If you subsequently find that you can't access
+via iotawatt.local, you can use the IP address by 
+typing HTTP://xx.xx.xx.xx as a URL in the browser.
+
+If you are reading this because you didn't assign a static IP
+and now can't access your IoTaWatt with iotawatt.local, 
+you may need to turn off your computer and the IoTaWatt, restart
+your router, then restart your computer and the IoTaWatt.
+
+What else?
+----------
+
+Throughout this section, we have been using the name iotawatt.local.
+If you changed the name of the IoTaWatt using the Device setup
+of the configuration app, you would appended .local to that name.  
+If you changed the name to *ttawatoi* you would use
+ttawatoi.local.
+
+The last word
+-------------
+
+Fix the IP and write it down.

Docs/index.rst

@@ -28,5 +28,6 @@ IoTaWatt documentation
     Query API <query.rst>
     Message Log <messageLog>
     Troubleshooting <troubleshooting>
+    Accessing iotawatt.local <dotlocal>
 
 

Firmware/.gitignore

@@ -2,8 +2,6 @@
 .piolibdeps
 .vscode/c_cpp_properties.json
 .vscode/
-.pioenvs
-.piolibdeps
 .clang_complete
 .gcc-flags.json
 .vscode/launch.json

Firmware/IotaWatt/IotaScript.cpp

@@ -299,3 +299,33 @@ double    Script::evaluate(double result, uint8_t token, double operand){
           default:     return 0;        
         }
 }
+
+          // Sort the Scripts in the set
+          // Uses callback comparison
+          // Simple bubble sort
+
+void  ScriptSet::sort(std::function<int(Script*, Script*)> scriptCompare){
+  int count = _count;
+  while(--count){
+    Script* link = nullptr;
+    Script* a = _listHead;
+    Script* b = a->_next;
+    for(int i=0; i<count; i++){
+      int comp = scriptCompare(a, b);
+      if( comp > 0){
+        a->_next = b->_next;
+        b->_next = a;
+        if(link){
+          link->_next = b;
+        } else {
+          _listHead = b;
+        }
+        link = b;
+      } else {
+        link = a;
+      }
+      a = link->_next;
+      b = a->_next;
+    }
+  }
+}

Firmware/IotaWatt/IotaScript.h

@@ -94,6 +94,10 @@ class ScriptSet {
       }
     }
 
+    //typedef std::function<int(Script*, Script*)> scriptCompare;
+
+    void sort(std::function<int(Script*, Script*)> scriptCompare);
+
     size_t    count();      // Retrieve count of Scripts in the set.
     Script*   first();      // Get -> first Script in set
 

Firmware/IotaWatt/Setup.cpp

@@ -19,8 +19,8 @@ void setup()
   //Serial.println(F("Serial Initialized"));
 
   //*************************************** Start SPI *************************************************
-    
-  pinMode(pin_CS_ADC0,OUTPUT);                    // Make sure all the CS pins are HIGH
+
+  pinMode(pin_CS_ADC0, OUTPUT);                // Make sure all the CS pins are HIGH
   digitalWrite(pin_CS_ADC0,HIGH);
   pinMode(pin_CS_ADC1,OUTPUT);
   digitalWrite(pin_CS_ADC1,HIGH);
@@ -186,9 +186,9 @@ if(spiffsBegin()){
   authLoadPwds();  
 
 //*************************************** Start the WiFi  connection *****************************
-
   WiFi.hostname(deviceName);
   WiFi.setAutoConnect(true);
+  WiFi.setAutoReconnect(true);
   WiFi.begin();
   if(WiFi.status() != WL_CONNECTED){
     WiFi.reconnect();
@@ -197,14 +197,14 @@ if(spiffsBegin()){
         // If the RTC is not running or power fail restart
         // Use the WiFi Manager.
 
-  if( ! RTCrunning || powerFailRestart){
+  if( (! RTCrunning) || powerFailRestart){
     uint32_t autoConnectTimeout = millis() + 3000UL;
     while(WiFi.status() != WL_CONNECTED){
       if(millis() > autoConnectTimeout){
         setLedCycle(LED_CONNECT_WIFI);
         WiFiManager wifiManager;
         wifiManager.setDebugOutput(false);
-        wifiManager.setConfigPortalTimeout(180);
+        wifiManager.setConfigPortalTimeout(120);
         String ssid = "iota" + String(ESP.getChipId());
         String pwd = deviceName;
         log("Connecting with WiFiManager.");
@@ -217,6 +217,11 @@ if(spiffsBegin()){
           wifiManager.autoConnect(ssid.c_str(), pwd.c_str());
           endLedCycle();
         }
+        if(! WiFi.isConnected()){
+          log("Did not connect after power-fail. Restarting to reset WiFi.");
+          delay(500);
+          ESP.restart();
+        }
         break;
       }
       yield();

Firmware/IotaWatt/getConfig.cpp

@@ -406,6 +406,11 @@ bool configOutputs(const char* JsonStr){
     return false;
   }
   outputs = new ScriptSet(outputsArray);
+  // outputs->sort([](Script* a, Script* b)->int{
+  //   int res = strcmp(a->name(), b->name());
+  //   Serial.printf("%s, %s, %d\r\n",a->getUnits(), b->getUnits(), res);
+  //   return res;
+  // });
   return true;
 } 
 

Firmware/IotaWatt/influxDB.cpp

@@ -6,6 +6,7 @@ bool      influxStarted = false;                    // True when Service started
 bool      influxStop = false;                       // Stop the influx service
 bool      influxRestart = false;                    // Restart the influx service
 bool      influxLogHeap = false;                    // Post a heap size measurement (diag)
+bool      influxStaticKeyset = true;                // True if keyset has no variables (except $device)
 uint32_t  influxLastPost = 0;                       // Last acknowledge post for status report
 
     // Configuration settings
@@ -42,15 +43,15 @@ uint32_t influxService(struct serviceBlock* _serviceBlock){
   static IotaLogRecord* oldRecord = nullptr;
   static uint32_t lastRequestTime = 0;          // Time of last measurement in last or current request
   static uint32_t lastBufferTime = 0;           // Time of last measurement reqData buffer
-  static uint32_t UnixNextPost = UTCtime();    // Next measurement to be posted
+  static uint32_t UnixNextPost = UTCtime();     // Next measurement to be posted
   static xbuf reqData;                          // Current request buffer
   static uint32_t reqUnixtime = 0;              // First measurement in current reqData buffer
   static int  reqEntries = 0;                   // Number of measurement intervals in current reqData
   static int16_t retryCount = 0;                // HTTP error count
   static asyncHTTPrequest* request = nullptr;   // -> instance of asyncHTTPrequest
-  static uint32_t postFirstTime = UTCtime();   // First measurement in outstanding post request
-  static uint32_t postLastTime = UTCtime();    // Last measurement in outstanding post request
-  static size_t reqDataLimit = 3000;            // transaction yellow light size
+  static uint32_t postFirstTime = UTCtime();    // First measurement in outstanding post request
+  static uint32_t postLastTime = UTCtime();     // Last measurement in outstanding post request
+  static size_t reqDataLimit = 4000;            // transaction yellow light size
   static uint32_t HTTPtoken = 0;                // HTTP resource reservation token
   static Script* script = nullptr;              // current Script
 
@@ -323,12 +324,6 @@ uint32_t influxService(struct serviceBlock* _serviceBlock){
 
       }
 
-          // If not enough entries for bulk-send, come back in one second;
-
-      if(((currLog.lastKey() - influxLastPost) / influxDBInterval + reqEntries) < influxBulkSend){
-        return UTCtime() + 1;
-      } 
-
           // If buffer isn't full,
           // add another measurement.
 
@@ -362,23 +357,35 @@ uint32_t influxService(struct serviceBlock* _serviceBlock){
 
         script = influxOutputs->first();
         trace(T_influx,7);
+        String lastMeasurement;
+        String thisMeasurement;
         while(script){
           double value = script->run(oldRecord, logRecord, elapsedHours);
           if(value == value){
-            reqData.write(influxVarStr(influxMeasurement, script));
-            if(influxTagSet){
-              trace(T_influx,71);
-              influxTag* tag = influxTagSet;
-              while(tag){
-                reqData.printf_P(PSTR(",%s=%s"), tag->key, influxVarStr(tag->value, script).c_str());
-                tag = tag->next;
+            thisMeasurement = influxVarStr(influxMeasurement, script);
+            if(influxStaticKeyset && thisMeasurement.equals(lastMeasurement)){
+              reqData.printf_P(PSTR(",%s=%.*f"), influxVarStr(influxFieldKey, script).c_str(), script->precision(), value);
+            } else {
+              if(lastMeasurement.length()){
+                reqData.printf(" %d\n", UnixNextPost);
               }
+              reqData.write(thisMeasurement);
+              if(influxTagSet){
+                trace(T_influx,71);
+                influxTag* tag = influxTagSet;
+                while(tag){
+                  reqData.printf_P(PSTR(",%s=%s"), tag->key, influxVarStr(tag->value, script).c_str());
+                  tag = tag->next;
+                }
+              }
+              reqData.printf_P(PSTR(" %s=%.*f"), influxVarStr(influxFieldKey, script).c_str(), script->precision(), value);
             }
-            reqData.printf_P(PSTR(" %s=%.*f"), influxVarStr(influxFieldKey, script).c_str(), script->precision(), value);
-            reqData.printf(" %d\n", UnixNextPost);
           }
+          lastMeasurement = thisMeasurement;
           script = script->next();
         }
+        reqData.printf(" %d\n", UnixNextPost);
+
         trace(T_influx,7);
         delete oldRecord;
         oldRecord = logRecord;
@@ -388,6 +395,7 @@ uint32_t influxService(struct serviceBlock* _serviceBlock){
         reqEntries++;
         lastBufferTime = UnixNextPost;
         UnixNextPost +=  influxDBInterval - (UnixNextPost % influxDBInterval);
+        return 1;
       }
 
             // If there's no request pending and we have bulksend entries,
@@ -577,16 +585,29 @@ bool influxConfig(const char* configObj){
       influxTagSet = tag;
       tag->key = charstar(tagset[i]["key"].as<const char*>());
       tag->value = charstar(tagset[i]["value"].as<const char*>());
+      if((strstr(tag->value,"$units") != nullptr) || (strstr(tag->value,"$name") != nullptr)) influxStaticKeyset = false;
     }
   }
 
+        // Build the measurement scriptset
+
   delete influxOutputs;
   influxOutputs = nullptr;
   JsonVariant var = config["outputs"];
   if(var.success()){
     trace(T_influxConfig,9);
-    influxOutputs = new ScriptSet(var.as<JsonArray>()); 
+    influxOutputs = new ScriptSet(var.as<JsonArray>());
+  }
+  else {
+    return false;
   }
+
+        // sort the measurements by measurement name
+
+  influxOutputs->sort([](Script* a, Script* b)->int {
+    return strcmp(influxVarStr(influxMeasurement, a).c_str(), influxVarStr(influxMeasurement, b).c_str());
+  });
+
   if( ! influxStarted) {
     trace(T_influxConfig,10);
     NewService(influxService, T_influx);

Firmware/IotaWatt/iotaInputChannel.cpp

@@ -19,10 +19,12 @@ void IotaInputChannel::reset(){
 }
 
 void IotaInputChannel::ageBuckets(uint32_t timeNow) {
-    double elapsedHrs = double((uint32_t)(timeNow - dataBucket.timeThen)) / 3600000E0;
-    dataBucket.accum1 += dataBucket.value1 * elapsedHrs;
-    dataBucket.accum2 += dataBucket.value2 * elapsedHrs;
-    dataBucket.timeThen = timeNow;    
+    if(timeNow > dataBucket.timeThen){
+        double elapsedHrs = double((uint32_t)(timeNow - dataBucket.timeThen)) / 3600000E0;
+        dataBucket.accum1 += dataBucket.value1 * elapsedHrs;
+        dataBucket.accum2 += dataBucket.value2 * elapsedHrs;
+        dataBucket.timeThen = timeNow;
+    }    
 }
 
 void IotaInputChannel::setVoltage(float volts, float Hz){

SD/graph2.htm

@@ -38,8 +38,8 @@
     <script src="https://cdnjs.cloudflare.com/ajax/libs/flot/0.8.3/jquery.flot.min.js" integrity="sha256-LMe2LItsvOs1WDRhgNXulB8wFpq885Pib0bnrjETvfI=" crossorigin="anonymous"></script>
     <script src="https://cdnjs.cloudflare.com/ajax/libs/flot/0.8.3/jquery.flot.stack.min.js" integrity="sha256-ktoUUP+bAQB7GIVsROjHJhc0t7wdfGJV7/V3xvewiYk=" crossorigin="anonymous"></script>
     <script src="https://cdnjs.cloudflare.com/ajax/libs/flot/0.8.3/jquery.flot.selection.min.js" integrity="sha256-1Zrfof7MBfulEOIkj6Y0yLhQkbT7Cq2WtO3xlRGUF9c=" crossorigin="anonymous"></script>
-    <script language="javascript" type="text/javascript" src="https://cdn.jsdelivr.net/npm/jquery.flot@0.8.3/jquery.flot.time.min.js"></script>
-
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/flot/0.8.3/jquery.flot.time.min.js" integrity="sha256-gCrSjRo/Z6W7Cfc1oEL6BH8HKjgiiO+ItV8A+z9Scpw=" crossorigin="anonymous"></script>
+    
     <!-- --------------------------------------------------------------------------------------------------------------->
 
     <div id="wrapper" style="max-width:1600px">