Browse Source

LuaOTA provisioning system (#2060)

Terry Ellison 6 years ago
parent
commit
26540cf098

+ 230 - 0
lua_examples/luaOTA/README.md

@@ -0,0 +1,230 @@
+## ESP8266 Lua OTA
+
+Espressif use an optional update approach for their firmware know as OTA (over the air).  
+This module offers an equivalent facility for Lua applications developers, and enables 
+module development and production updates by carrying out automatic synchronisation 
+with a named provisioning service at reboot.
+
+### Overview
+
+This `luaOTA` provisioning service uses a different approach to 
+[enduser setup](https://nodemcu.readthedocs.io/en/dev/en/modules/enduser-setup/).
+The basic concept here is that the ESP modules are configured with a pre-imaged file 
+system that includes a number of files in the luaOTA namespace.  (SPIFFS doesn't 
+implement a directory hierarchy as such, but instead simply treats the conventional
+directory separator as a character in the filename.  Nonetheless, the "luaOTA/"
+prefix serves to separate the lc files in the luaOTA namespace.) 
+
+-  `luaOTA/check.lc`  This module should always be first executed at startup.
+-  `luaOTA/_init.lc` 
+-  `luaOTA/_doTick.lc`
+-  `luaOTA/_provision.lc`
+
+A fifth file `luaOTA/config.json` contains a JSON parameterised local configuration that
+can be initially create by and subsequently updated by the provisioning process. Most
+importantly this configuration contains the TCP address of the provisioning service, and
+a shared secret that is used to sign any records exchanged between the ESP client and
+the provisioning service. 
+
+Under this approach, `init.lua` is still required but it is reduced to a one-line lua
+call which invokes the `luaOTA` module by a `require "luaOTA.check"` statement.
+
+The `config.json` file which provides the minimum configuration parameters to connect to
+the WiFi and provisioning server, however these can by overridden through the UART by
+first doing a `tmr.stop(0)` and then a manual initialisation as described in the
+[init.lua](#initlua) section below.
+
+`luaOTA` configures the wifi and connects to the required sid in STA mode using the
+local configuration. The ESP's IP address is allocated using DHCP unless the optional
+three static IP parameters have been configured. It then attempts to establish a
+connection to the named provisioning service. If this is absent, a timeout occurs or the
+service returns a "no update" status, then module does a full clean up of all the
+`luaOTA` resources (if the `leave` parameter is false, then the wifi stack is then also
+shutdown.), and it then transfers control by a `node.task.post()` to the configured
+application module and function.
+
+If `luaOTA` does establish a connection to IP address:port of the provisioning service,
+it then issues a "getupdate" request using its CPU ID and a configuration parameter
+block as context. This update dialogue uses a simple JSON protocol(described below) that
+enables the provision server either to respond with a "no update", or to start a
+dialogue to reprovision the ESP8266's SPIFFS.
+
+In the case of "no update", `luaOTA` is by design ephemeral, that is it shuts down the
+net services and does a full resource clean up. Hence the presence of the provisioning
+service is entirely optional and it doesn't needed to be online during normal operation,
+as `luaOTA` will fall back to transferring control to the main Lua application.
+
+In the case of an active update, **the ESP is restarted** so resource cleanup on
+completion is not an issue. The provisioning dialogue is signed, so the host
+provisioning service and the protocol are trusted, with the provisioning service driving
+the process.  This greatly simplifies the `luaOTA` client coding as this is a simple 
+responder, which actions simple commands such as: 
+- download a file, 
+- download and compile file,
+- upload a file
+- rename (or delete) a file
+
+with the ESP being rebooted on completion of the updates to the SPIFFS.  Hence in 
+practice the ESP boots into one one two modes: 
+- _normal execution_ or 
+- _OTA update_ followed by reboot and normal execution.
+
+Note that even though NodeMCU follows the Lua convention of using the `lua` and `lc`
+extensions respectively for source files that need to be compiled, and for pre-compiled
+files, the Lua loader itself only uses the presence of a binary header to determine the
+file mode. Hence if the `init.lua` file contains pre-compiled content, and similarly all
+loaded modules use pre-compiled lc files, then the ESP can run in production mode
+_without needing to invoke the compiler at all_.
+
+The simplest strategy for the host provisioning service is to maintain a reference
+source directory on the host (per ESP module). The Lua developer can maintain this under
+**git** or equivalent and make any changes there, so that synchronisation of the ESP
+will be done automatically on reboot.
+
+### init.lua
+
+This is typically includes a single line: 
+```Lua
+require "LuaOTA.check"
+```
+however if the configuration is incomplete then this can be aborted as manual process 
+by entering the manual command through the UART
+```Lua
+tmr.stop(0); require "luaOTA.check":_init {ssid ="SOMESID" --[[etc. ]]}
+```
+where the parameters to the `_init` method are: 
+
+-  `ssid` and `spwd`.  The SSID of the Wifi service to connect to, together with its 
+password.
+-  `server` and `port`.  The name or IP address and port of the provisioning server.
+-  `secret`.  A site-specific secret shared with the provisioning server for MD5-based 
+signing of the protocol messages.
+-  `leave`.  If true the STA service is left connected otherwise the wifi is shutdown
+-  `espip`,`gw`,`nm`,`ns`.  These parameters are omitted if the ESP is using a DHCP 
+service for IP configuration, otherwise you need to specify the ESP's IP, gateway, 
+netmask and default nameserver.
+
+If the global `DEBUG` is set, then LuaOTA will also dump out some diagnostic debug.
+
+### luaOTA.check
+
+This only has one public method: `_init` which can be called with the above parameters.
+However the require wrapper in the check module also posts a call to `self:_init()` as a
+new task.  This new task function includes a guard to prevent a double call in the case
+where the binding require includes an explicit call to `_init()` 
+
+Any provisioning changes results in a reboot, so the only normal "callback" is to invoke
+the application entry method as defined in `config.json` using a `node.task.post()`
+
+
+### luaOTAserver.lua
+
+This is often tailored to specific project requirements, but a simple example of a
+provisioning server is included which provides the corresponding server-side
+functionality. This example is coded in Lua and can run on any development PC or server
+that supports Lua 5.1 - 5.3 and the common modules `socket`, `lfs`, `md5` and `cjson`.
+It can be easily be used as the basis of one for your specific project needs.
+
+Note that even though this file is included in the `luaOTA` subdirectory within Lua
+examples, this is designed to run on the host and should not be included in the
+ESP SPIFFS.
+
+## Implementation Notes
+
+-  The NodeMCu build must include the following modules: `wifi`, `net`, `file`, `tmr`,
+`crypto` and`sjason`. 
+
+-  This implementation follow ephemeral practices, that it is coded to ensure that all
+resources used are collected by the Lua GC, and hence the available heap on 
+application start is the same as if luaOTA had not been called.
+
+-  Methods in the `check` file are static and inherit self as an upvalue.
+
+-  In order to run comfortably within ESP resources, luaOTA executes its main 
+functionality as a number of overlay methods. These are loaded dynamically (and largely
+transparently) by an `__index` metamethod.
+
+   -  Methods starting with a "_" are call-once and return the function reference
+
+   -  All others are also entered in the self table so that successive calls will use 
+the preloaded function. The convention is that any dynamic function is called in object
+form so they are loaded and executed with self as the first parameter, and hence are
+called using the object form self:someFunc() to get the context as a parameter.
+
+   -  Some common routines are also defined as closures within the dynamic methods
+
+-  This coding also makes a lot of use of tailcalls (See PiL 6.3) to keep the stack size 
+   to a minimum. 
+
+-  The update process uses a master timer in `tmr` slot 0.  The index form is used here 
+in preference to the object form because of the reduced memory footprint. This also
+allows the developer to abort the process early in the boot sequence by issuing a
+`tmr.stop(0)` through UART0.
+
+-  The command protocol is unencrypted and uses JSON encoding, but all exchanges are 
+signed by a 6 char signature taken extracted from a MD5 based digest across the JSON
+string. Any command which fails the signature causes the update to be aborted. Commands
+are therefore regarded as trusted, and this simplifies the client module on the ESP.
+
+-  The process can support both source and compiled code provisioning, but the latter 
+is recommended as this permits a compile-free runtime for production use, and hence 
+minimises the memory use and fragmentation that occurs as a consequence of compilation.
+
+-  In earlier versions of the provisioning service example, I included `luaSrcDiet` but 
+this changes the line numbering which I found real pain for debugging, so I now just
+include a simple filter to remove "--" comments and leading and trailing whitespace if
+the source includes a `--SAFETRIM` flag. This typically reduced the size of lua files
+transferred by ~30% and this also means that developers have no excuse for not properly
+commenting their code!
+
+-  The chip ID is included in the configuration identification response to permit the 
+provisioning service to support different variants for different ESP8266 chips.
+
+-  The (optional update & reboot) operate model also has the side effect that the 
+`LuaOTA` client can reprovision itself.
+
+-  Though the simplest approach is always to do a `luaOTA.check` immediately on reboot, 
+there are other strategies that could be applied, for example to test a gpio pin or a 
+flag in RTC memory or even have the application call the require directly (assuming that 
+there's enough free RAM for it to run and this way avoid the connection delay to the 
+WiFi.
+
+## Discussion on RAM usage
+
+`luaOTA` also itself serves as a worked example of how to write ESP-friendly 
+applications.
+
+-  The functionality is divided into autoloaded processing chunks using a self 
+autoloader, so that `self:somefunction()`  calls can load new code from flash in
+a way that is simple and largely transparent to the application.  The autoloader 
+preferentially loads the `lc` compiled code variant if available.
+
+-  The local environment is maintained in a self array, to keep scoping explicit.  Note
+that since loaded code cannot inherit upvalues, then `self` must be passed to the 
+function using an object constructor `self:self:somefunction()`, but where the function
+can have a self argument then the alternative is to use an upvalue binding.  See the 
+`tmr` alarm call at the end of `_init.lua` as an example:
+```Lua
+  tmr.alarm(0, 500, tmr.ALARM_AUTO, self:_doTick())
+```
+-  The `self:_doTick()` is evaluated before the alarm API call.  This autoloads 
+`luaOTA/_doTick.lc` which stores `self` as a local and returns a function which takes 
+no arguments; it is this last returned function that is used as the timer callback,
+and when this is called it can still access self as an upvalue.
+
+-  This code makes a lot of use of locals and upvalues as these are both fast and use 
+less memory footprint than globals or table entries.
+
+-  The lua GC will mark and sweep to reclaim any unreferenced resources: tables, 
+strings, functions, userdata.  So if your code at the end of a processing phase leaves
+no variables (directly or indirectly) in _G or the Lua registry, then all of the 
+resources that were loaded to carry out your application will be recovered by the GC. 
+In this case heap at the end of a "no provisioning" path is less than 1Kb smaller than
+if luaOTA had not been called and this is an artifact of how the lua_registry system 
+adopts a lazy reuse of registry entries.
+
+-  If you find that an enumeration of `debug.getregistry()` includes function references
+or tables other than ROMtables, then you have not been tidying up by doing the 
+appropriate closes or unregister calls.  Any such stuck resources can result in a 
+stuck cascade due to upvalues being preserved in the function closure or entries in a 
+table.

+ 76 - 0
lua_examples/luaOTA/_doTick.lua

@@ -0,0 +1,76 @@
+tmr.stop(0)--SAFETRIM 
+-- function _doTick(self)
+
+  -- Upvals
+  local self                = ...
+  local wifi,net            = wifi,net 
+  local sta                 = wifi.sta
+  local config,log,startApp = self.config,self.log,self.startApp
+  local tick_count          = 0
+  
+  local function socket_close(socket) --upval: self, startApp
+    if rawget(self,"socket") then
+      self.socket=nil                        -- remove circular reference in upval 
+      pcall(socket.close,socket)
+      return startApp("Unexpected socket close")
+    end
+  end
+
+  local function receiveFirstRec(socket, rec)  -- upval: self, crypto, startApp, tmr
+    local cmdlen   = (rec:find('\n',1, true) or 0) - 1
+    local cmd,hash = rec:sub(1,cmdlen-6), rec:sub(cmdlen-5,cmdlen)
+    if cmd:find('"r":"OK!"',1,true) or cmdlen < 16 or
+      hash ~= crypto.toHex(crypto.hmac("MD5", cmd, self.secret):sub(-3)) then
+      print "No provisioning changes required"
+      self.socket = nil
+      self.post(function()  --upval: socket
+                  if socket then pcall(socket.close, socket) end 
+                end)
+      return startApp("OK! No further updates needed")
+    end
+    -- Else a valid request has been received from the provision service free up
+    -- some resources that are no longer needed and set backstop timer for general 
+    -- timeout.  This also dereferences the previous doTick cb so it can now be GCed.
+    collectgarbage()
+    tmr.alarm(0, 30000, tmr.ALARM_SINGLE, self.startApp)
+    return self:_provision(socket,rec)
+  end
+
+  local function socket_connect(socket) --upval: self, socket_connect
+    print "Connected to provisioning service"
+    self.socket = socket
+    socket_connect = nil                  -- make this function available for GC
+    socket:on("receive", receiveFirstRec)
+    return self.socket_send(socket, self.config)
+  end
+
+  local conn 
+  return function() -- the proper doTick() timer callback
+    tick_count = tick_count + 1
+    log("entering tick", tick_count, sta.getconfig(false), sta.getip())
+    if (tick_count < 20) then             -- (wait up to 10 secs for Wifi connection)
+      local status, ip = sta.status(),{sta.getip()}
+      if (status == wifi.STA_GOTIP) then
+        log("Connected:", unpack(ip))
+        if (config.nsserver) then
+          net.dns.setdnsserver(config.nsserver, 0)
+        end
+        conn = net.createConnection(net.TCP, 0)
+        conn:on("connection",     socket_connect)
+        conn:on("disconnection",  socket_close)
+        conn:connect(config.port, config.server)
+
+        tick_count = 20
+      end
+
+    elseif (tick_count == 20) then        -- assume timeout and exec app CB
+      return self.startApp("OK: Timeout on waiting for wifi station setup")
+
+    elseif (tick_count == 26) then        -- wait up to 2.5 secs for TCP response
+      tmr.unregister(0)
+      pcall(conn.close, conn)
+      self.socket=nil
+      return startApp("OK: Timeout on waiting for provision service response")
+    end
+  end
+-- end

+ 49 - 0
lua_examples/luaOTA/_init.lua

@@ -0,0 +1,49 @@
+--SAFETRIM
+-- function _init(self, args)
+  local self, args = ...
+  
+  -- The config is read from config.json but can be overridden by explicitly 
+  -- setting the following args.  Setting to "nil" deletes the config arg.
+  -- 
+  --    ssid, spwd                Credentials for the WiFi
+  --    server, port, secret      Provisioning server:port and signature secret
+  --    leave                     If true then the Wifi is left connected
+  --    espip, gw, nm, nsserver   These need set if you are not using DHCP
+
+  local wifi, file, json, tmr = wifi, file, sjson, tmr
+  local log, sta, config = self.log, wifi.sta, nil
+
+  print ("\nStarting Provision Checks")
+  log("Starting Heap:", node.heap())
+
+  if file.open(self.prefix .. "config.json", "r") then
+    local s; s, config = pcall(json.decode, file.read())
+    if not s then print("Invalid configuration:", config) end
+    file.close()
+  end
+  if type(config) ~= "table" then config = {} end
+
+  for k,v in pairs(args or {}) do config[k] = (v ~= "nil" and v) end
+
+  config.id = node.chipid()
+  config.a  = "HI"
+
+  self.config = config
+  self.secret = config.secret
+  config.secret = nil
+
+  log("Config is:",json.encode(self.config))
+  log("Mode is", wifi.setmode(wifi.STATION, false), config.ssid, config.spwd)
+  log("Config status is", sta.config(
+            { ssid = config.ssid, pwd  = config.spwd, auto = false, save = false } ))
+
+  if config.espip then
+    log( "Static IP setup:", sta.setip(
+            { ip = config.espip, gateway = config.gw, netmask = config.nm }))
+  end
+  sta.connect(1)
+
+  package.loaded[self.modname] = nil
+  self.modname=nil
+  tmr.alarm(0, 500, tmr.ALARM_AUTO, self:_doTick()) 
+-- end

+ 125 - 0
lua_examples/luaOTA/_provision.lua

@@ -0,0 +1,125 @@
+--SAFETRIM
+-- function _provision(self,socket,first_rec)
+ 
+local self, socket, first_rec = ...
+local crypto, file, json, node, table = crypto, file, sjson,   node, table
+local stripdebug, gc = node.stripdebug, collectgarbage
+
+local buf = {}
+gc(); gc() 
+
+local function getbuf()  -- upval: buf, table
+  if #buf > 0 then return table.remove(buf, 1) end -- else return nil
+end
+
+-- Process a provisioning request record
+local function receiveRec(socket, rec)  -- upval: self, buf, crypto
+  -- Note that for 2nd and subsequent responses, we assme that the service has 
+  -- "authenticated" itself, so any protocol errors are fatal and lkely to 
+  -- cause a repeating boot, throw any protocol errors are thrown.
+  local buf, config, file, log = buf, self.config, file, self.log
+  local cmdlen = (rec:find('\n',1, true) or 0) - 1
+  local cmd,hash = rec:sub(1,cmdlen-6), rec:sub(cmdlen-5,cmdlen)
+  if cmdlen < 16 or
+     hash ~= crypto.toHex(crypto.hmac("MD5",cmd,self.secret):sub(-3)) then
+    return error("Invalid command signature")
+  end
+
+  local s; s, cmd = pcall(json.decode, cmd)
+  local action,resp = cmd.a, {s = "OK"}
+  local chunk
+
+  if action == "ls" then
+    for name,len in pairs(file.list()) do
+      resp[name] = len
+    end
+
+  elseif action == "mv" then
+    if file.exists(cmd.from) then
+      if file.exists(cmd.to) then file.remove(cmd.to) end
+      if not file.rename(cmd.from,cmd.to) then
+        resp.s = "Rename failed"
+      end
+    end
+
+  else
+    if action == "pu" or action == "cm"  or action == "dl" then
+      -- These commands have a data buffer appended to the received record
+      if cmd.data == #rec - cmdlen - 1 then
+        buf[#buf+1] = rec:sub(cmdlen +2)
+      else
+        error(("Record size mismatch, %u expected, %u received"):format(
+              cmd.data or "nil", #buf - cmdlen - 1))
+      end
+    end
+
+    if action == "cm" then
+      stripdebug(2)
+      local lcf,msg = load(getbuf, cmd.name)
+      if not msg then
+        gc(); gc()
+        local code, name = string.dump(lcf), cmd.name:sub(1,-5) .. ".lc"
+        local s = file.open(name, "w+")
+        if s then
+          for i = 1, #code, 1024 do
+            s = s and file.write(code:sub(i, ((i+1023)>#code) and i+1023 or #code))
+          end
+          file.close()
+          if not s then file.remove(name) end
+        end
+        if s then
+          resp.lcsize=#code
+          print("Updated ".. name)
+        else 
+          msg = "file write failed"
+        end
+     end
+     if msg then
+        resp.s, resp.err  = "compile fail", msg
+     end
+     buf = {}
+
+    elseif action == "dl" then
+      local s = file.open(cmd.name, "w+")
+      if s then
+        for i = 1, #buf do
+           s = s and file.write(buf[i])
+        end
+        file.close()
+      end
+   
+      if s then
+        print("Updated ".. name)
+      else 
+        file.remove(name)
+        resp.s = "write failed"
+      end
+      buf = {}
+   
+    elseif action == "ul" then
+      if file.open(cmd.name, "r") then
+        file.seek("set", cmd.offset)
+        chunk = file.read(cmd.len)
+        file.close()
+      end
+
+    elseif action == "restart" then
+      cmd.a = nil
+      cmd.secret = self.secret
+      file.open(self.prefix.."config.json", "w+")
+      file.writeline(json.encode(cmd))
+      file.close()
+      socket:close()
+      print("Restarting to load new application")
+      node.restart()  -- reboot just schedules a restart
+      return
+    end
+  end
+  self.socket_send(socket, resp, chunk)
+  gc()
+end
+
+-- Replace the receive CB by the provisioning version and then tailcall this to
+-- process this first record. 
+socket:on("receive", receiveRec)
+return receiveRec(socket, first_rec)

+ 66 - 0
lua_examples/luaOTA/check.lua

@@ -0,0 +1,66 @@
+--SAFETRIM
+--------------------------------------------------------------------------------
+-- LuaOTA provisioning system for ESPs using NodeMCU Lua
+-- LICENCE: http://opensource.org/licenses/MIT
+-- TerryE  15 Jul 2017
+--
+-- See luaOTA.md for description and implementation notes
+
+--------------------------------------------------------------------------------
+
+-- upvals
+local crypto, file, json,  net, node, table, tmr, wifi =
+      crypto, file, sjson, net, node, table, tmr, wifi
+local error, pcall   = error, pcall 
+local loadfile, gc   = loadfile, collectgarbage
+local concat, unpack = table.concat, unpack or table.unpack 
+
+local self = {post = node.task.post, prefix = "luaOTA/", conf = {}}
+
+self.log     = (DEBUG == true) and print or function() end 
+self.modname = ...  
+
+--------------------------------------------------------------------------------------
+-- Utility Functions
+
+setmetatable( self, {__index=function(self, func) --upval: loadfile
+    -- The only __index calls in in LuaOTA are dynamically loaded functions. 
+    -- The convention is that functions starting with "_" are treated as 
+    -- call-once / ephemeral; the rest are registered in self 
+    func = self.prefix .. func
+    local f,msg = loadfile( func..".lc")
+    if msg then f, msg = loadfile(func..".lua") end
+    if msg then error (msg,2) end
+    if func:sub(8,8) ~= "_" then self[func] = f end
+    return f
+  end} )
+
+function self.sign(arg)  --upval: crypto, json, self
+  arg = json.encode(arg)
+  return arg .. crypto.toHex(crypto.hmac("MD5", arg, self.secret):sub(-3)) .. '\n'
+end
+
+function self.startApp(arg) --upval: gc, self, tmr, wifi
+  gc();gc()
+  tmr.unregister(0)
+  self.socket = nil
+  if not self.config.leave then wifi.setmode(wifi.NULLMODE,false) end
+  local appMod    = self.config.app   or "luaOTA.default"
+  local appMethod = self.config.entry or "entry"
+  if not arg then arg =  "General timeout on provisioning" end
+  self.post(function() --upval: appMod, appMethod, arg 
+              require(appMod)[appMethod](arg)
+            end)
+end 
+
+function self.socket_send(socket, rec, opt_buffer)
+  return socket:send(self.sign(rec) .. (opt_buffer or ''))
+end
+
+self.post(function() -- upval: self
+            -- This config check is to prevent a double execution if the 
+            -- user invokes with "require 'luaOTA/check':_init( etc>)" form  
+            if not rawget(self, "config") then self:_init() end 
+          end)
+
+return self

+ 1 - 0
lua_examples/luaOTA/config.json

@@ -0,0 +1 @@
+{"leave":0,"port":8266,"ssid":"YourSID","spwd":"YourSSIDpwd","server":"your_server","secret":"yoursecret"}

+ 12 - 0
lua_examples/luaOTA/default.lua

@@ -0,0 +1,12 @@
+-- 
+local function enum(t,log) for k,v in pairs(t)do log(k,v) end end   
+return {entry = function(msg)
+    package.loaded["luaOTA.default"]=nil
+    local gc=collectgarbage; gc(); gc()
+    if DEBUG then 
+      for k,v in pairs(_G) do print(k,v)  end 
+      for k,v in pairs(debug.getregistry()) do print(k,v) end 
+    end
+    gc(); gc() 
+    print(msg, node.heap())
+  end}

+ 254 - 0
lua_examples/luaOTA/luaOTAserver.lua

@@ -0,0 +1,254 @@
+--------------------------------------------------------------------------------
+-- LuaOTA provisioning system for ESPs using NodeMCU Lua
+-- LICENCE: http://opensource.org/licenses/MIT
+-- TerryE  15 Jul 2017
+--
+-- See luaOTA.md for description
+--------------------------------------------------------------------------------
+
+--[[ luaOTAserver.lua - an example provisioning server 
+
+  This module implements an example server-side implementation of LuaOTA provisioning 
+  system for ESPs used the SPI Flash FS (SPIFFS) on development and production modules. 
+
+  This implementation is a simple TCP listener which can have one active provisioning
+  client executing the luaOTA module at a time.  It will synchronise the client's FS
+  with the content of the given directory on the command line.
+
+]]
+
+local socket     = require "socket"
+local lfs        = require "lfs"
+local md5        = require "md5"
+local json       = require "cjson"
+require "etc.strict"  --  see http://www.lua.org/extras/5.1/strict.lua
+
+-- Local functions (implementation see below) ------------------------------------------
+
+local get_inventory     -- function(root_directory, CPU_ID)
+local send_command      -- function(esp, resp, buffer)
+local receive_and_parse -- function(esp)
+local provision         -- function(esp, config, files, inventory, fingerprint)
+local read_file         -- function(fname)
+local save_file         -- function(fname, data)
+local compress_lua      -- function(lua_file)
+local hmac              -- function(data)
+
+-- Function-wide locals (can be upvalues)
+local unpack = table.unpack or unpack
+local concat = table.concat
+local load   = loadstring or load
+local format = string.format
+-- use string % operators as a synomyn for string.format
+getmetatable("").__mod = 
+   function(a, b)
+      return not b and a or 
+       (type(b) == "table" and format(a, unpack(b)) or format(a, b))
+      end
+
+local ESPport    = 8266
+local ESPtimeout = 15
+
+local src_dir    = arg[1] or "."
+
+-- Main process ------------------------ do encapsulation to prevent name-clash upvalues
+local function main ()
+  local server     = assert(socket.bind("*", ESPport))
+  local ip, port   = server:getsockname()
+
+  print("Lua OTA service listening on  %s:%u\n After connecting, the ESP timeout is %u s" 
+     % {ip, port, ESPtimeout})
+
+  -- Main loop forever waiting for ESP clients then processing each request ------------
+
+
+  while true do
+    local esp = server:accept()           -- wait for ESP connection 
+    esp:settimeout(ESPtimeout)            -- set session timeout
+    -- receive the opening request 
+    local config = receive_and_parse(esp)
+    if config and config.a == "HI" then 
+      print ("Processing provision check from ESP-"..config.id)
+      local inventory, fingerprint = get_inventory(src_dir, config.id)
+      -- Process the ESP request
+      if config.chk and config.chk == fingerprint then
+        send_command(esp, {r = "OK!"})       -- no update so send_command with OK
+        esp:receive("*l")                    -- dummy receive to allow client to close
+      else
+        local status, msg = pcall(provision, esp, config, inventory, fingerprint)
+        if not status then print (msg) end
+  
+    end
+    end
+    pcall(esp.close, esp)
+    print ("Provisioning complete")
+  end
+end
+
+-- Local Function Implementations ------------------------------------------------------
+
+local function get_hmac_md5(key)
+  if key:len() > 64 then 
+    key = md5.sum(key) 
+  elseif key:len() < 64 then
+    key = key .. ('\0'):rep(64-key:len())
+  end
+  local ki = md5.exor(('\54'):rep(64),key)
+  local ko = md5.exor(('\92'):rep(64),key)
+  return function (data) return md5.sumhexa(ko..md5.sum(ki..data)) end
+end
+
+-- Enumerate the sources directory and load the relevent inventory
+------------------------------------------------------------------
+get_inventory = function(dir, cpuid)
+  if (not dir or lfs.attributes(dir).mode ~= "directory") then
+    error("Cannot open directory, aborting %s" % arg[0], 0)
+  end
+
+  -- Load the CPU's (or the default) inventory 
+  local invtype, inventory = "custom", read_file("%s/ESP-%s.json" % {dir, cpuid})  
+  if not inventory then
+    invtype, inventory = "default", read_file(dir .. "/default.json")
+  end
+
+  -- tolerate and remove whitespace formatting, then decode
+  inventory = (inventory or ""):gsub("[ \t]*\n[ \t]*","")
+  inventory = inventory:gsub("[ \t]*:[ \t]*",":")
+  local ok; ok,inventory = pcall(json.decode, inventory)
+  if ok and inventory.files then
+    print( "Loading %s inventory for ESP-%s" % {invtype, cpuid})
+  else
+    error( "Invalid inventory for %s :%s" % {cpuid,inventory}, 0)
+  end
+
+  -- Calculate the current fingerprint of the inventory
+  local fp,f = {},inventory.files
+  for i= 1,#f do
+    local name, fullname = f[i], "%s/%s" % {dir, f[i]}
+    local fa = lfs.attributes(fullname)
+
+    assert(fa, "File %s is required but not in sources directory" % name)
+    fp[#fp+1] = name .. ":" .. fa.modification
+    f[i] = {name = name, mtime = fa.modification, 
+            size = fa.size, content = read_file(fullname) }
+    assert (f[i].size == #(f[i].content or ''), "File %s unreadable" % name )
+  end
+
+  assert(#f == #fp, "Aborting provisioning die to missing fies",0)
+  assert(type(inventory.secret) == "string", 
+         "Aborting, config must contain a shared secret")
+  hmac = get_hmac_md5(inventory.secret) 
+  return inventory, md5.sumhexa(concat(fp,":"))
+end
+
+
+-- Encode a response buff, add a signature and any optional buffer
+------------------------------------------------------------------
+send_command = function(esp, resp, buffer)
+  if type(buffer) == "string" then 
+    resp.data = #buffer 
+  else 
+    buffer = ''
+  end
+  local rec = json.encode(resp)
+  rec = rec .. hmac(rec):sub(-6) .."\n"
+-- print("requesting ", rec:sub(1,-2), #(buffer or ''))
+  esp:send(rec .. buffer)  
+end
+
+
+-- Decode a response buff, check the signature and any optional buffer
+----------------------------------------------------------------------
+receive_and_parse = function(esp)
+  local line = esp:receive("*l")
+  local packed_cmd, sig = line:sub(1,#line-6),line:sub(-6)
+-- print("reply:", packed_cmd, sig)
+  local status, cmd = pcall(json.decode, packed_cmd)
+  if not hmac or hmac(packed_cmd):sub(-6) == sig then
+    if cmd and cmd.data == "number" then
+      local data = esp:receive(cmd.data)
+      return cmd, data
+    end
+    return cmd
+  end
+end
+
+      
+provision = function(esp, config, inventory, fingerprint)
+
+  if type(config.files) ~= "table" then config.files = {} end 
+  local cf = config.files
+
+  for _, f in ipairs(inventory.files) do
+    local name, size, mtime, content = f.name, f.size, f.mtime, f.content
+    if not cf[name] or cf[name] ~= mtime then
+      -- Send the file
+      local func, action, cmd, buf
+      if  f.name:sub(-4) == ".lua" then
+        assert(load(content, f.name))  -- check that the contents can compile
+        if content:find("--SAFETRIM\n",1,true) then
+          -- if the source is tagged with SAFETRIM then its safe to remove "--"
+          -- comments, leading and trailing whitespace.  Not as good as LuaSrcDiet,
+          -- but this simple source compression algo preserves line numbering in
+          -- the generated lc files, which helps debugging. 
+          content = content:gsub("\n[ \t]+","\n")
+          content = content:gsub("[ \t]+\n","\n")
+          content = content:gsub("%-%-[^\n]*","")
+          size = #content
+        end
+        action = "cm"
+      else
+        action = "dl"
+      end
+      print ("Sending file ".. name)
+    
+      for i = 1, size, 1024 do
+        if i+1023 < size then
+          cmd = {a = "pu", data = 1024}
+          buf = content:sub(i, i+1023)
+        else
+          cmd = {a = action, data = size - i + 1, name = name}
+          buf = content:sub(i)
+        end 
+        send_command(esp, cmd, buf)
+        local resp = receive_and_parse(esp)
+        assert(resp and resp.s == "OK", "Command to ESP failed")
+        if resp.lcsize then 
+          print("Compiled file size %s bytes" % resp.lcsize)
+        end
+      end
+    end
+
+    cf[name] = mtime
+  end
+
+  config.chk = fingerprint
+  config.id  = nil
+  config.a   = "restart"
+  send_command(esp, config)
+
+end
+
+-- Load contents of the given file (or null if absent/unreadable)
+-----------------------------------------------------------------
+read_file = function(fname)
+  local file = io.open(fname, "rb")
+  if not file then return end
+  local data = file and file:read"*a"
+  file:close()
+  return data
+end
+
+-- Save contents to the given file
+----------------------------------
+save_file = function(fname, data)
+  local file = io.open(fname, "wb")
+  file:write(data) 
+  file:close()
+end
+
+--------------------------------------------------------------------------------------
+
+main()  -- now that all functions have been bound to locals, we can start the show :-)
+
+