This FAQ was started by Terry Ellison as an unofficial FAQ in mid 2015. This version as at April 2017 includes some significant rewrites.
This FAQ does not aim to help you to learn to program or even how to program in Lua. There are plenty of resources on the Internet for this, some of which are listed in Where to start. What this FAQ does is to answer some of the common questions that a competent Lua developer would ask in learning how to develop Lua applications for the ESP8266 based boards running the NodeMcu firmware. This includes the NodeMCU Devkits. However, the scope of the firmware is far wider than this as it can be used on any ESP8266 module.
The NodeMCU company was set up by Zeroday to develop and to market a set of Lua firmware-based development boards which employ the Espressif ESP8266 SoC. The initial development of the firmware was done by Zeroday and a colleague, Vowstar, in-house with the firmware being first open-sourced on Github in late 2014. In mid-2015, Zeroday decided to open the firmware development to a wider group of community developers, so the core group of developers now comprises 6 community developers (including this author), and we are also supported by another dozen or so active contributors, and two NodeMCU originators.
This larger active team has allowed us to address most of the outstanding issues present at the first version of this FAQ. These include:
Because the development is active this list will no doubt continue to be revised and updated. See the development README for more details.
The NodeMCU firmware implements Lua 5.1 over the Espressif SDK for its ESP8266 SoC and the IoT modules based on this.
Whilst the Lua standard distribution includes a stand-alone Lua interpreter, Lua itself is primarily an extension language that makes no assumptions about a «main» program: Lua works embedded in a host application to provide a powerful, lightweight scripting language for use within the application. This host application can then invoke functions to execute a piece of Lua code, can write and read Lua variables, and can register C functions to be called by Lua code. Through the use of C functions, Lua can be augmented to cope with a wide range of different domains, thus creating customized programming languages sharing a syntactical framework.
The ESP8266 was designed and is fabricated in China by Espressif Systems. Espressif have also developed and released a companion software development kit (SDK) to enable developers to build practical IoT applications for the ESP8266. The SDK is made freely available to developers in the form of binary libraries and SDK documentation. However this is in a closed format, with no developer access to the source files, so anyone developing ESP8266 applications must rely solely on the SDK API (and the somewhat Spartan SDK API documentation). (Note that for the ESP32, Espressif have moved to an open-source approach for its ESP-IDF.)
The NodeMCU Lua firmware is an ESP8266 application and must therefore be layered over the ESP8266 SDK. However, the hooks and features of Lua enable it to be seamlessly integrated without losing any of the standard Lua language features. The firmware has replaced some standard Lua modules that don’t align well with the SDK structure with ESP8266-specific versions. For example, the standard io
and os
libraries don’t work, but have been largely replaced by the NodeMCU node
and file
libraries. The debug
and math
libraries have also been omitted to reduce the runtime footprint (modulo
can be done via %
, power
via ^
).
NodeMCU Lua is based on eLua, a fully featured implementation of Lua 5.1 that has been optimized for embedded system development and execution to provide a scripting framework that can be used to deliver useful applications within the limited RAM and Flash memory resources of embedded processors such as the ESP8266. One of the main changes introduced in the eLua fork is to use read-only tables and constants wherever practical for library modules. On a typical build this approach reduces the RAM footprint by some 20-25KB and this makes a Lua implementation for the ESP8266 feasible. This technique is called LTR and this is documented in detail in an eLua technical paper: Lua Tiny RAM.
The main impacts of the ESP8266 SDK and together with its hardware resource limitations are not in the Lua language implementation itself, but in how application programmers must approach developing and structuring their applications. As discussed in detail below, the SDK is non-preemptive and event driven. Tasks can be associated with given events by using the SDK API to registering callback functions to the corresponding events. Events are queued internally within the SDK, and it then calls the associated tasks one at a time, with each task returning control to the SDK on completion. The SDK states that if any tasks run for more than 15 mSec, then services such as WiFi can fail.
The NodeMCU libraries act as C wrappers around registered Lua callback functions to enable these to be used as SDK tasks. You must therefore use an Event-driven programming style in writing your ESP8266 Lua programs. Most programmers are used to writing in a procedural style where there is a clear single flow of execution, and the program interfaces to operating system services by a set of synchronous API calls to do network I/O, etc. Whilst the logic of each individual task is procedural, this is not how you code up ESP8266 applications.
core
, coroutine
, string
and table
are implemented.file
library. The SPIFFS file system is designed for SPI NOR flash devices on embedded targets, and is optimised for static wear levelling and low RAM footprint. For further details, see the link. How much Flash is available as SPIFFS file space depends on the number of modules included in the specific firmware build.debug
library support. So you have to use 1980s-style «binary-chop» to locate errors and use print statement diagnostics though the system’s UART interface. (This omission was largely because of the Flash memory footprint of this library, but there is no reason in principle why we couldn’t make this library available in the near future as a custom build option).function table.pack()
will cause a runtime error because you can’t write to the global table
. Standard sand-boxing techniques can be used to achieve the same effect by using metatable based inheritance, but if you choose this option, then you need to be aware of the potential runtime and RAM impacts of this approach.init.lua
script. It then «listens» to the serial port for input Lua chunks, and executes them once syntactically complete.init.lua
script.net
, tmr
, wifi
, etc.) use the SDK callback mechanism to bind Lua processing to individual events (for example a timer alarm firing). Developers should make full use of these events to keep Lua execution sequences short.socket:send()
are on consecutive lines in a Lua programme, then the first has completed by the time the second is executed. This is wrong. A socket:send()
request simply queues the send task for dispatch by the SDK. This task can’t start to process until the Lua code has returned to is calling C function to allow this running task to exit. Stacking up such requests in a single Lua task function burns scarce RAM and can trigger a PANIC. This is true for timer, network, and other callbacks. It is even the case for actions such as requesting a system restart, as can be seen by the following example which will print twenty «not quite yet» messages before restarting.node.restart(); for i = 1, 20 do print("not quite yet -- ",i); end
tmr.wdclr()
function, but this should be avoided.In essence, the NodeMCU firmware is a C application which exploits the ability of Lua to execute as a embedded language and runtime to mirror this structure at a Lua scripting level. All of the complexities of, and interface to, the SDK and the hardware are wrapped in firmware libraries which translate the appropriate calls into the corresponding Lua API.
init.lua
from the SPIFFS file system. This init.lua
module can then be used to do any application initialisation required and to call the necessary timer alarms or library calls to bind and callback routines to implement the tasks needed in response to any system events.mytimer:alarm(interval, repeat, callback)
calls a function in the tmr
library which registers a C function for this alarm using the SDK, and when this C alarm callback function is called it then in turn invokes the Lua callback.This event-driven approach is very different to a conventional procedural applications written in Lua, and different from how you develop C sketches and applications for the Arduino architectures. There is little point in constructing poll loops in your NodeMCU Lua code since almost always the event that you are polling will not be delivered by the SDK until after your Lua code returns control to the SDK. The most robust and efficient approach to coding ESP8266 Lua applications is to embrace this event model paradigm, and to decompose your application into atomic tasks that are threaded by events which themselves initiate callback functions. Each event task is established by a callback in an API call in an earlier task.
Understanding how the system executes your code can help you structure it better and improve both performance and memory usage.
If you are not using timers and other callback, then you are using the wrong approach.
If you are using poll loops, then you are using the wrong approach.
If you are executing more an a few hundred lines of Lua per callback, then you are using the wrong approach.
SDK Callbacks include:
Lua Module | Functions which define or remove callbacks |
---|---|
tmr | register([id,] interval, mode, function()) |
node | task.post([task_priority], function) , output(function(str), serial_debug) |
wifi | startsmart(chan, function()) , sta.getap(function(table)) |
net.server | sk:listen(port,[ip],function(socket)) |
net | sk:on(event, function(socket, [, data])) , sk:send(string, function(sent)) , sk:dns(domain, function(socket,ip)) |
gpio | trig(pin, type, function(level)) |
mqtt | client:m:on(event, function(conn[, topic, data]) |
uart | uart.on(event, cnt, [function(data)], [run_input]) |
For a comprehensive list refer to the Module documentation on this site.
The following is all standard Lua and is explained in detail in PiL etc., but it is worth summarising here because understanding this is of particular importance in the NodeMCU environment.
All variables in Lua can be classed as globals, locals or upvalues. But by default any variable that is referenced and not previously declared as local
is global and this variable will persist in the global table until it is explicitly deleted. If you want to see what global variables are in scope then try
for k,v in pairs(_G) do print(k,v) end
a
to b
. In the case of a value then it is simply copied into b
. In the case of a reference, both a
and b
now refer to the same object, and no copying of content takes place. This process of referencing can have some counter-intuitive consequences. For example, in the following code by the time it exists, the variable timer2func
is out of scope. However a reference to the function has now been stored in the Lua registry by the alarm API call, so it and any upvalues that it uses will persist until it is eventually entirely dereferenced (e.g. by tmr2:unregister()
.
Lua
do
local tmr2func = function() ds.convert_T(true); tmr1:start() end
tmr2:alarm(300000, tmr.ALARM_AUTO, tmr2func)
end
--
You need to understand the difference between when a function is compiled, when it is bound as a closure and when it is invoked at runtime. The closure is normally bound once pretty much immediately after compile, but this isn’t necessarily the case. Consider the following example from my MCP23008 module below. ```Lua – Bind the read and write functions for commonly accessed registers
for reg, regAddr in pairs {
IODOR = 0x00,
GPPU = 0x06, -- Pull-up resistors register for MCP23008
GPIO = 0x09,
OLAT = 0x0A,
} do
dev['write'..reg] = function(o, dataByte)
write(MCP23008addr, regAddr, dataByte)
end
dev['read'..reg] = function(o)
return read(MCP23008addr, regAddr)
end
end
* This loop is compiled once when the module is required. The opcode vectors for the read and write functions are created during the compile, along with a header which defines how many upvalues and locals are used by each function. However, these two functions are then bound _four_ times as different functions (e.g. `mcp23008.writeIODOR()`) and each closure inherits its own copies of the upvalues it uses so the `regAddr` for this function is `0x00`). The upvalue list is created when the closure is created and through some Lua magic, even if the outer routine that initially declared them is no longer in scope and has been GCed (Garbage Collected), the Lua RTS ensures that any upvalue will still persist whilst the closure persists.
* On the other hand the storage for any locals is allocated each time the routine is called, and this can be many times in a running application.
* The Lua runtime uses hashed key access internally to retrieve keyed data from a table. On the other hand locals and upvalues are stored as a contiguous vector and are accessed directly by an index, which is a lot faster. In NodeMCU Lua accesses to Firmware-based tables is particularly slow, which is why you will often see statements like the following at the beginning of modules. *Using locals and upvalues this way is both a lot faster at runtime and generates less bytecode instructions for their access.*
```Lua
local i2c = i2c
local i2c_start, i2c_stop, i2c_address, i2c_read, i2c_write, i2c_TRANSMITTER, i2c_RECEIVER =
i2c.start, i2c.stop, i2c.address, i2c.read, i2c.write, i2c.TRANSMITTER, i2c.RECEIVER
lua_call()
. Even system initialisation which executes the dofile("init.lua")
is really a special case of this. Each function can invoke other functions and so on, but it must ultimately return control to the C library code which then returns control the SDK, terminating the task.local
variables only exist within the context of an executing Lua function, and so locals are unreferenced on exit and any local data (unless also a reference type such as a function, table, or user data which is also referenced elsewhere) can therefore be garbage collected between these lua_call()
actions.So context can only be passed between event routines by one of the following mechanisms:
nil
to it. Globals can be readily enumerated, e.g. by a for k,v in pairs(_G) do
, so their use is transparent.All Lua callbacks are called by C wrapper functions within the NodeMCU libraries that are themselves callbacks that have been activated by the SDK as a result of a given event. Such C wrapper functions themselves frequently need to store state for passing between calls or to other wrapper C functions. The Lua registry is a special Lua table which is used for this purpose, except that it is hidden from direct Lua access, but using a standard Lua table for this store enables standard garbage collection algorithms to operate on its content. Any content that needs to be saved is created with a unique key. The upvalues for functions that are global or referenced in the Lua Registry will persist between event routines, and hence any upvalues used by them will also persist and can be used for passing context.
on()
function passes the socket to the connection callback as it’s first argument sck
. This is local variable in the callback function, and it also references the same socket as the upvalue srv
. So functionally srv
and sck
are interchangeable. So why pass it as an argument? Normally garbage collecting a socket will automatically unregister any of its callbacks, but if you use a socket as an upvalue in the callback, the socket is now referenced through the Register, and now it won’t be GCed because it is referenced. Catch-22 and a programming error, not a bug.srv:on("connection", function(sck, c)
svr:send(reply)
end)
for k,v in pairs(debug.getregistry()) do print (k,v) end
to track the registry size. If this is growing then you’ve got a leak.See the Unofficial LUA FAQ: Detecting Undefined Variables.
My approach is to avoid using them unless I have a very good reason to justify this. I track them statically by running a luac -p -l XXX.lua | grep GLOBAL
filter on any new modules and replace any accidental globals by local or upvalued local declarations.
On NodeMCU, _G’s metatable is _G, so you can create any globals that you need and then ‘close the barn door’ by assigning
_G.__newindex=function(g,k,v) error ("attempting to set global "..k.." to "..v) end
and any attempt to create new globals with now throw an error and give you a traceback of where this has happened.
The use of upvalues is a core Lua feature. This is explained in detail in PiL. Any Lua routines defined within an outer scope my use them. This can include routines directly or indirectly referenced in the globals table, _G, or in the Lua Registry.
The number of upvalues associated with a given routine is calculated during compile and a stack vector is allocated for them when the closure is bound to hold these references. Each upvalues is classed as open or closed. All upvalues are initially open which means that the upvalue references back to the outer function’s register set. However, upvalues must be able to outlive the scope of the outer routine where they are declared as a local variable. The runtime VM does this by adding extra checks when executing a function return to scan any defined closures within its scope for back references and allocate memory to hold the upvalue and points the upvalue’s reference to this. This is known as a closed upvalue.
This processing is a mature part of the Lua 5.x runtime system, and for normal Lua applications development this «behind-the-scenes» magic ensures that upvalues just work as any programmer might expect. Sufficient garbage collector metadata is also stored so that these hidden values will be garbage collected correctly when properly dereferenced.
One further complication is that some library functions don’t implicitly dereference expired callback references and as a result their upvalues may not be garbage collected and this application error can be be manifested as a memory leak. So using upvalues can cause more frequent and difficult to diagnose PANICs during testing. So my general recommendation is still to stick to globals during initial development, and explicitly dereference resources by setting them to nil
when you have done with them.
Think about the implications of these last few answers.
-- prepare message
status = mail.send(to, subject, body)
-- move on to next phase of processing.
-- prepare message
local ms = require("mail_sender")
return ms.send(to, subject, body, function(status)
loadfile("process_next.lua")(status)
end)
If you are used coding in a procedural paradigm then it is understandable that you consider using tmr.delay()
to time sequence your application. However as discussed in the previous section, with NodeMCU Lua you are coding in an event-driven paradigm.
If you look at the app/modules/tmr.c
code for this function, then you will see that it executes a low level ets_delay_us(delay)
. This function isn’t part of the NodeMCU code or the SDK; it’s actually part of the xtensa-lx106 boot ROM, and is a simple timing loop which polls against the internal CPU clock. tmr.delay()
is really intended to be used where you need to have more precise timing control on an external hardware I/O (e.g. lifting a GPIO pin high for 20 μSec). It does this with interrupts enabled, because so there is no guarantee that the delay will be as requested, and the Lua RTS itself may inject operations such as GC, so if you do this level of precise control then you should encode your application as a C library.
It will achieve no functional purpose in pretty much every other usecase, as any other system code-based activity will be blocked from execution; at worst it will break your application and create hard-to-diagnose timeout errors. We therefore deprecate its general use.
Most of us have fallen into the trap of creating an init.lua
that has a bug in it, which then causes the system to reboot and hence gets stuck in a reboot loop. If you haven’t then you probably will do so at least once.
init.lua
as simple as possible – say configure the wifi and then start your app using a one-time tmr.alarm()
after a 2-3 sec delay. This delay is long enough to issue a file.remove("init.lua")
through the serial port and recover control that way.init.lua
by creating it as init_test.lua
, say, and manually issuing a dofile("init_test.lua")
through the serial port, and then only rename it when you are certain it is working as you require.See «Uploading code» → init.lua for an example.
We recommend that you install Lua 5.1 on your delopment host. This often is useful for debugging Lua fragments on your PC. You also use it for compile validation.
You can also build luac.cross
on your development host if you have Lua locally installed. This runs on your host and has all of the features of standard luac
, except that the output code file will run under NodeMCU as an lc file.
node.compile()
to pre-compile any production code will remove all compiled code including error line info and so is not recommended except for stable production code where line numbers are not needed.nil
dereferences the previous context of that variable. (Note that reference-based variables such as tables, strings and functions can have multiple variables referencing the same object, but once the last reference has been set to nil
, the collector will recover the storage.node.egc.setmode(node.egc.ON_MEM_LIMIT, 4096)
is a good compromise of performance and having enough free headboard.require()
library function creates a reference for the loaded module in the package.loaded
table, and this reference prevents the module from being garbage collected. To make a module volatile, you should remove this reference to the loaded module by setting its corresponding entry in package.loaded
to nil
. You can’t do this in the outermost level of the module (since the reference is only created once execution has returned from the module code), but you can do it in any module function, and typically an initialisation function for the module, as in the following example:local s=net.createServer(net.TCP)
s:listen(80,function(c) require("connector").init(c) end)
connector.lua
would be a standard module pattern except that the M.init()
routine must include the lineslocal M, module = {}, ...
...
function M.init(csocket)
package.loaded[module]=nil
...
end
--
return M
require()
will automatically search for connector.lc
followed by connector.lua
, so the code will work for both source and compiled variants.--
local s=net.createServer(net.TCP)
s:listen(80,function(c) require("connector")(c) end)
local module = _ -- this is a situation where using an upvalue is essential!
return function (csocket)
package.loaded[module]=nil
module = nil
...
end
...
local s=net.createServer(net.TCP)
local connector = require("connector") -- don't do this unless you've got the RAM available!
s:listen(80,connector)
Note that there are two methods of saving compiled Lua to SPIFFS:
node.compile()
on the .lua
source file, which generates the equivalent bytecode .lc
file. This approach strips out all the debug line and variable information.loadfile()
to load the source file into memory, followed by string.dump()
to convert it in-memory to a serialised load format which can then be written back to a .lc
file. The amount of debug saved will depend on the node.stripdebug() settings.The memory footprint of the bytecode created by method (2) is the same as when executing source files directly, but the footprint of bytecode created by method (1) is typically 10% smaller than a dump with the stripdebug level of 2 or 60% smaller than a dump with a stripdebug level of 0, because the debug information is almost as large as the code itself.
In general consider method (1) if you have stable production code that you want to run in as low a RAM footprint as possible. Yes, method (2) can be used if you are still debugging, but you will probably be changing this code quite frequently, so it is easier to stick with .lua
files for code that you are still developing.
Note that if you use require("XXX")
to load your code then this will automatically search for XXX.lc
then XXX.lua
so you don’t need to include the conditional logic to load the bytecode version if it exists, falling back to the source version otherwise.
luac
against your source on your PC with the -l -s
option will give you a good idea of what your code will generate. The main difference between these two variants is the size_t for ESP8266 is 4 bytes rather than the 8 bytes size_t found on modern 64bit development PCs; and the eLua variants generate different access references for ROM data types. If you want to see what the string.dump()
version generates then drop the -s
option to retain the debug information. You can also build luac.cross
with this firmware and this generate lc code for the target ESP architecture..lc
files to the PC and disassemble them there. There are a number of Lua code disassemblers which can list off the compiled code that your application modules will generate, if
you have a script to upload files from your ESP8266 to your development PC. I use ChunkSpy which can be downloaded here , but you will need to apply the following patch so that ChunkSpy understands eLua data types: --- a/ChunkSpy-0.9.8/5.1/ChunkSpy.lua 2015-05-04 12:39:01.267975498 +0100
+++ b/ChunkSpy-0.9.8/5.1/ChunkSpy.lua 2015-05-04 12:35:59.623983095 +0100
@@ -2193,6 +2193,9 @@
config.AUTO_DETECT = true
elseif a == "--brief" then
config.DISPLAY_BRIEF = true
+ elseif a == "--elua" then
+ config.LUA_TNUMBER = 5
+ config.LUA_TSTRING = 6
elseif a == "--interact" then
perform = ChunkSpy_Interact
node.heap()
regularly through your code.Functions have fixed overheads, so in general the more that you group your application code into larger functions, then the less RAM used will be used overall. The main caveat here is that if you are starting to do «copy and paste» coding across functions then you are wasting resources. So of course you should still use functions to structure your code and encapsulate common repeated processing, but just bear in mind that each function definition has a relatively high overhead for its header record and stack frame. So try to avoid overusing functions. If there are less than a dozen or so lines in the function then you should consider putting this code inline if it makes sense to do so.
luac
to generate a bytecode listing of your code and to validate new code syntactically before downloading to the ESP8266. This will also allow you to develop server-side applications and embedded applications in a common language.