ESP8266 SPI Spy

I came across a very useful post by Thomas Scherrer that describes how to read data from a Peacefair PZEM-021 energy meter by spying on the SPI bus with an Arduino. I decided to do the same thing with an ESP-12F WiFi module so that I could view the results remotely and plot graphs, etc. It took me a lot longer to get this working than I anticipated due to a few problems along the way.

The main hardware difference is the ESP8266 is a 3.3V device but the Arduino is 5V. The PZEM-021 is actually a mixture. The RN8208G metering chip is a 5V device. It is a SPI slave, the SPI master is an STM32 ARM processor that is 3.3V but with 5V tolerant inputs. That means the signals originating from the CPU can go straight into the ESP8266 but the data out from the RN8208G would need attenuating.

Copying Thomas, I removed the mains dropper components C1 and R1 and powered the PZEM-021 from an external supply. This allows it to measure voltages right down to zero instead of cutting out at 80V. I used this little 12V 1A supply recommended by Big Clive on YouTube.

I powered the ESP-12F from the same supply with a tiny 3.3V MP2307 buck converter module.

WARNING: when connected to PZEM-021 the 0V rail is at mains neutral potential. This has to be treated with the same precautions as live because if the mains lead happened to be be reversed or the neutral connection broke it would become live. Don't connect a USB programmer or a scope ground to the circuit unless you power it with a mains isolation transformer. Fortunately the ESP8266 can be programmed wirelessly from the Arduino IDE once a sketch with ArduinoOTA has been installed and this is much faster than USB.

During the hardware development I used a mains isolation unit that I made after watching this video by Paul Carlson, another of my favourite YouTubers. I built this originally to allow me to repair a switch mode PSU that was part of a friend's home cinema unit. It allows me to connect my scope to the live part of a circuit but then of course it is no longer isolated, so great care has to be taken not to touch the high voltage bits.

It is a DiBond box with 3D printed frame, handles and rubber feet containing an isolation transformer and a small variac. It also has two 100W light bulbs in parallel with a bypass switch for optional current limiting, a different model of power meter and some 4mm jack sockets allowing me to attach my Mooshimeter to log voltage and current.

The first task was to solder wires on to the PZEM-021 PCB to bring power in and data out to my circuit.

I didn't see a need for the chip select / word latch signal and I took the data from the other end of R9 compared to Thomas. The signals are as follows:

• Red is 12V, black is ground. 
• The word latch going to the display from chip U2 pin 7 on the left, used for synchronisation. 
• MISO data coming from the energy chip via R9.
• The SPI clock coming from U2 pin 12.

The two wires on the chip are tricky to solder due to the fine pitch, which is much smaller than a practical soldering iron bit. I used the following technique:

• Strip some wire-wrap wire and trim the end to be the length of the flat part of the pin.
• Tin it. It doesn't hold much solder due to being such a small radius.
• Add plenty of liquid flux around the pin.
• Line the wire up along the top of the pin with an Andonstar ADSM201 microscope camera.
• Put a very small amount of solder on a small chisel bit.
• Press the iron on top of the wire and leave it long enough to conduct the heat through the wire to boil the flux and melt the solder on the pad below.

The amount of solder on the iron bit is crucial. If you use too much it will bridge the pins. To remove a bridge add plenty of flux and wipe the pin with a large flat bit. Surface tension will cause some solder to wick onto the bit, wipe it off and repeat until the bridge breaks.

When I looked at the levels of the signals I got a bit of a surprise. I was expecting MISO on channel 2 to go to 5V but the other two come from the ARM and should be 3.3V signals.

They seem to have a 20kHz ripple that takes them up to 4V. I looked back at Thomas' oscilloscope pictures and see the same ripple there. It doesn't make any difference for him because he is using a 5V chip but I didn't want to stuff 4V into my ESP8266!

On further investigation I found that the whole 3.3V rail had this ripple on it. It comes from a Holtek HT7133-1 3.3V LDO regulator. The datasheet for that suggests 10uF decoupling capacitors on the input and output. The circuit has what looks like a 10uF tant on its input but the output decoupler is a tiny MLCC that looks too small to be 10uF. I added a 10uF electrolytic in parallel and that killed the oscillation. It is the blue radial cap across C1 in the photo above.

That just left the issue of the 5V signal to contend with. R9 turns out to be 1K so simply adding a 2K2 to ground drops the signal low enough.

It also changes the first eight bits from FF to 00. This is actually when the command byte is being sent to the meter chip on MOSI and MISO is tri-state, so it doesn't matter.

Hardware done, on to the firmware. Before starting the project I had seen that the ESP8266 has a spare SPI port available and I assumed it would be straightforward to just read a stream of 13 bytes, wrong! It wasn't because the SPI port it a relatively complex device with dozens of registers that doesn't just send and receive bytes. It actually works at the command level, expecting to send or receive a command and address and then send or receive status or data bytes.

When in host mode it is possible to configure it to send and receive arbitrary bytes, indeed that is what the Arduino SPI class does and it works on the ESP8266 more or less the same as it does on an AVR. In slave mode though it has to receive a command and address according to the technical manual and the command defines what happens next. The relevant sections are:

4.3.2. Communication Format Supported by Slave SPISlave ESP8266SPI communication format is almost the same as that of the master mode, i.e. command+address+read/write data, but the slave read/write operation has itshardware command and undeletable address, which is,

• Command: a must; length: 3 ~ 16 bits; master output and slave input (MOSI).

• Address: a must; length: 1 ~ 32 bits; master output and slave input (MOSI).

• Read/write data: optional; length: 0 ~ 512 bits (64 Bytes); master output and slave input (MOSI) or master input and slave output (MISO).

4.3.3.Command Definition Supported by Slave SPIThe length of slave receiving command should at least be 3 bits. For low 3 bits, there are hardware reading and writing operation, which is,

• 010 (slave receiving) : Write the data sent by master into the register of slave data caching via MOSI, i.e. SPI_FLASH_C0 to SPI_FLASH_C15.

• 011 (slave sending):Send the data in the register of slave data caching (from SPI_FLASH_C0 to SPI_FLASH_C15) to master via MOSI.

• 110 (slave receiving and sending): Send slave data caching to MISO and write the master data in MOSI into data caching SPI_FLASH_C0 to SPI_FLASH_C15.

On the face of it this looks like it will not work in this application, which is unbelievable that a SPI port can't read arbitrary data as a slave and is hard coded for emulating flash chips and the like. 

I couldn't find any proper register level documentation for the ESP8266 other than various .h files on the web. The Arduino library uses shortened names which are very cryptic but some headers use longer names. While searching for these I came across the technical manual for the ESP32 and realised it has SPI ports that are nearly the same. It also has register level documentation in it's technical manual so I was able to get a slightly better understanding. After a lot of experimentation I found a solution.

The first n bits get interpreted as a command, where n is 3 or more and defaults to 8. The start of the 13 byte sequence that I want to receive is always zero now that I added the pull down resistor. So it always receives command zero. The bottom three bits normally control what happens next and zero defaults to read status. However, it is possible to override this with user definable commands by setting  SPI_SLV_CMD_DEFINE in SPI_SLAVE_REG. That makes the SPI_SLAVE3_REG define what the command values are for four different actions. By setting SPI_SLV_WRBUF_CMD_VALUE to zero and the other three values to nonzero I was able to make it interpret the command as write buffer, so the rest of the data gets written into the buffer.

Well that was the theory but I was missing two bytes at the beginning. This was because it was being interpreted as an address. The default address length is 24 bits. I changed it to 8 by setting SPI_USER1_REG and now I get the 12 bytes of data following the zero byte. Now this doesn't make sense because if the first 8 bits are being interpreted as an address where is the command coming from? The command length is set by  SPI_USER2_REG and it defaults to 8. I was expecting to have to set it to say four and set the address length to four bits so that they could share the first byte but any value between 2 and 8 seems to work and makes no difference. I don't know if it is a bug in the chip or I don't understand something but I get to read my 12 bytes of data

So in conclusion you can't read a completely arbitrary SPI data stream but you can if the first few bits are a known value.

Here is my code that sets up the HSPI:

When the data is received I get an interrupt where I copy it from the HSPI's dword buffer into a byte buffer in RAM.

I set up the sync pin to generate an interrupt on the falling edge. This resets the HSPI and unmarshals  the data:

I found that when I switched the load on and off I sometimes got a data packet shifted right one bit. That points to a spike on the SPI clock line into the ESP8266 but I could not find one with a 100MHz scope triggered on the mains switching edge. I added a snubber close to the switch but that didn't cure it, so I bodged it in the firmware and moved on. If a voltage sample is not more than half the previous one I ignore it once.

Another oddity is that when there is no load and the voltage is relatively high the power reading would go slightly negative. I detect that and set it to zero.

The foreground loop looks for the new_readings flag to be set and does the conversion to real units:

The conversion constants are close to what Thomas used but slightly different, so the ARM must have some per unit factory calibration constants in it.

So it took me longer than I anticipated to follow in Thomas' footsteps because of the complexity and lack of proper documentation for the ESP8266 HSPI peripheral but I got there in the end. I hope others find this as useful as I found Thomas' work.

By now you are probably wondering what this is all for. Well, while investigating a strange line regulation problem in a power supply I got annoyed that the mains voltage in our house is constantly varying, making it hard to make measurements of input versus output. This is what it looks like logged every 10 seconds over a couple of days using my Mooshimeter.

As well as the small modern variac shown above I have a old WM5 model that I got on eBay. The only data I could find on it was from 1955. It still works fine so I decided to automate it with a small stepper motor to maintain a specified voltage. I.e. make a WiFi control IOT variac.

Another DiBond and printed part creation but the front panel had to be acrylic to let the WiFi out. Fortunately the front of my bench faces towards my router. The ESP8266 controls the motor via a Pololu stepper driver.

The control interface is a simple web form that also shows the current readings using AJAX as described here.

The control algorithm is very simple. The motor is stepped at the specified speed by the error in voltage multiplied by the gain. The readings update every 200ms but they lag a lot. That causes overshoot if the gain is set high. It would probably benefit from a full PID controller but it works well enough for now. The deadband setting allows it to stay in a range without constantly moving because I don't want to wear out the variac. Servo control of a variac is a standard way of regulating mains but I have no idea how long the brushes last. Manual mode disables the motor.

The readings can be read and the settings changed from the command line or from a Python script using cURL. Here is an example that sweeps the voltage from 10 to 250 in 10 Volt steps and reads the current and power.

Here is a graph showing current, power and resistance versus voltage of a 60W light bulb

I plan to use this to plot a power supply line regulation graph. For that I will need a way of reading the output voltage in my script. I have two scopes that have network APIs as well as my Mooshimeter that has a BLE interface. I just need to work out which is the easiest to access in Python.

Here is the full ESP8266 sketch.