Debugging I2C

The I2CDriver is a new product from James Bowman of Excamera Labs. It is pictured at right. I’m going to get around to talking about it shortly.

Our controller board has two components on it: a processor and a clock. The processor talks to the clock using a two-wire protocol called I2C. I2C allows a controller to talk to peripherals using a low-speed serial protocol. More importantly, it allows for multiple peripherals to be on the same bus. Here, “bus” means “everyone is talking on the same wires,” but each peripheral has a unique address, so that they only listen for (and respond to) their own messages.

On our controller board, we can do things like set the clock by sending I2C messages to the clock chip. Or, we can plug additional layers into our sensor stack, and use I2C to talk to other peripherals on other layers. Most recently, we have been working on a board that has only one component on it: a barometric pressure and temperature sensor.

The sensor is small; while it looks huge (at right), it is actually only around 6mm in diameter. It has 8 connections underneath it, and two of them are for a processor to talk to it using I2C.

We have tried, repeatedly, to build boards that use this sensor. We have assembled three, and each one failed to work. We put some code on our controller, ran it, and nothing worked. We have one board with the same sensor from Sparkfun, and when we run our code, it works perfectly. So, we knew something was wrong with our board.

One of our first steps was to reflow the board. We got out our trusty air rework station, held the part in place, and carefully heated the pads. The solder reflowed under the pads, further securing the sensor to the board we designed. We tested continuity—making sure that the sensor is actually connected, electrically, to the controller—and everything tested out.

Once we did this, we plugged in our I2CDriver. This allowed us to send individual I2C messages to the sensor. Based on our reading of the datasheet, we should probably send a reset when the sensor first powers up. So, we did.

i2ccl /dev/cu.usbserial w 0x76 0x1E

This command uses the i2ccl program (which James Bowman provides with his I2CDriver) to write the message 0x1E (which is hex, and represents the pattern 0001110 in binary) to the I2C device that has address 0x76 (which our sensor does).

IT WORKED!

This was exciting. We had a sensor that worked! It hadn’t worked before! Clearly, it was our soldering!

So, we sent another command. This command told the sensor to measure the temperature using it’s build-in analog to digital converter (ADC).

i2ccl /dev/cu.usbserial w 0x76 0x48

IT WORKED!

This was awesome. We’ve been trying to get these sensors to work for weeks. We were feeling awesome. So, we then told the sensor to set its read pointer to memory location 0x00, which is where the result of the temperature read is stored on the sensor. We had to do this in order to read the temperature value out.

i2ccl /dev/cu.usbserial w 0x76 0x00

This caused i2ccl to crash. (I need to figure out why, and submit a bug report.)

We tried this multiple times, and every time, it caused it to crash. The same sequence worked fine on the board from Sparkfun, but it did not work on our board.

After roughly 1.5 hours of investigation, Maddie and I had to move on. Later that day, however, I decided to stare at the PCB layout again. Our boards have two GND pins, and two 3.3V connections; this is to simplify (arguably) our layout on individual layers of the sensor.

We had connected GND to GND, but we had not connected 3.3V to 3.3V. Why? Because we knew they were connected on the controller board.

But, we were testing without the controller board.

So, as a result, our sensor was not getting power.

However, when we plugged in the I2CDriver, our sensor was getting just enough power from the I2C lines themselves that it was able to turn on and respond to some low-power commands (like reset). However, asking it to take a measurement put it into a mode where it required roughly 1.5mA. This is a very small amount of current… but, it was more than could be drawn “parasitically” from the I2C pins. And, therefore, it was probably enough to kill the processor that is inside of the sensor. As a result, our sensor “crashed” whenever we asked it to take a reading, or worse, it “crashed part way,” and which would put it in a state of sending garbage back to the I2CDriver after we attempted a read.

To be honest, I don’t know whether our sensor crashed “all the way” when we asked it to do a temperature reading without enough current avaialable, or if it ended up in some half-way, indeterminate state… my guess is, based on how it failed, that we managed to only partially crash the sensor. I suppose we might say that the sensor was only mostly dead.

 

Either way, figuring out what went wrong took a fair bit of debugging to find the error in our board. In the sensor stack, it wouldn’t have been a problem. On the breadboard, while testing, it was a huge problem.

The fix? Connect 3.3V to 3.3V on the layers of the sensor, and all will be fine in the stack and in testing.

---

The takeaways of this story:

1. Debugging hardware is really hard. You have limited tools to debug what is going on, and when it comes to digital communications, you need a logic analyzer of some sort. We have both a Saleae Logic 8 Pro in the lab, and now the I2CDriver. We have used both in our work on these sensors, and they’re both invaluable.

2. Debugging hardware takes patience. You have to read documentation carefully, test and probe systematically, and question every single thing about your design and build. There is nothing that can be taken as a given.

3. Debugging hardware is a game of constant learning. I have been learning for the past 10 years the ways that hardware can fail, the mistakes you can make in a board design, and how to use the tools I have to debug problems when they arise. I will have to keep learning, because I expect I will keep making new mistakes.

I was, however, very glad to have the I2CDriver. At just under $25, it is a no-brainer tool to have on the bench. I probably could have debugged this with the Saleae, but being able to use the I2CDriver to send commands one-at-a-time was what let us figure out exactly where the failure was. I’m tempted to get a second for the lab, just so I can leave one connected up to a test/dev machine at all times.

Our next step is to get our battery charging circuit working. I have no idea what will be wrong with it, or how we will debug it. But, no doubt, something will be wrong…