Public Lab Research note

Video/Audio interface, Web page & Microcontroller

by donblair | October 14, 2012 21:27 14 Oct 21:27 | #4407 | #4407

The dream: every smart phone, laptop, or desktop can analyze data from an external, microcontroller (Arduino)-based temperature, pressure, etc sensor -- inexpensive, powerful, ubiquitous scientific instruments! The Problem: transmitting data between an external sensor (e.g., an Arduino + temperature sensor) and a computer or smart phone usually requires installing special software on the computer / smartphone that can communicate with the sensor, e.g. via a USB cable. This is often inconvenient (various versions of the software are required, depending on the operating system, etc); and, in many important cases, impossible (many high school computing facilities do not allow software to be installed without permission). The Solution: -- audio and video!

Nearly every computer or laptop or smartphone now has an internet browser, and the most recent "modern" browsers are now capable of accessing webcams or audio ports on the the device. This is how the PLOTS spectral workbench works, for example. The aim is then to develop a browser-based recording, plotting, analysis, and uploading tool that can interpret data received via the audio jack or the webcam, and then record, plot, analyze, and save such data.

I first heard of this idea at LEAFFEST, when Jeff Warren mentioned the idea of using blinking LEDs to calibrate the web-based PLOTS thermal flashlight tool. Later, at Public Lab's Brooklyn laboratory, Leif mentioned efforts to use the audio communication idea. And Todd, a friend who teaches high school chemistry, has long bemoaned his inability to install software on student-accessible computers in his district. This note is intended to get this idea out there in the PLOTS community & to get folks thinking about how this sort of interface might be put together ... the resultant device / software would be very useful!

Towards a solution ...

  1. Audio The least-expensive and most robust solution seems to be to transmit signals via the audio port. E.g., and Arduino could create an audio signal on one of its output pins, and this signal could enter a computer or smart phone via an audio jack. There are extant projects and products with designs for accomplishing this. Some relevant links are listed below.

  2. Video This might be done by having an Arduino display a QR code pattern on an LCD, and having a webcam on a computer / smart phone interpret this signal. One advantage would be that no cable is required. But this is likely more expensive, and data transmission rates would likely be much slower than via audio.

Annotated Links / Resources


-- An Arduino shield called a "SoftModem" that provides an Arduino library + audio interface to iPhones and Android phones.

-- A nice writeup: "Sensor data to iphone through the headphone jack using arduino".

-- Code for generating FSK in Javascript -- compatible with the SoftModem Arduino library above.

-- Sparkfun: audio jack modem for android.

-- The Hijack project -- data and power from phone’s audio port. Hardware for sale at SeeedStudio.

-- Relevant post on an arduino forum.

-- Assorted other links:


QR code method: Using an LCD driven by an Arduino-based sensor to display a QR code, which could then be read by any webcam on a laptop or smart phone.

-- A javascript / HTML5 implementation of a QR code reader, with code.

First steps: -- Just generated a QR code here, for the text:

{"tempC": 30.3,"pressATM": 1.01}

which generated a 29 x 29 QR code.

Then I tried a "data packet" (I don't even know if these are well-formed JSON):

{"packet": {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, {"tempC": 30.3, "pressATM": 1.1}, }

which generated a QR code that was under 80x80.

-- There's a 160x128 LCD on Adafruit for $10 (which might mean it's even cheaper to find them elsewhere if one knows where to look); -- and they've already assembled it into a breakout board for $25; -- which has a tutorial for using it with an Arduino; -- Next step: one could quickly test the resolution limits and etc by displaying example QR codes on a laptop screen, then reading them from the laptop with one's phone camera into an HTML5 reader ... -- Guidelines for estimating the minimal QR code size for a given amount of data, as well as the maximum distance a typical smart phone can be at in order to read a given QR code size

Other (easier / better ?) methods:

-- Should also look into simply blinking an LED as a way of encoding data!

-- Or: transmitting a binary string by turning a series of LEDs on / off simulataneously ...


these are great ideas-- video especially. I've dreamt of creating classic analog tools (like a spring-based tension meter, etc) that could be videoed and turned into real-time sensors. It would be a great way to visualize the math behind measurement. Of course, video analysis is a wicked, processor-intensive problem.

Open CV is trying to address issues like this.

Reply to this comment...

ohh! see simpleCV too

Reply to this comment...

Hey Mathew -- Nice links! You're totally right about video analysis being processor-intensive; Jeff's suggestion to consider audio first made a lot of sense to me.

So last night, Ben Gamari and I hooked up an Arduino running the SoftModem library to the mic jack of my laptop. SoftModem transmits data by representing in binary, then sending sound of two different frequencies to encode "0" and "1". I guess this sort of encoding is called FSK -- -- I'm just learning about all this cool stuff now! We wrote some quick code to transmit the number "21" once per second, and then used Audacity to record the sound from the laptop mic. I took a sample of three of these transmissions, and slowed down the third one a lot. The resultant sound is here:

SoftModem translates "21" into its binary form to transmit it; "21" in binary is "010101". SoftModem also adds a initial sound to each transmission so that the receiver knows to expect something. So what you hear is first, that long initial sound; then, the two chirps corresponding to "21" being broadcast at regular speed; then the third chirp has been slowed down a lot, so you can hear the initial handshake sound, then the "010101" pattern.

Now a next step might be to find a way to decode this using HTML5 / a browser, so that any laptop / smartphone with an audio jack + modern browser can get data from an Arduino. FUN!

Reply to this comment...

The audio jack seems like a decent stop-gap. Long term, however, I would highly suggest trying to work with the likes of Mozilla to get some kind of browser USB access (and maybe Bluetooth as well) codified. Mozilla is doing a good bit of related work already for Firefox OS, and most new phones should be able to play host to USB devices thanks to USB OTG. There will come a time when high-speed data transfer to scientific web applications will be invaluable, so I say let's get started on it now.

Reply to this comment...

Great progress, Don!

pitch detection!

Reply to this comment...

Jeff -- WHERE did you find that link?! I think you just solved the problem!

Cov -- I totally agree, and actually it took me a while to convince Ben that spending any time at all on this audio comm protocol was a good idea :) Firefox OS is really exciting; once web apps can access USB, then we're off to the races. Fully featured scientific web applications are going to require high-speed data transfer, and that's really where we should be heading.

The reasons (such as they are) that I'd give for justifying this sort of tinkering in the meantime are: 1) If we can get audio comm working, we can start prototyping and using the relevant web app software now. Having working prototypes will help speed and motivate the web app development. 2) There are currently lots of high school students in classrooms with desktop computers for which even the teacher doesn't have permission to install software. Audio comm + web apps could circumvent this problem; 3) This sort of hack demonstrates that communication doesn't have to occur via industrially-established protocols / equipment. I.e., end users can more readily recognize that they don't have to be at the mercy of the telecom industry's priorities / agendas for how we use the hardware we ostensibly own; 4) I felt like Jodie Foster in "Contact" when we saw the "010101" in our Audacity-captured audio signal. So #4 is "You can feel like Jodie Foster in 'Contact'".

All that as an aside, I totally agree with you -- we also should work with Mozilla to push for browser USB access. It's going to be so cool when that happens.

I'm still trying to wrap my mind around how awesome it is that these web apps allow you to "right-click view source". Truly open source, copy-paste-modify. Crazy cool.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

I'd been looking into decoding audio for an unrelated, semi-silly project some time ago:

and by the way, if you can't figure out tone decoding, there's no shame in using full-on, full-off noise and measuring volume -- you can make short/long tones and basically do Morse code. Might not be a bad idea for a first implementation/proof-of-concept!

Reply to this comment...

Jeff --

-- Hellschreiber.js: Holy (sic) guacamole! A few years ago, a physicist presented a Darpa-funded project at UMass. Darpa's request was: build us a system that would allow for long-distance communication after an EMP attack -- i.e., if all one's electronics were kaput. His solution involved making lots of little droplets of chemicals that underwent oscillatory chemical reactions that could be controlled, so that the material would show as a dark dot, or a light dot -- basically, a system could communicate in morse code visually, as long as one had a telescope and an array of these things (I guess). I need to wrap my head around the hellschreiber.js project. I tend to conflate things I shouldn't conflate, but, prima facie (from site explaining the technology), it seems related to me.

-- Morse code encoding / decoding is a great idea. A lot easier for me to get a handle on, conceptually, than implementing an audio version of an RS-232 protocol or whatnot. Also: engaging for high school students to see two computers communicating via an easy-to-understand alphabet. And eminently hackable.

-- Heck, if you get a browser to understand morse code via audio, you could even communicate with the browser by tapping out the signal yourself! How cool would that be as a demo of physical computing / communication, for students. And, jeez, I just want to be able to do that.

-- Just saw the irkit. Brain. Exploding. Stop with the cool ideas, for a sec? Whimper.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

I find it hilarious that DARPA would pay someone to reconceptualize optical semaphore, which is more than 200 years old.

awesome work on the audio jack! keep it up.

Reply to this comment...


Thanks for the optical semaphore link! This really sent me down a wonderful rabbit hole of fun ideas. I had no idea that Hooke gave a seminal lecture on semaphore tech whilst he was in the midst of characterizing how springs respond to stretching. May those glorious times of working on projects in several different 'fields' return! (I guess Hooke was working before there were these silly 'field' definitions ...)

One link I landed upon was this fellow at CalArts who seems to be wonderfully obsessed with data transmission schemes. Really helpful historical review of various methods: -- history of character codes: -- "bits, bauds, and modulation rates":

I was talking to a friend a day or so ago, and he seconded Jeff's suggestion of simply looking at the signal amplitude as a way of conveying bits, rather than processing frequencies. This certainly seems like a simpler scheme. I'm not yet hip enough to all of this to see what advantages frequency might have -- I'm guessing it allows one to avoid having to establish a baseline amplitude when interpreting a signal via amplitude? But anyway, might as well start with amplitude, as it's so much easier (I think).

Then I thought this morning (I think this just goes back to one of Jeff's original ideas): if one is simply measuring amplitude of a visual signal, is that relatively lightweight in terms of processing? Certainly less processor-intensive than, e.g., reading a QR code. And the code for interpreting e.g. a blinking LED wouldn't look so different from interpreting a high/low audio signal ...

The semaphore connection is, again, really, really cool. Leads one to imagine all sorts of alternative communication schemes, involving everything from purely mechanical lens + shutter + ink recording, to laser modulation at a distance ... might be a nice way of getting around FCC rules on radio broadcasts, when communicating with an airborne platform (camera on a kite, Raspberry Pi on a balloon) ... could a light sensor on the ground interpret flashes from a strobe light on a balloon? How much power do strobe lights require? Too much fun ...

Is this a question? Click here to post it to the Questions page.

Reply to this comment...

A friend just showed me this TED Talk -- "wireless data from every light bulb" -- the fellow makes some nice arguments for using visible light to stream data wirelessly, and includes a nice demo:

Reply to this comment...

One more link for this, semi-related:

Reply to this comment...

Oh, wow! "Audio-read-along" worked in my browser (latest stable Chrome, on Ubuntu) -- and apparently should work on latest stable versions of Opera, Chrome, Firefox ... okay, just gotta dig in and figure out how it works, now. Exciting.

Reply to this comment...

Login to comment.