I received a bunch of emails today pointing to this blog from Jason Perlow. Jason has an interesting thought with regards to the Apple and HTC lawsuit that is brewing. Let me first say that I understand that companies have to protect their IP and that there are clearly important and enforceable patents out there, say Coca Cola's formula for well, Coca Cola. Apple certainly has a lot of valuable IP as well and they are spending a lot of dollars in making the user experience better. Some of these patents are vague at best, so I have mixed feelings on this and since I am not a lawyer I am going to leave it at that.

Jason's blog evolves around hypervisors and his fantasy world in which they can change the way we build mobile phones. This is not a fantasy world, in the fact that what Jason wants is technically very possible. However, it would also require alliances between some of the fiercest competitors in one of the biggest markets in the world. Unlikely to happen, however, allow me to dream with him.

Virtualization can be used separate the nuts (the piece that makes a phone talk over the network) from the glory (the piece that makes it looks pretty and provides applications), no nuts no glory, they are both needed, but they need to be kept separate. The nuts would be real-time, need to be kept secure and are proprietary to the hand-set, it contains drivers and logic to make the phone talk 3G, or 4G, whatever. The nuts is very hardware specific, it would use an RTOS.

The glory would be the user-interface, it could be any OS, it could use Java, Flash, be built by Apple, Google, Wind River, Microsoft, a hobbyist. What the glory needs is a well-defined API to talk to the nuts. Unlikely to be standardized as I mentioned above, but possible. The glory is the part that you could install as per Jason's story.

The important part is the separation, many mobile operating systems currently have nuts and glory combined into one (Android for example), but to make Jason's blog a reality, would require to split them.

Mobile is only one of the markets that can benefit from a story like the one Jason mentions. Generalizing the solution here, what we are really doing is enabling multi-OS on a single piece of hardware. That hardware can be multicore or single core.

The enabling technology for Multi-OS is virtualization through a hypervisor. A hypervisor allows one to run multiple different operating systems, strongly partitioned, fairly scheduled. It is a breakthrough technology for the embedded market and it enables new thinking.

To give another example in industrial control: Many industrial control devices currently consists of 2 separate boxes, talking together over ethernet to provide a single solution, one box handles real-time (RTOS) the other the UI (Windows, Linux). Ethernet is notoriously not real-time, but if your timing requirements are in the milliseconds, then Ethernet is ok.

Moving to a single multi-OS solution on a single chip (single- or multicore) would remove the need for ethernet and replace it with communication over shared memory. All of a sudden the latency for communication improves dramatically. It goes from milliseconds to microseconds and this drastically changes what this particular device could do, giving the manufacturer a strong leg up over the competition.

These types of examples exist for virtually every industry that uses embedded systems and I spend my days talking to customer explaining how they need to get on the virtualization band wagon or they will miss the next revolution in embedded systems.

The revolution is happening now, thanks Jason for your views on this and thanks for calling my video 'geeky', that was kind of the intent :)

It may not be the Wild West, but there are some similarities. I'm moderating (or is it refereeing) a discussion between two (opinionated?) personalities, Jim Turley and Clive "Max" Maxfield. The topic of the debate is embedded vs. discrete processors. The topic arises from a recent announcement by Actel, who built an FPGA around an ARM Cortex M3 core. Hence, is it better to embed or not to embed?

The panel will be held at the Embedded Systems Conference (ESC) in San Jose, April 27, at 4:30. Jim likes the discrete approach, while Max is more of an embedded guy. Shouldn't be any surprise there. But if you have an opinion in the matter, or just want to see these two guys go at each other (which promises to be a lot of fun), stop by and throw a question or two into the mix.

Max also offers his take on the panel as well as some other events happening at ESC.

When I first heard about Altium's NanoBoard 3000 development kit, I thought it would be far too involved for me, both in the kit's complexity as well as the time investment I'd have to make to really get to know the kit. But I was pleasantly surprised. I followed the simple direction to put the hardware together, then installed the software, and was on my way pretty quickly.

The kit has lots of different options, but the one I got has a pair of Xilinx Spartan FPGAs at its core (the main device is a Spartan-3AN). The other FPGAs the board could come with are an Altera Cyclone III or a Lattice ECP2. The kit also includes stereo speakers, a small touchscreen color LCD, a bunch of LEDs, an IR remote, and just about every kind of I/O that you could need.

I just went through some basic steps and I was able to program it to the point that I could get an image on the display, get the LEDs to light up, and get some sounds out of the speakers. I could easily see how you could take this kit pretty far down the road in terms of a prototype. Obviously it's way over-built, but you could get just about all your code written and deployed on this platform (assuming you're good with the hardware mix that Altium has assembled).

Altium has out a lot of time and effort into the kit to ensure a good (and simple) user experience. There's a lot of collateral on their web site, so you should be off and running pretty quickly. The complete kits costs around $400.

The first of many MCU announcements are upon us, from ARM with its M4 core. This one is a little different obviously as it's just a core and not a real microcontroller. But with the M0 and M3, and now the M4, ARM is going full bore after the MCU market. The margins for MCUs can be razor thin, but the potential number of units can be staggering.

Expect to see a bunch of MCU announcements next week as Embedded World opens in Nuremberg, then a bunch more as we get closer to the Embedded Systems Conference in late April. In general, and to the credit of the vendors, most of the new MCUs you'll see is specialized for a certain aspect, such as wireless capability, super low-power, higher performance, etc.

Stay tuned, as I'll be writing about the first batch next week from Nuremberg.

The Accellera standards organization has been in existence for 10 years now. I remember like it was yesterday when the boards of directors of Open Verilog International (OVI) and VHDL International (VI)decided that merging the two organizations made sense for the industry. The “standards war” between Verilog and VHDL had ceased to be interesting, and the two hardware description languages were coexisting fairly peacefully. Forming a single organization to oversee the development and adoption of standards for language-based design of integrated circuits made a lot of sense.

The merged organization was named Accellera, and it brought efficiency and focus to standards development in the design automation industry. (Some people thought the name sounded like a car or a vegetable, but I thought it had a nice ring. Gabe Moretti tells the naming story on his blog.)

Several of us had been representatives on both of the OVI and VI boards, so it was a relief to attend half the number of meetings while accomplishing the same amount of – or even more – work towards providing standards for the industry. I recall one day going to a VI board meeting in the morning where people said, “Oh, those Verilog guys…” and then going to an OVI board meeting in the afternoon where people said, “Oh, those VHDL guys…” I kept thinking since “those guys” are “us guys” it sure would be nice to have just one board.

Over the past decade, Accellera has had many successes. Obviously, they have provided numerous market-relevant, well-adopted standards for improving the chip design process. Two of my favorites are SystemVerilog and the Unified Power Format (UPF), both of which are now IEEE standard 1800 and 1801, respectively.

SystemVerilog’s widespread adoption is witness to its success. Starting out with technical donations from Accellera member companies BlueSpec, Mentor Graphics, Motorola, Novas, Real Intent, and Synopsys – with additional contributions from user companies – SystemVerilog expanded upon the long-time industry standard language, Verilog, to address modern design challenges.

Speaking of SystemVerilog and the IEEE, another noteworthy accomplishment of Accellera is the key role it played in supporting the IEEE Standards Association as it established its corporate standards program. SystemVerilog was the second standard to go through the entity-based (one company, one vote) standardization process of the IEEE. It helped pave the way for about 20 corporate standards projects that have been completed or are currently underway in the IEEE.

One of these corporate standards is 1801, which began in the IEEE when Accellera transferred its UPF standard to them. UPF was developed in record time – less than 6 months from the official start of the working group – and arguably, at a reasonable cost to the industry. I was part of the Accellera working group that produced UPF. When it was finished, I made some rough calculations to estimate how much the companies had paid for their employees to develop UPF. For the 9 month period from concept to completion, I counted the number of people in the group, estimated how much of their work week they spent on the standard’s creation, and included an approximation of how much time employees of two standards organizations (Accellera and Si2) spent discussing the landscape for UPF. Not scientific, but interesting anyway, my guesstimate was that 7.3 manyears were invested by the UPF working group. Using a fully-burdened hourly rate of $200, the total cost would have been about $3M. Contrast this to the estimate of 100 manyears (see the last paragraph of the article) that were spent developing the Common Power Format (CPF). That would have been a cost of over $41M! These figures may have been greatly under- or overestimated, but to me it shows the value of cooperation among competitors when producing an industry standard.

Accellera is going to celebrate its 10th birthday with a lunchtime event at DVCon on Tuesday, February 23. I hope you join me in wishing Accellera many more decades of success.

Failure to tag certain types of variables with C’s volatile keyword, can cause a number of symptoms in a system that works properly only when the compiler’s optimizer is set to a low level or disabled. The volatile qualifier is used during variable declarations, where its purpose is to prevent optimization of the reads and writes of that variable.

For example, if you write code that says:

g_alarm = ALARM_ON; // Patient dying--get nurse!
// Other code; with no reads of g_alarm state.
g_alarm = ALARM_OFF; // Patient stable.

the optimizer will generally try to make your program both faster and smaller by eliminating the first line above--to the detriment of the patient. However, if g_alarm is declared with volatile this optimization will not take place.

Best Practice: The volatile keyword should be used to declare any: (a) global variable shared by an ISR and any other code; (b) global variable accessed by two or more RTOS tasks (even when race conditions in those accesses have been prevented); (c) pointer to a memory-mapped peripheral register (or register set); or (d) delay loop counter.

Note that in addition to ensuring all reads and writes take place for a given variable, the use of volatile also constrains the compiler by adding additional “sequence points”. All lines of code above the read or write of a volatile variable must be executed prior to that access; likewise, all lines of code below the access must be executed afterward.

Firmware-Specific Bug #2

Technically, the problem of a non-reentrant functions is a special case of the problem of a race condition. For that reason the run-time errors caused by a non-reentrant function are similar and also don’t occur in a reproducible way—making them just as hard to debug. Unfortunately, a non-reentrant function is also more difficult to spot in a code review than other types of race conditions.

In a typical scenario, the software entities subject to preemption are RTOS tasks. But rather than manipulating a shared object directly, they do so by way of function call indirection. For example, suppose that Task A calls a sockets-layer protocol function, which calls a TCP-layer protocol function, which calls an IP-layer protocol function, which calls an Ethernet driver. In order for the system to behave reliably, all of these functions must be reentrant.

But the functions of the driver module manipulate the same global object in the form of the registers of the Ethernet Controller chip. If preemption is permitted during these register manipulations, Task B may preempt Task A after the Packet A data has been queued but before the transmit is begun. Then Task B calls the sockets-layer function, which calls the TCP-layer function, which calls the IP-layer function, which calls the Ethernet driver, which queues and transmits Packet B. When control of the CPU returns to Task A, it finally requests its transmission. Depending on the design of the Ethernet controller chip, this may either retransmit Packet B or generate an error. Either way, Packet A's data is lost and does not go out onto the network.

In order for the functions of this Ethernet driver to be callable from multiple RTOS tasks (near-) simultaneously, those functions must be made reentrant. If each function uses only stack variables, there is nothing to do; each RTOS task has its own private stack. But drivers and some other functions will be non-reentrant unless carefully designed.

The key to making functions reentrant is to suspend preemption around all accesses of peripheral registers, global variables (including static local variables), persistent heap objects, and shared memory areas. This can be done either by disabling one or more interrupts or by acquiring and releasing a mutex; the specifics of the type of shared data usually dictate the best solution.

Best Practice: Create and hide a mutex within each library or driver module that is not intrinsically reentrant. Make acquisition of this mutex a pre-condition for the manipulation of any persistent data or shared registers used within the module as a whole. For example, the same mutex may be used to prevent race conditions involving both the Ethernet controller registers and a global (or static local) packet counter. All functions in the module that access this data, must follow the protocol to acquire the mutex before manipulating these objects.

Beware that non-reentrant functions may come into your code base as part of third party middleware, legacy code, or device drivers. Disturbingly, non-reentrant functions may even be part of the standard C or C++ library provided with your compiler. For example, if you are using the GNU compiler to build RTOS-based applications, take note that you should be using the reentrant “newlib” standard C library rather than the default.

A race condition is any situation in which the combined outcome of two or more threads of execution (which can be either tasks or main() plus an an interrupt service routine) varies depending on the precise order in which the instructions of each are interleaved.

For example, suppose you have two threads of execution in which one regularly increments a global variable (g_counter += 1;) and the other occasionally resets it (g_counter = 0;). There is a race condition here if the increment cannot always be executed atomically (i.e., in a single instruction cycle). A collision between the two updates of the counter variable may never or only very rarely occur. But when it does, the counter will not actually be reset in memory; its value is henceforth corrupt. The effect of this may have serious consequences for the system, though perhaps not until a long time after the actual collision.

Best Practice: Race conditions can be prevented by surrounding the “critical sections” of code that must be executed atomically with an appropriate preemption-limiting pair of behaviors. To prevent a race condition involving an ISR, at least one interrupt signal must be disabled for the duration of the other code’s critical section. In the case of a race between RTOS tasks, the best practice is the creation of a mutex specific to that shared object, which each task must acquire before entering the critical section. Note that it is not a good idea to rely on the capabilities of a specific CPU to ensure atomicity, as that only prevents the race condition until a change of compiler or CPU.

Shared data and the random timing of preemption are culprits that cause the race condition. But the error might not always occur, making tracking down such bugs from symptoms to root causes incredibly difficult. It is, therefore, important to be ever-vigilant about protecting all shared objects.

Best Practice: Name all potentially shared objects—including global variables, heap objects, or peripheral registers and pointers to the same—in a way that the risk is immediately obvious to every future reader of the code. Netrino’s Embedded C Coding Standard advocates the use of a 'g_' prefix for this purpose.

Locating all potentially shared objects is the first step in a code audit for race conditions.

Last year a friend had a St. Jude pacemaker attached to his heart. When he reported an unexpected low battery reading (displayed on an associated digital watch) to his doctor a month later, he learned this was the result of a firmware bug known to the manufacturer. The battery was fine and would last on the order of a decade more. His new-model pacemaker's firmware didn't include a bug fix that was provided the year before to wearers of old-model.

Another friend owns a Land Rover LR2 SUV with back-up sensors. When the car is in reverse and nearing an obstacle or another car, the driver is alerted via a beeping sound. Except that the back-up sensors don't always work. Some "reboots" of the SUV don't seem to have this feature enabled. He suspects there is a "race condition" during the software startup sequence.

Yet another friend has driven a Toyota Prius hybrid over 100,000 miles. He reports that the brakes very occasionally have an odd/different feel. But his older model Prius is not expected to be subject to the 2010 model year recall.

These are just a few of the personal anecdotes behind the headlines. Embedded software is everywhere now, with over 4 billion new devices manufactured each year. Increasingly the quality and safety of products is a side-effect of the quality and safety of the software embedded inside.

One important question is, can we trust future software updates any more than we can trust the existing firmware? How do we know that the Toyota Prius hybrids with upgraded braking firmware will be safer than those with the factory firmware?

It's nice to see that more people are taking security issues seriously. A release crossed my desk last week from Dell, saying that they've hooked up with Integrity Global Security to provide a secure operating system on desktop PCs for government applications. Integrity Global Security is a spin-off of Green Hills Software, the group that was tasked with selling the company's secure OS to government and military outlets.

The agreement between Dell and Integrity Global Security makes Dell the exclusive provider of the Integrity separation kernel for general-purpose secure computing to government agencies.

It's nice to see a company like Dell, who has its roots in PCs and servers, make a move like this one. From all the documentation I've seen, there's no disputing that Integrity really is a secure OS. It's been certified by NIAP to EAL6+1 and High Robustness. Coupled with the servers offered by Dell, it's a nice combination.

This will simplify the process for government workers, who will be able to operate a PC more like PC, knowing that their data will remain secure. Hence, there's little to no training required for the user.

The hardware-software combination that'll be offered by Dell includes the Integrity-178B separation kernel and Dell's OptiPlex line of desktop PCs, which are already used by government customers.