Flip-Side of Embedded Linux

Since the inception of Linux way way back (in Internet time, that is), it has generated a lot of fan-hood and excitement. Finally after ages of Windoze-ism and when-do-you-crash-today-ism, Linux offers a viable alternative to the highly geeky mass.

Embedded devices are always a group by themselves because of three main constraints: power, memory, and 'closeness to hardware'.

The last item demands some explanation: Most embedded device will feature some non-standard peripherals. It could be a special memory-mapped region for retrieving of network packets, an external I2C bus on which the CPU talks to an FPGA, or even just a FIFO where the north CPU talks to the south CPU (yes, 2 separate CPUs, not dual core, no, not north bridge and south bridge). Therefore in embedded system there are a lot of work that involve interfacing and talking to the bare metal. You just can't avoid those like what you deal on regular PCs. Ok, I lied, this kind of work exists for all known computing devices, just that they are taken care of by poor firmware engineers like Cuppa.

"But my company's embedded device runs [your favorite OS] and engineers still don't need to care about the underlying hardware!", you may retort. Well, there are two possibilities to this. First, your company most probably has outsourced the firmware development (buying so called the board support packages which take care the low-level stuff). Second, the design your company has is based on a reference platform where again most of the dirty work has been done for you. It could be a linear combination of 1, 2, or both. However I will argue that no matter case 1 or case 2, any serious embedded software work will open the can of worms and someone eventually needs to take care the low-level details. The question is just who and when.

Linux becomes favorable because all source codes are open and free (both as in beer and speech). With source code in hand, the engineers can prune and trim according to need. This seems like a dream come true: We got something cool, and it is free!

Like everything in life, there are always down-sides for Linux. In fact, there are quite significant weakness in Linux-based embedded solutions.

First of all, beside the Linux core kernel, you need a fair number of software modules to get the system up and running. Note that these software modules are usually maintained by different people and the quality variation is significant. No worry, there are companies that sell this kind of solution, say solution from manufacturer X, they say they have an 'integrated, pre-test' solution on one 'open source development platform'. So far so good, you bought solution X version N (XvN), implemented your product on top of it, and it is good to go. Perfect, but this is actually the easiest part of the whole development cycle.

The horror starts when you get past the software release and enter maintenance phase. Note that XvN you bought earlier represents a snap-shot of the open source landscape some time ago, and worse still, due to stability and testing purpose, company X usually will not source the latest source code from the said software modules. In other words, don't expect your XvN to contain the latest features, bug fixes, and security fixes.

On top of the problem XvN is obsolete right after it is delivered, there is a problem where feature changes in new release break some other software in the system. Recall that the underlying modules are maintained by different people/organizations, even though the maintainers of key modules will exercise certain cautions when doing modifications, there is no official coordinator to draft out a system plan to ensure backward compatibility and coherency for _your_ system. Therefore there is no guarantee the newer release of a key module will continue to work for your system or even XvN. It is far not uncommon to have a system completely fall flat on its face when doing a minor upgrade because a subtle change in package functionality. Did I mention some package maintainers just disappear or give up the software and you are left with a year 2000 version of package XYZ?

This post is inspired by this post. Read it, if you are very into Linux.