Tech Books I Will Read Again

I have read a crap-ton of technical books, mostly on software, but some either more general or hardware related so I felt the need to generalize the genre to "technical" books. If you've been following my blog for the last year, you'll agree that especially recently, my reading rate has been arguably excessive. I'm reaching a point where I'd like to slow down and focus on some other things in my free time, but I'm also reflecting on all of the great and not-so-great tech books I've read. One of the defining factors in whether I think a tech book is excellent versus merely good is if I have the urge to read it again. (For anyone wondering, there is no distinction needed for the bad tech books.) This feeling might happen right after I finish it, or even while I'm reading it the first time. It also might take a while to percolate and rise back to the surface as a book I want to go back to. The bottom line is, a mark of a great tech book is that it's worth revisiting, so what follows is a list of tech books I've read that I thought were so great that I'm going to read them again.

Tech Book Face Off: Facts and Fallacies of Software Engineering Vs. Programming Pearls 2

Since I've been hitting the tech books pretty hard for a while now, for this Tech Book Face Off I wanted to take a bit of a breather and do a couple of relatively easy reads. These books have been on my to-read list for some time, so I decided to finally check them out. The first one, Facts and Fallacies of Software Engineering by Robert L. Glass is a book in a similar vein as The Pragmatic Programmer in that it relates various tidbits of advice on the craft of software engineering. As for Programming Pearls 2 by Jon Bentley, this book surprised me. I thought it would be somewhat similar to Facts and Fallacies, just more directly related to instructive programming examples than to the software engineering field at large, but it turned out to be quite a bit different, as we'll see in this review.

VS.

Tech Book Face Off: The New Turing Omnibus Vs. Patterns of Software

I'm churning through tech books now, finishing off a bunch that I had started a while back, but couldn't find the time to finish until now. The pair that I'll look at here are a couple of older books that I picked up through recommendations on blog posts. The first one, The New Turing Omnibus: 66 Excursions in Computer Science by A.K. Dewdney, is a survey of 66 topics in a wide range of areas of Computer Science. The second book, Patterns of Software by Richard P. Gabriel, is about advice and experiences on a variety of topics in software development. Whereas NTO is of a strictly technical nature, Patterns of Software has much more of the human aspect of working with computers and software. Let's see how these older books hold up today.

VS.

In Search of the Best Code Editor

I have used quite a few code editors over the years, and since I spent the last four posts explaining how to use my current favorite editor, I thought I would take a step back and go through a broader overview of some of the editors I've used recently. Some are specific to certain languages or platforms, and some can easily be used with almost any language on multiple platforms. Each editor has its strengths and weaknesses, its cool features and warts.

I'll briefly cover what makes each editor special and why I do or don't like it. Choosing an editor ends up being a very personal choice, and different programmers will gravitate to wildly different editors. That makes this review necessarily opinionated, so if you don't agree with me, that's expected. The important thing is to try out different editors for yourself to figure out what works best for you. When you find your best editor, you'll know.

Tech Book Face Off: Practical Node.js Vs. Node.js the Right Way

I've recently gotten into some server-side JavaScript development at work, and I needed to take a crash course in Node.js to make better progress, so I did what I always do. I picked up a couple of books on the subject and dug in. I did some perusing of reviews on Amazon.com and settled on a couple of books that appeared to be solid: Practical Node.js by Azat Mardan and Node.js the Right Way by Jim R. Wilson.

The funny thing about Node.js (I'll refer to it as Node from now on) is that there's not that much to talk about, so you quickly move on to talking about all of the libraries that you can plug into Node to get stuff done. That's because Node isn't a framework like Ruby on Rails or ASP.NET MVC. Node is a runtime that runs on the Chrome V8 engine, and all it does is allow you to run JavaScript directly on any machine outside of a browser. It includes a small, tight collection of APIs for working with file systems, operating systems, and networking interfaces, and it establishes an asynchronous paradigm for writing and executing code.

This asynchronous paradigm takes a little getting used to. Basically, any function call that could block, be it a file system access, database access, or HTTP request, will take a callback function as its last parameter and return immediately. When the slower operation finally finishes, the callback will be executed with the response from the operation as its parameters. If you forget about this behavior, you'll very quickly run into situations where you're trying to read things out of a database that haven't been written into it yet, or getting empty responses from HTTP requests. Basically, all dependent code needs to be executed inside callbacks, and that leads to the well-known Node existence of Callback Hell. We'll talk more about how to deal with that in a minute, but that pretty much is Node in a nutshell. Let's take a look at the books.

VS.

Finding Optimal Friction

In the last twenty years, the Internet and mobile devices have reduced or eliminated friction in numerous industries. Obviously the communication sector has been dramatically affected, including telecom, music, television, and publishing industries. Now anyone with an Internet connection can put their stuff up on-line for the world to see, and new players like Netflix have been able to challenge the big networks for our prime time hours.

The Internet has leveled the playing field across the communication industries, and it's now easier than ever for competing producers to get their products in front of customers. That's one way to look at the concept of friction in markets, from the perspective of producers. Another way to look at friction is from the consumer's perspective, and that friction has been dramatically reduced, as well. From having all of the world's information literally at your fingertips to being able to buy nearly anything at the click of a button and having it shipped to your door, the Internet has gone a long way in removing friction from consumers' lives.

However, not all industries or aspects of our lives have been affected equally by the Internet, and sectors like energy and transportation still have a lot of friction that could be reduced with the right advances in technology. Energy production and automobiles are ripe for a technological revolution.

Reducing friction isn't the be-all and end-all for making our lives easier, though. Reducing friction comes with its own cost, and I think we sometimes forget how high that cost can be. We can end up wasting more time and energy in a frictionless environment due to distraction and an overwhelming amount of choice. Finding the right balance means recognizing where too much friction is wasting our energy so that we can target those inefficiencies and realizing where too little friction is wasting our time so that we can avoid those time sinks. It's a constant struggle as we push forward with technology.

Exploring Different Types of Programming

Not all programming is the same. In fact, programming can be split into quite different categories depending on what type of program you're writing or what type of hardware you're writing for. Different types of programming, let's call them programming styles, have different constraints and require different environments and design trade-offs for programmers to work effectively in that style.

Programmers working in different styles will care deeply about entirely different aspects of programming. That's why you often see programmers vehemently debating the merits of opposing programming methods without ever reaching agreement. From each of their perspectives, their way is right. The defining difference in the argument is actually one of context.

To get a general overview of the different programming styles out there, and to get a better sense of what other programmers are concerned about with their code, let's look at the various programming styles. Some of these styles I know quite well because they are my day job, some of them I'm learning, some of them I've dabbled in, and some of them I know only from what I've heard. I'll try to be careful in what I say about those styles that I don't know too much about. There may be other styles that I haven't thought of, but these should be the main ones.

Embedded Programming

The main programming style I've been doing for the last few years is embedded programming. In this style, you are programming for a particular microprocessor or closely related family of microprocessors. The software you write is called 'firmware' and it will be programmed directly into flash on the processor or an external flash chip connected to the processor. Normally, this firmware is not changed very often, if at all, during the life of the product in which it resides. It's firm, not soft, hence the name.

Even within embedded programming, the differences in constraints from one programmer to another can be quite large, depending on whether the programmer is programming for a beefy quad-core GHz ARM processor or a meager 8-bit micro-controller. In most cases, the processor is chosen to have just enough horsepower to do the job it's meant to do, or more likely, slightly less power than the job requires, and the programmer has to make up the difference with crafty coding skills. Low power is a common consideration when choosing an embedded processor, and much of firmware design involves figuring out how often, how long, and how deeply you can put the processor to sleep.

Embedded processors have a number of communication interfaces and peripherals, and the embedded programmer must become well versed in bit-banging registers in the processor to configure these peripherals to interface with the outside world through attached sensors, storage devices, network interfaces, and user interfaces. Programs are mainly interrupt driven with real-time deadlines to meet. An RTOS (Real-Time Operating System) will provide mechanisms for defining interrupt service routines, tasks to run when interrupts aren't being handled, events and message queues for communicating between interrupts and tasks, and locks and semaphores for synchronization.

Programming environments are most often proprietary and provided by the microprocessor vendor. All of the ones I've used have been Eclipse based, and they provide a debugging interface with a hardware emulator that you connect to the processor to provide the normal debugging features of breaking into and stepping through code, reading and writing to memory, and giving access to the processor's registers. They also usually display peripheral register values in a decoded, human-readable way, show various characteristics of the RTOS state, and allow some level of profiling of running code. Non-proprietary IDEs are available as well, but they tend to be expensive.

In general, embedded programming happens very close to the metal, and you have to exert fine control over the resources available in the processor. Programming is usually done in C or C++, although Lua, JavaScript, and Python are starting to make inroads. It's as close to classic programming as you can get, with all of the considerations of limited memory spaces, special hardware instructions, and printing characters through a UART terminal included.

Systems Programming

Systems programming is the design and implementation of software that interfaces between hardware and other types of software, otherwise known as operating systems. Windows, Linux, iOS, Android, and the RTOS that an embedded programmer uses are all examples of operating systems. Systems programming is similar to embedded programming in many ways because the programmer needs intimate knowledge of the hardware, but whereas an embedded program normally targets a specific microprocessor, an operating system will run on a wider variety of hardware and include drivers for many, many more peripherals.

Operating systems provide the basic services that other programs use, like disk and file management, virtual memory, preemptive multitasking, device drivers, and application loading to name a few. The systems programmer has to worry about designing algorithms that will have high performance on a wide variety of hardware, writing drivers that work with an incredible variety of peripherals, and making sure that programs can play nice together without stepping on each others toes or bringing down the operating system altogether. Security and reliability are constant concerns of the systems programmer.

Most systems programming will involve at least some C and more likely a lot of C. The core of the operating system, referred to as the kernel, is normally written in C. C++ and Java are also commonly used outside of the kernel. The development environment is as varied as the programmer that's doing systems programming, but there are often a lot of specialized tools written specifically to support developers working on an operating system. Systems programming requires strong knowledge of algorithms and data structures, a meticulous eye for detail, and an ability to think about software running at the low level of the hardware-software interface.

Language and Compiler Design

Designing languages and writing compilers and interpreters is similar to systems programming in that programming languages are an interface between hardware and the software programs that run on that hardware. When the language runs on a virtual machine (VM) it even further blurs the line between language design and systems programming because the VM provides many of the same services as an OS. A programming language is not an OS, though. It's a lower level construct than an OS, and while an OS provides abstractions for various hardware features and peripherals of the processor, programming languages provide abstractions for the machine instructions and computational model of the processor.

Like embedded programmers and systems programmers, compiler writers are concerned with low-level performance, but they are concerned with performance in at least three languages—the host language that they're writing the compiler in, the source language that is being compiled, and the target language that the compiler outputs. After all, the compiler's basic function is to translate from a source programming language that other programmers write in to a target language that either the VM or the processor uses. Most compilers are written in C, although some are written in other languages. Rubinius is a Ruby interpreter written in Ruby, for example.

Compiler writers need to know the deep, dark corners of these languages to make the host and target code as fast as possible and to cover all of the things that any programmer could possibly do in the source language. Programmers want fast compile times, good interpreter performance, and the best runtime performance they can get, so compiler writers need to be well-versed in all of the low-level optimizations they can do to squeeze every last bit of performance out of their code, both the code the compiler runs on and the code it produces. On top of that, the code for the source language has to be interpreted correctly in all cases, even cases that most programmers will never think to exercise. Compiler writers need to think about and deal with complicated edge cases that result from interactions between seemingly simple language features that would make anyone's head hurt. I can't even begin to imagine how hard it must be to write a compiler for the more complicated languages (I'm looking at you, C++).

Application Development

The field of application development is as wide and varied as embedded programming. It ranges from the creation of simple utility apps like Notepad or a calculator to huge, complex apps like Adobe Photoshop or Firefox. The bigger an app gets, the more its development looks like systems programming, but at its core, an app provides a user interface and features to help the user do something useful with the computer. A computer with only an OS doesn't do anything meaningful. It's the applications that give the user the power to create and consume in all of the ways we've come to depend on.

For the most part, performance doesn't matter as much in application development. Normally an application has huge amounts of memory, storage, and processor performance to work with, and it's normally waiting for user input—a slow process compared with the high performance of the underlying hardware. More computationally or I/O intensive parts of an app may need to be optimized for performance, but the code should be profiled for hot spots and bottlenecks before blindly trying to improve slow operations. It's easy to get fooled into optimizing parts of the app that don't need it and ignoring the parts that are actually causing poor performance. However, in most cases the application programmer can get by with writing clean, clear code in a straightforward way and letting the compiler, OS, and hardware take care of making the app fast.

The application programmer has enough to worry about with making the app as easy to use and useful as possible. Choosing the right features, organizing the app so that it makes sense to most users, and creating an awesome user experience is insanely hard to get right. Users are fickle and demanding (not on purpose as they have their own lives and stuff they want to get done), and striking the right balance of features in any application is a constant struggle.

The environments for application development are generally quite good. Debugging features are much better than those for embedded programming, and IDEs like Visual Studio and Xcode make building a GUI and setting up the boilerplate code for an application much faster than it used to be. Application development can be done in almost any language, but native applications are most commonly written in C#, Java, Objective-C, and, to a lesser extent these days, C++.

Mobile Development

I'm not convinced that mobile development is much different than application development, except that it's done on a different platform than the target platform of the app. That gives mobile development some characteristics of embedded development because you need to connect to the system that the code is running on to debug it. Mobile development tools include a simulator to run the code on the host platform so that more debugging information is available while you're getting the app up and running. Otherwise, mobile development is very similar to application development in most respects.

Web Development

Web development is similar to application development in that the programmer is concerned with making a great user experience and producing a product that meets the needs of the majority of users for a given set of tasks. Also like application development, as a web application gets larger and more complex, development starts to look more and more like systems programming. The server infrastructure and services provided through APIs become the overarching concern as a web application grows and attracts more users.

Web development is different from application development in that network and database interfaces are a prerequisite for web development. A native application may have these things, especially the bigger applications, but a web application without them would be meaningless. The client and the server are linked, but undeniably separated, and the server needs to manage connections to hundreds, thousands, or even millions of clients whereas most native applications deal with a handful of users at most.

Web developers work in a wide variety of environments and languages. Text editors like Vim, Emacs, and Sublime are commonly used, but full-fledged IDEs like Eclipse, NetBeans, Visual Studio, or anything from JetBrains are also popular. To avoid reinventing a bunch of wheels in web programming, you'll use some kind of web framework, and there are numerous options in any language you choose. The most well known frameworks are (in no particular order) Ruby on Rails for Ruby, Django for Python, ASP.NET MVC for C#, Grails for Java, Laravel and Symphony2 for PHP, and AngularJS and Ember.js for JavaScript. Tools for web development are evolving quickly, and debugging requires a wide variety of skills from classic print statements and reading logs to modern site monitoring and testing with tools provided by various companies in the cloud.

Game Programming

I don't know much about game programming, since I've never made any serious video games, but I'll give this description my best shot. Game programming has elements of application development because of the game's user interface, systems programming because a game will normally have its own memory and storage management, and embedded programming because the game code needs to be as close to the hardware as possible and has real-time deadlines. The video game programmer is a slave to the frame clock, and everything that needs to be done for a particular frame—user input, computer AI, physics calculations, 3D model calculations, and graphics redrawing—needs to be done before the next frame gets drawn on the screen. That sequence needs to happen at least 60 times per second to make a smooth user experience.

Game programming may have more similarities with the next style, scientific computing, than anything else. The graphics programming is especially computationally intensive, and the graphics card is a massively parallel processor that's designed specifically for doing complex vector calculations. The most common game programming languages are chosen for their raw speed and closeness to the physical hardware that they're running on. That means C, C++, and assembly for most video games. Tools and environments are mostly custom and proprietary because they need to be so specialized for the type of development they're used for.

Scientific Computing

Surprisingly, scientific computing is most similar to game programming in that it is defined by massively parallel computation. While games model a fictional universe, the goal of scientific computing is to model a real-world system with enough accuracy to explore new truths about the system being modeled. Some examples of scientific computing include weather simulations, galaxy formation, fluid dynamics, and chemical reactions. Strong mathematics and algorithm knowledge is a definite requirement for scientific computing, and parallel programming is obviously common both when programming on a graphics card with CUDA and on a server cluster.

I'm hesitant to say that any of these programming styles is more complex or requires more skill than any of the others. Each style has its own set of issues, things that are challenging, and things that are interesting. They each overlap in interesting ways with other styles. Exceptional programmers populate every one of these programming categories, and what differentiates the programming styles is not how hard they are, but the types of problems that need to be overcome to create awesome software.

Creativity and Hard Work Are Both Necessary for Problem Solving

As programmers, as engineers, we solve problems all the time. It's a significantly large part of what we do. How do we solve problems efficiently and effectively? It requires a balanced mix of creativity and hard work. Both are hard to achieve, and a good combination of the two is even harder. Creativity requires getting yourself in a mood where you can let your thoughts run free and play with ideas, and hard work requires a mood where you can knuckle down and produce results. These moods are in constant conflict, and you can be at your most productive when you manage the two moods well.

I was going to make this post about how creativity is not what most people think. Instead of being driven by flashes of insight, I was going to argue that it was driven mainly by hard work. While I still think that's true in a sense, and I'll get into that more later, after looking at my notes and links to references on the subject, I believe creativity is a combination of insight and hard work in the form of intense thinking. Both things require time and space to accomplish, and both are required to creatively solve problems.

How to be Creative

I recently found this excellent talk by John Cleese on how to be creative. Much of his talk is based on the work of Dr. Donald W MacKinnon, and although I haven't read the book, I thoroughly enjoyed the talk by John Cleese. The gist of the talk is that there's no defined way to be creative, as that's somewhat of an oxymoron, but there are ways to enable creativity, and that is what Cleese focuses his talk on.

He starts off with an explanation of what creativity is, and it's not a personality trait, but a way of operating:

...the most creative had simply acquired a facility for getting themselves into a particular mood - a way of operating - which allowed their natural creativity to function. 

That mood is playfulness. They would play with ideas without any immediate practical purpose, but purely for the enjoyment of doing so. Cleese goes on to explain the two modes of operation that he labels the open and closed modes. Creativity requires an open mode of operation, but completion of the idea requires a closed mode of operation. To be most efficient, we must be able to readily switch between the open and closed modes.

The ability to get into the open mode requires five elements:

1. Space free from distraction
2. Time also free from distraction
3. Time playing with uncertainty before deciding
4. Confidence that silly thoughts may still be fruitful
5. Humor to keep the mind playful and exploratory

Once you achieve this setup, Cleese recommends about an hour and a half to play (with ideas)—a half hour to let your mind calm down and an hour of creative work. I realized that this time frame mirrors my own flow for blog posts pretty closely. Every post is a problem that I need to solve, and when I sit down at night to work on a post, I'll spend about a half an hour collecting my thoughts, looking through my notes, and considering what I want to write and how to write about it. Sometimes I have to wrangle my thoughts into submission because my mind wanders around, not wanting to focus on the task at hand. Then I'll have an hour or so of good productive time where I can write down thoughts and play with ideas in a pretty good state of flow. I'll reach a point sometime after the two hour mark where I start to tire out and my productivity slows down. I thought it had more to do with getting tired before going to bed, but it might be as much a factor of having worked creatively for an hour or two and I'm simply tired of being creative.

You probably noticed there was not one, but two time elements in the list. The second time element is surprising, and Cleese had a great justification for it:

One of my Monty Python colleagues who seemed to be more talented than I was never produced scripts as original as mine. And I watched for some time and then I began to see why. If he was faced with a problem and saw a solution he was inclined to take it even if he knew it was not very original.  Whereas if I was in the same situation, while I was sorely tempted to take the easy way out, and finish by 5 o’clock, I just couldn’t. I’d sit there with the problem for another hour and a quarter and by sticking at it, would in the end, almost always come up with something more original. It was that simple.

My work was more creative than his simply because I was prepared to stick with the problem longer. So imagine my excitement when I found this was exactly what MacKinnon found in his research. He discovered the most creative professionals always played with the problem for much longer before they tried to resolve it. Because they were prepared to tolerate that slight discomfort, as anxiety, that we all experience when we haven’t solved it.

I find this type of thing happening frequently when I'm trying to solve program design problems. I'll come to a workable solution fairly quickly most of the time, but if I pocket that solution for later and think about the problem a while longer, I can usually come up with a more elegant solution.

Humor may seem out of place in solving problems, but Cleese strongly held that creativity is closely related to humor - it is a way of connecting two separate ideas that results in something new and interesting. In the case of humor, it's funny, and in the case of creativity, it's useful for solving a problem. He certainly opposed the idea that humor and serious business should be separated:

There is a confusion between serious and solemn. Solemnity…I don’t know what it’s for. What is the point of it? The two most beautiful memorial services I’ve ever attended both had a lot of humor. And it somehow freed us all and made the services inspiring and cathartic. But solemnity serves pomposity. And the self important always know at some level of their consciousness that their egotism is going to be punctured by humor, and that’s why they see it as a threat. And so dishonestly pretend that their deficiencies makes their views more substantial, when it only makes them feel bigger…ptttttth.

Humor is an essential part of spontaneity, an essential part of playfulness, an essential part of the creativity we need to solve problems no matter how serious they may be.

Solemnity should have no place in the creative process. It kills our confidence in being able to play with ideas freely and distracts us from thinking fully about the problem by limiting thought to the safest, most acceptable paths. Humor is definitely more productive. My coworkers and I joke around a lot at work, and we get a tremendous amount of creative work done. On the other hand, I've never heard someone claim that their workplace is serious and they do especially creative work. More likely seriousness is accompanied with words like deliberate, careful, and protocol. Oh, and if a company tells you that they work hard, but they know how to have fun once in a while, that's a big warning sign.

Hard Work is Still Necessary

Creativity alone is not enough, and while creativity does involve hard work, coming up with a great idea to solve a problem is not the end of the task. Now comes the extra hard work of implementing the idea, resolving all of the details that you hadn't thought of before, and actually finishing solving the problem. Scott Berkun has written extensively on this topic, and it's some of his better writing. In his post on The Secrets of Innovation Secrets, he reminds us of how much work is involved in innovation:

If there’s any secret to be derived from Steve Jobs, Jeff Bezos, or any of the dozens of people who often have the name innovator next to their names, is the diversity of talents they had to posses, or acquire, to overcome the wide range of challenges in converting their ideas into successful businesses.

These people didn't succeed because of flashes of insight that just happened to them, and they didn't succeed by coming up with a great idea and selling it. The ideas themselves would have been nothing if it wasn't for their superb execution. (The importance of execution is one reason why patent trolls are so destructive to innovation—they haven't done any of the hard work of implementing an idea, but they want to get paid royalties simply for patenting an idea with no intention of doing the hard work themselves.)

Even the flash of insight that leads to an idea is the result of a lot of hard work. If it wasn't for all of the studying, researching, and experimenting that came before the actual moment of clarity, the idea may not have materialized at all. Scott Berkun again has some clear reasoning on why epiphanies are more of a process than an event:

One way to think about the experience of epiphany is that it’s the moment when all of the pieces fall into place. But this does not require that the last piece has any particular significance (the last piece might be the hardest, but it doesn’t have to be). Whichever piece of the puzzle is sorted out last becomes the epiphany piece and brings the satisfying epiphany experience. However the last piece isn’t necessarily more magical than the others, and has no magic without its connection to the other pieces.

I have this experience all the time when I'm trying to solve problems (which is also pretty much all of the time). I'll be mulling over a problem for hours, days, or weeks, struggling to find a workable solution, and then the answer hits me all of a sudden, usually while I'm laying down to go to sleep. I call it Bedtime Debugging because that's when it normally happens to me, but other people have it happen when they're taking a shower or brushing their teeth. It feels like a sudden eureka moment, but it would never have happened if I hadn't spent all of that time thinking about the problem, researching options, and studying the code I'm working on. The final connection that made everything make sense may have happened in a moment, but bringing all of the other pieces of the puzzle together so that that moment could happen took much more time.

Preparing for Epiphanies

I spend huge amounts of time reading and learning. I never know when I'll need to use a particular piece of knowledge, and the more of it that I have at my disposal, the better. Constantly exercising my mind and learning new things also keeps my thinking process flexible so that I can connect ideas from different fields and think about problems in new and different ways.

Our culture tends to romanticize the eureka moment while ignoring all of the hard work that's involved in the process because the eureka moment is so much more exciting than the work that came before it and must follow it. For one of innumerable examples, Cal Newport, an assistant professor of computer science at Georgetown University, contrasts the theatrical impression of Stephen Hawking's discovery of Hawking Radiation in The Theory of Everything with the reality:

In a pivotal scene in the Stephen Hawking biopic, The Theory of Everything, the physicist is staring into the embers of a dying fire when he has an epiphany: black holes emit heat!

The next scene shows Hawking triumphantly announcing his result to a stunned audience — and just like that, his insight vaults him into the ranks of scientific stardom.…

In reality, Hawking had encountered a theory by two Russian physicists that argued rotating black holes should emit energy until they slowed to a stationary configuration.

Hawking, who at the time was a promising young scientist who had not yet made his mark, was intrigued, but also skeptical.

So he decided to look deeper.

In the (many) months that followed, Hawking trained his preternatural analytical skill to investigate the validity of the Russians’ claims. This task required any number of minor breakthroughs, all orbiting the need to somehow reconcile (in a targeted way) both quantum theory and relativity.

The reality of Hawking's discovery is a clear example of the hard work involved in solving big problems. He needed to have a lot of knowledge about multiple complex fields of physics and develop new advances in those fields over a long stretch of time to make progress towards his goal of answering a question he had about a theory he had come across. Creativity was absolutely involved in this process, but the eureka moment is a vanishingly small part of the story. These instances of major breakthroughs and discoveries through hard work are the norm, rather than the exception. After all, if these moments were as simple and easy as they are portrayed, they would be much more common than they actually are. We can't all just sit staring into a campfire or reading under a tree and expect all of the answers to hit us in the head.

In a way, creativity is only a small part of succeeding in solving a big problem, such as producing a great product, creating a market around it, and building a business from it. The idea of what product to make is a drop in the bucket compared to the colossal amount of work involved in this process. Yet creativity is also an integral part of the entire process. Every step of the way there are little problems to be solved and details to be resolved, and doing this creatively is incredibly important. Creativity and hard work are intimately woven together in the process of solving big problems.

Tech Book Face Off: Database Design for Mere Mortals Vs. Seven Databases in Seven Weeks

In my quest to fill in some of the holes in my software development knowledge, I picked up a couple books on databases to read through. I somehow made it this far without studying anything in depth about databases, and up until now, when I needed to interact with a database I searched for what I needed at the moment and muddled my way through.

When I want to learn about a topic, I like to go through at least two books to get different perspectives and hopefully get a broader and more thorough understanding of the topic. This time I was not disappointed, and I picked two dramatically different books that worked quite well together to cover the vast subject of databases. Database Design for Mere Mortals is a book about how to design a database in a simple, straightforward way. Seven Databases in Seven Weeks: A Guide to Modern Databases and the NoSQL Movement is exactly what it sounds like, a book that runs through the features, advantages, and disadvantages of seven different modern databases to give you an overview of what's currently available and what these databases are capable of. Let's take a closer look at what each of these books has to offer.

VS.

Database Design for Mere Mortals

Databases are the foundation of many types of software applications, and I wanted to get a working understanding of how to design a database well. Michael Hernandez does an excellent job in this book laying out a process for designing a database with good data integrity and consistency, two important goals when designing a database. His writing style is direct, clear, and pleasant, and I found this book surprisingly easy to read considering it's covering a topic that threatens to be utterly dry and boring.

The book is split into three parts: an overview of relational database design, the details of the design process, and some other database design issues. The first part covers the history of database models, what a relational database is, the design objectives for a database design process, and the terminology of relational databases. The relational database that we know and love was conceived all the way back in 1970 in more or less it's current form with tables, fields, and relationships. Like with so many of the computer innovations of the 60s and 70s, we are standing on the shoulders of giants today with relational databases.

The second part of the book contains the majority of the content and lays out the details of the database design process. The process starts with defining the goals and objectives of the database and analyzing the current database that's in use, if there is one. This step, as well as most of the other steps in the process, involves numerous interviews with various stakeholders of the database, and example interviews are spread throughout this part of the book. The process continues with establishing the table structure of the database, defining the different types of keys for the tables, writing field specifications, designing table relationships, establishing business rules, defining views to represent database queries, and reviewing data integrity.

The third part of the book wraps up with some bad design practices to watch out for and advice for when you might think about bending or breaking the rules of good database design. Throughout the book, Hernandez focuses on the relational database and doesn't even bring up other modern database models, like the models covered in Seven Databases in Seven Weeks, but the process he lays out could be adapted to other types of databases fairly easily.

Early in the book Hernandez takes a decidedly database-centric view of application design by assuming the database is the starting point of an application, and the data structure should be established first with the rest of the application being built up around it. It's worth noting that this is but one way to design an application, and it normally results in an application with a data-centric feel to it. Another way to design an application is to start with the user interface (UI) and user experience (UX) and design the database to work with the application structure that results. Such an application will have a much more user-centric look and feel. Both approaches have their own benefits and issues, and of course a real project will likely be a hybrid of these two extremes because the design of the database and user interface is not going to be a serial process.

As for the database design process itself, I was surprised at how similar it is to good software design. Much of good database design boils down to eliminating duplication in the database, choosing descriptive names for everything in the database, and making the database flexible enough to change easily. Good software design will follow these same principles. Most of the book felt very familiar except that the terminology referred to databases instead of programming languages.

The reason that duplication in a database is a bad thing is that if the same field is replicated in multiple tables, then the data for those fields needs to be constantly kept in sync. It becomes much more likely that the data will be updated in one place and not the other, causing data integrity problems. The process of eliminating duplication is referred to as normalizing the database, but using Hernandez' design methodology, duplication is never allowed into the database, so normalization shouldn't be necessary. The equivalent principle in software design is the DRY principle, and it is usually achieved through refactoring.

Naming is one of the hardest problems in software design, and it seems from this book, the same is true of database design. At least half of the book is about how to name various database structures, and many times naming a table or field will provide clues as to how it should be changed to make the design better. One place where naming can reveal design issues is when a table has a multi-part field. A multi-part field is a field with values that have more than one distinct subject. The canonical example is an address field that contains full addresses as string values. An address consists of a street name, city, state, and zip code. A better design would have separate fields for each of these parts of the address.

While the advice on naming was generally good, and I learned quite a lot about subtle naming issues in database design, some of the advice struck me as odd. Hernandez recommends using an abbreviated form of the table name as a prefix to disambiguate field names that would otherwise be the same across tables, like when there are address fields for employees, customers, and suppliers in their respective tables. He advises having names like "EmpAddress," "CustAddress," and "SuppAddress." Later he abbreviates things like "Classes" to "Cls" and "Student" to "Std," which makes even less sense to me. I would think this practice would add confusion, and adds mental effort by having to do the translation in your head every time you use these abbreviated fields. I've come to appreciate the advantages of not abbreviating names in programming, and the cost of a little extra typing is not terribly significant. I would prefer the unabbreviated names for clarity.

Another issue I had with this design process was how much it seemed like a Big Up-Front Design (BUFD) process. The entire database design is decided in the beginning through meetings and interviews, and Hernandez recommends everything be documented in paper folders. And I mean everything:

Once you’ve identified which web pages you need to review, take a screenshot of each page. Copy the screenshots into a word processing document, print the document, and then store the document in a folder for later use.

It seemed quite rigid and optimistic to me with how much requirements can change during a development project. Maybe there's something about database design that requires this type of BUFD, but I can see why programmers can get frustrated with the database team if this is how they have to operate. I wonder if there's a way to make the design process more flexible and (dare I say) agile without sacrificing data integrity and consistency in the database design.
Despite these misgivings, Database Design for Mere Mortals was an excellent introduction to database design overall. Hernandez' approach was simple, straightforward, and methodical, and he did a great job of laying out a process that should be accessible to most people. In his own words, database design isn't hard:

I’ve always believed that you shouldn’t have to be a rocket scientist in order to design a database properly. It should be a relatively straightforward task that can be performed by anyone possessing a good amount of common sense. As long as you follow a good database design method, you should be able to design a sound and reliable database structure.

I always thought that to be true, from what little I knew about databases, and it's good to have the confirmation from an expert. I definitely recommend this book to anyone looking for a good way to design a database effectively. Maybe you can even find a way to make it more adaptable to change.

Seven Databases in Seven Weeks

I've been wanting to read one of these Seven X in Seven Weeks books for a while now, and this is the first one I was able to get to. It was excellent. I'm amazed by how much Eric Redmond and Jim Wilson were able to fit into 350 pages, and it's entirely coherent while being an immensely enjoyable read. They do actually cover seven databases in reasonable detail in this book, dedicating roughly 45 pages to each database.

It's an interesting exercise to think about. If you had to cover a database in 45 pages, what would you include? What would you leave out? I think the authors struck the perfect balance. They decided not to cover installation of any of the databases, nor did they explain much about the programming languages that they used to interface with the databases (and they used an impressive number of languages: Ruby, Java, JavaScript, Groovy, and Gremlin). You were either expected to know it or be able to figure it out on your own. That was fine with me because they covered a ton of stuff in the short amount of space available to each database, and their explanations were crisp, brisk, and understandable.

The basic format of the book is one chapter per database, and each chapter is split into three sections representing three days of study. The first day covers the basic API and main features of the database. The second day goes through the more advanced features of the database that make it unique and interesting. The third day does something more involved using a large data set to populate the database and showing what kinds of structures and queries are best suited to that particular database. The end of each day includes some homework exercises to guide the reader in learning more about how each database works.

I thought this format worked extremely well, although the "seven weeks" in the title makes it seem like it would take longer to get through the book than it actually does. Since each chapter consists of three days, you could finish everything in three weeks if you did something everyday. Each day's work is also short enough that it is conceivable that you could squeeze seven databases into seven days instead of weeks, but you wouldn't be able to do anything else other than eat, sleep, and study to do it, so probably not realistic.

At any rate it's a great way to learn something significant about seven databases in a short amount of time. You'll learn their strengths and weaknesses and how to interface with them through code as well as the provided command line or HTTP interface. If you were looking for a new database to use on a project, this book would give you all the right information to figure out which database would work best as well as get you started on using it for your project.

I've come this far without even mentioning what the seven databases are, so I'll stop raving about the book and briefly describe each database that the authors cover.

PostgreSQL
PostgreSQL is a traditional relational database that has proven itself many times over on large software projects. As a relational database, the basic structure of the data it stores is in the form of tables that are made up of fields that describe characteristics of the data and rows that hold the values of those characteristics for specific entries in a table. Tables can be linked together with uniquely-identified primary keys, and queries can filter and sort the table data and use keys to find linked data in very flexible ways.

PostgreSQL has the advantages of being fast and having great driver support in most programming languages. It's a mature, reliable database with decades of development behind it, and being a relational database, most developers will be familiar with how it stores data and how to query it. As for weaknesses, it does suffer from scalability issues if you need to partition it onto multiple servers. Most relational databases don't handle partitioning nearly as easily as other types of databases. Also, if your data doesn't fit well into a rigid schema structure or mostly contains large blobs of data, PostgreSQL probably isn't the right database for your data.

Riak
Riak is a distributed key-value store, which means it's built from the ground up to run on multiple servers, and it stores and retrieves data using unique keys for each piece of data. The data can be anything from plain text to video clips, and the keys can be automatically generated or manually assigned. Data can be further divided into buckets to separate key-value pairs in a logical way. The interface to Riak is HTTP requests, and all the normal CRUD (Create, Read, Update, and Delete) operations are done through a slick REST interface using URLs so it's easy to interact with Riak through any programming language's HTTP library.

Because Riak is distributed, it cannot be fully consistent, available, and partition tolerant at the same time. This type of trade-off pops up in all kinds of places. In project management you can't optimize fully for time, cost, and features. In programming you can't optimize code fully for performance, readability, and development time. You have to pick two of the three options. Riak steps around this problem by allowing you to pick two options for each request to the server. One write to the database could be consistent and available, and the next write could be available and partition tolerant instead. That's cool.

HBase
The authors' description introduction to HBase is pretty good: 

Apache HBase is made for big jobs, like a nail gun. You would never use HBase to catalog your corporate sales list, just like you’d never use a nail gun to build a dollhouse. If your data is not measured by many gigabytes, you probably need a smaller tool.

It's a column-oriented database similar to Google's BigTable database, and it's designed to be fault tolerant and highly scalable. The structure of the database is a set of tables with each table containing rows that are kept sorted by the row key. The rows are divided into column families, and each column family in a row can have it's own set of columns. The same column family in two different rows can contain different columns. Every change to a row will create a new version of the row, so the database provides revision control for free.

Being a highly scalable database, HBase's major weakness is that it's not for small data sets. Like the quote says, if you're not dealing in huge amounts of data, HBase is going to be overkill.

MongoDB
MongoDB is a document database. Instead of storing data in tables, columns, and rows or key-value pairs, it stores data in documents. A document is a JSON string that contains an _id field and any number of other fields with a name and a corresponding value. That value can be a string, a number, an array, or even another hash, which is what the base document is. There is no defined schema, so each document can be completely different than the others, although in practice such an ad hoc database would be unusable. The application is responsible for maintaining the data structure that it requires.

MongoDB is another scalable database designed for storing massive amounts of data and running on large clusters of machines. It has more flexible querying capabilities than Riak or HBase, but you can easily shoot yourself in the foot with its lack of schemas. The application's data model has to be well defined, or else the database can quickly become a humongous mess.

CouchDB
CouchDB is a document database like MongoDB, and it has a REST interface like Riak. It was designed to be simple and flexible, and it can run on almost anything—from large data centers to your smart phone. It never deletes data stored to it. When you update a document, it creates a new version of it so you get revision control for free, and this mechanism is also the way CouchDB ensures data consistency in a distributed system.

Neo4j
Neo4j is an entirely different type of database, as the authors describe:

Neo4j is a new type of NoSQL datastore called a graph database. As the name implies, it stores data as a graph (in the mathematical sense). It’s known for being “whiteboard friendly,” meaning if you can draw a design as boxes and lines on a whiteboard, you can store it in Neo4j. Neo4j focuses more on the relationships between values than on the commonalities among sets of values (such as collections of documents or tables of rows). In this way, it can store highly variable data in a natural and straightforward way.

It would be perfect for storing a family tree or, as explored in the book, actor connections in movies so that you could query for actors six degrees from Kevin Bacon. It's an interesting departure from most other databases, and it's great for researching certain kinds of data problems.

Redis
Redis is another key-value store like Riak, but its defining characteristic is its speed. Redis is fast, and if it's configured to be a purely in-memory store, it's blazing fast. It's more like a cache than a database when set up this way, and normally it is backed by one or more other databases. Redis is used to speed up access to other databases more so than storing data itself. It also has a very fully-featured command line interface with over 120 commands that make it easy to use and integrate into a larger system.


These summaries give you an inkling of what's covered in Seven Databases in Seven Weeks, but I can't begin to do any of these databases justice in this short space. The authors even claim that they only scratch the surface, although I would say they do a phenomenal job of it. It was quite an entertaining read, and I finished it with a much better understanding of the wide variety of databases out there and what their strengths and weaknesses are. The appendix even includes a great comparison table of the different features and capabilities of the seven databases, a great reference to come back to later. I highly recommend this book for anyone who needs to find a new database to solve a gnarly problem or who is curious about the different kinds of databases available beyond the ubiquitous relational database.

A Crash Course in Databases

I set out to learn some useful things about databases, and with these two books, I definitely accomplished that. Database Design for Mere Mortals gave me a good understanding of the issues involved in designing for a relational database and how to solve common database problems. The design process should be fairly easily adaptable to other types of databases as well, and it gives a clear picture of where things can go wrong that will come in handy when you're designing for a schema-less database. Seven Databases in Seven Weeks exposed me to a wide variety of databases in a great hands-on way that was incredibly engaging. I'm looking forward to reading more of the Seven X in Seven Weeks series, and I hope they're all as good as this one.

Between the two books, I learned a ton of stuff about databases, but not everything, of course. There wasn't much coverage of relational algebra or the implementation of some of the fundamental features of databases, like indexing or storage management. I'll have to find a different book for those topics, but these two books were excellent in what they covered. They complement each other well with almost no overlap in material, and together they're a great crash course in databases.

Avoid Analysis Paralysis by Experimenting

Have you ever been stuck on a problem, not because you didn't have any ideas for how to solve it, but because you had too many? It's a common situation to fall into in software engineering because any given problem will have a multitude of solutions, and each of them boil down to writing and changing the right code, a process that can be quite quick and has a tight feedback loop. It's a much faster process than designing complex hardware systems, like integrated circuits or automobiles, and the turn-around time on an idea can often be measured in minutes to days instead of months to years. Often the programmer is faced with numerous potentially promising solutions at once. With so many choices and fast implementation times, a good way to get out of the analysis paralysis rut is to roll up your sleeves and start experimenting.

Software is flexible and malleable in a way that hardware is not. Software is more like writing or movies or music in that it can be bent, stretched, molded, and rearranged on a whim, so trying out ideas should be the default method of attacking small- to medium-sized problems. Obviously, attacking large problems in software by constantly tearing up and replacing the basic architecture of the code base will inevitably lead to disaster. Those decisions require more forethought and design, but for the day-to-day problems we all have to deal with, experimentation is a great way to discover what works best and keep moving forward.

If you find yourself in a discussion or meeting, debating the best course of action on a development problem, and you start to realize that the combined time of everyone in the room is approaching the time it would take to implement at least two of the options, it's probably better to start spinning up some experiments to prove out a winner instead of continuing to hash out arguments that have no clear answer.

One reason why we may resist experimenting in favor of discussions is that we have a mental block for changing working code in an unfinished project. Even though version control makes it easy to try out ideas and roll them back, it can still become harder and harder to make changes to a project as it matures. The project builds up inertia, and starts to resist change. Scott Berkun has a nice article about how ideas can become too precious, and they start to take on a life of their own:

Being precious means you’re behaving as if the draft, the sketch, the idea you’re working on is the most important thing in the history of the universe. It means you’ve lost perspective and can’t see the work objectively anymore. When you treat a work in progress too preciously, you trade your talents for fears. You become conservative, suppressing the courage required to make the tough choices that will resolve the work's problems and let you finish.

In the case of a software project the precious ideas are the accumulated work on the project that exists in the current state of the code base. To make progress, we have to push against the inertia of the code base and force improvements through experimentation. To push things in a new direction, branch the code and start experimenting with options until one emerges as a clear winner. If there is no clear winner, then you can still pick something from the implemented options and run with it, and you haven't wasted time debating options that didn't have much differentiation anyway.

When you're experimenting, don't be afraid to throw stuff away. It can feel like you're writing a lot of code that will never get used, but the code is not the important part. You're not throwing away work, you're gaining knowledge. The superfluous code is just a byproduct of gaining that knowledge. Over time the knowledge will accumulate and mature into wisdom. You'll be able to make better decisions, sidestep more pitfalls, and create viable solutions faster with the buildup of knowledge from regularly experimenting.

Experimentation turns out to be a useful way to make progress in many fields, some not so obvious. Take mathematics for example. My early impression of advanced mathematics was along the lines of mathematicians standing in front of blackboards, pondering proofs until they had an epiphany, and then writing down the proof in a burst of genius. The reality is quite different. Mathematics does involve plenty of thinking, of course, but it also involves plenty of experimenting with ideas. Often a hard problem must be broken up into pieces or changed into an easier problem to solve first. Lemmas, propositions, and minor theorems need to be built up to help in proving a major theorem, and a lot of the work of mathematics is building these tools to create the final product. If experimentation can work for such an abstract field as mathematics, it can certainly work for software development.

To get better at experimenting, it's important to get comfortable with spinning up an experiment quickly. The faster you can get a programming experiment started, the less friction there will be when you need to run an experiment to make a decision. I'm reminded of when I used to throw pottery (not literally picking up pots and throwing them, but making ceramic pots on a potter's wheel). When throwing pots, the first thing you have to do is center the clay on the wheel so that when the wheel spins at high speeds, the clay doesn't wobble at all. If the clay isn't centered properly, the pot won't stay symmetrical and may very well tear itself apart before you finish it. Much of learning to throw involves learning how to center, and once you can do it well and do it quickly, you can make more and better pots. You can run a lot more throwing experiments with your wheel time, and you more quickly discover what works and doesn't for making beautiful pottery.

The analog in programming is to get good at writing the necessary startup code to get to a basic working program as fast as possible. If you do web development, practice bringing up a new website and connecting it to a database. If you do embedded development, practice starting a new embedded application with tasks, interrupts, and communication interfaces that you use regularly. If you do mobile development, practice creating a new app and hooking up a basic GUI. Normally, you only do these things once at the beginning of a project, so the process isn't always fresh in your mind. If you practice it, then you can do it quickly and run more experiments with new code without having to tear up an existing architecture to do it.

Finally, experimenting with problems in code and trying out solutions will uncover issues that you would have never thought of otherwise. Sometimes a solution will fit right into the existing code base, much better than you thought it would, and can even make the code it interacts with simpler and more elegant. Other times it will require major transformations of other algorithms, data structures, and interfaces that you may not have thought through when the solution was an image in your mind's eye. Experimentation provides useful feedback on those things that you may have missed, and when you experiment with real code, you'll often find that you didn't need the complex, high-performance solution that you thought you did.

The ability to experiment is a useful tool to have in your programmer's toolbox. The results of an experiment are much more convincing than vague arguments when deciding among a set of options. Don't worry about writing code that may be thrown away because the useful work is the knowledge that's gained, and that knowledge can be reused to solve similar problems in the future. Instead, worry about getting faster at running experiments by practicing how to bring up new applications quickly in your environment. Then you can get the definitive answers you need to make progress.

Exploring the Tension in Software Abstractions

Ever since we started making tools as a species, we've been dealing with abstractions. We've been adding layer upon layer of abstraction to every form of work imaginable since those first hand-made chisels and knives long ago. With every layer of abstraction that we add, we can do the same amount of work with less effort or more work with the same amount of effort. Those layers of abstraction don't come for free, though. They add complexity. Especially within software engineering, where we have fine control over abstractions and can add many layers of abstraction even within one program, a tension exists between the abstractions we create and the complexity that results. We have to make intelligent choices with those abstractions while being aware of the trade-offs.

Movement and Computation

To get an idea of how we use abstractions to increase the efficiency of doing work, we can look at something physical that we've abstracted over time, like movement. How we get from point A to point B has had a huge impact on the development of civilization, and we've been abstracting how we get around for ages. We started out on our own two feet, but once we figured out how to train other animals to let us ride them, we could move faster with less effort and carry more stuff with us. That abstraction didn't come for free, though. We had to invest time training horses, donkeys, and camels, caring for our animals to keep them alive and healthy, and making equipment like saddles, bridles, and stables to improve the animals' usefulness. The additional work we had to do to support the abstraction of using animals for movement was worth the effort, and we could move around much better and get much more done than what we gave up in time spent caring for horses.

Another abstraction we layered on top of horse riding was attachments: buggies, wagons, and carriages. We could attach essentially a platform on wheels (with many variations of platform) to one or more horses so that we could carry much more stuff for much longer distances in more relative comfort than we could on horseback alone. This abstraction added more work to build, maintain, and repair the wagons, but it was still a big win in efficiency.

From here we could look at all kinds of other abstractions like bicycles, trains, airplanes, and rockets. Every abstraction of movement has its own advantages, trade-offs, and applications, but the one we'll look at is the automobile. It has replaced the horse with horsepower in the form of an engine (or most recently an electric motor) and combined that with the wagon for transporting people and stuff. We have spent massive amounts of time and energy designing, manufacturing, and supporting this abstraction, and we've made great gains in efficiency and convenience because of it.

Anyone who owns a car knows that cars need to be maintained. Periodically, the oil and filters need to be changed, the brakes need to be replaced, and various components need to be inspected for wear. Most of the time you don't need to know much about how the car works. You just need to know how to drive it to get around, but every once in a while the abstraction leaks. The car breaks down, and if you don't know enough about the abstraction to go under the hood and fix it, you might have to hoof it without the actual abstraction of a horse. Although, you likely have a modern abstraction of communication to help you line up a tow truck, a mechanic, and a ride to where you need to go.

Now compare the abstractions of movement we've developed with abstractions of computation. They have a lot in common. Computation has gone through its own series of abstractions, from teams of human "computers" that did calculations by hand to huge computing machines that read punch cards and used vacuum tubes for logic to the modern day computer with billions of transistors, terabytes of storage, and teraFLOPS of performance (at least). A modern software program is also built on many layers of abstraction, from a high-level programming language expression at the top to the silicon bandgap at the bottom.

Like transportation abstractions, our computation abstractions require huge amounts of design, manufacturing, and support effort. We have to maintain and repair our computers, and all of the abstractions that make them up can leak from time to time. They also allow us to do huge amounts of work with orders of magnitude less effort than we could do without them. Computers enable us to do all kinds of things that would be impossible without them, like measure the creation of previously undiscovered particles.

Layers of abstraction exist within a software program as well. Every class and function provides an abstraction of the computation that's done within it, and there are some trade-offs to consider when deciding when and where to add these abstractions. Software is quite flexible, and it's relatively easy to add or remove layers of abstraction to hide or expose more of a computation in a given function. That ability to finely layer abstractions for little cost changes the analysis a bit from the big abstractions of automobiles or microprocessors. In software design we are constantly doing small cost-benefit analyses, and that adds another dimension to the choices we make when writing code.

Queues and Logs

The trade-offs involved when choosing where to add abstractions to a program create a tension in the design of the program. There is a constant pull at every level of the design between hiding implementation details so that they can be ignored and showing those details so that they can be reasoned about in the context of where they are used. To explore this tension, let's look at an example.

Let's assume we're designing a logging feature for an embedded system. The embedded system could be anything, but let's say it's a robot. We want to keep track of vital information about the state of the robot while it's running, and be able to read back that state if something goes wrong to get clues about where the program messed up.

We don't have infinite space for this log, especially because it's an embedded system, so a circular queue would be a good data structure to use. We can add new log entries to the front of the queue and read them from the tail of the queue. Once the queue gets full, it will start overwriting the oldest entries at the tail of the queue with the newest entries that are being written to the head. Here is a basic implementation of a queue written in C:

/*
 * Queue.h
 */

#ifndef SOURCES_QUEUE_H_
#define SOURCES_QUEUE_H_

#include <stdint.h>

typedef uint8_t * QUEUE_ELEM_PTR;

typedef struct QUEUE {
  QUEUE_ELEM_PTR base;
  QUEUE_ELEM_PTR head;
  QUEUE_ELEM_PTR tail;
  uint32_t elem_size;
  uint32_t elem_count;
  uint32_t q_size;
} QUEUE, *QUEUE_PTR;

void Queue_init(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_q_base,
                uint32_t elem_size, uint32_t q_size);
void Queue_enqueue(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_new_elem);
int32_t Queue_dequeue(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_elem);

#endif /* SOURCES_QUEUE_H_ */

/*
 * Queue.c
 */

#include <string.h>
#include "queue.h"

void Queue_advance_ptr(QUEUE_PTR p_q, QUEUE_ELEM_PTR *p) {
  (*p) += p_q->elem_size;
  if (*p >= p_q->base + p_q->q_size) {
    (*p) = p_q->base;
  }
}

void Queue_init(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_q_base, 
                uint32_t elem_size, uint32_t q_size) {
  p_q->base = p_q_base;
  p_q->head = p_q_base;
  p_q->tail = p_q_base;

  p_q->elem_size = elem_size;
  p_q->elem_count = 0;
  p_q->q_size = q_size;
}

void Queue_enqueue(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_new_elem) {
  QUEUE_ELEM_PTR p_write = p_q->head;

  Queue_advance_ptr(p_q, &p_q->head);

  // Advance the tail if the head has caught it.
  if (p_q->head == p_q->tail) {
    Queue_advance_ptr(p_q, &p_q->tail);
    p_q->elem_count--;
  }

  // Now add event in the old location.
  memcpy(p_write, p_new_elem, p_q->elem_size);

  p_q->elem_count++;
}

int32_t Queue_dequeue(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_elem) {
  if (p_q->tail == p_q->head) {
    return -1;
  } else {
    memcpy(p_elem, p_q->tail, p_q->elem_size);
    Queue_advance_ptr(p_q, &p_q->tail);
    p_q->elem_count--;
    return p_q->elem_count;
  }
}

This queue implementation is an abstraction, and we can already see some tension between it and the code that would use it. For instance, the QUEUE structure contains an element size and a queue size that are set by the caller, and the Queue_init routine doesn't check to make sure that the queue size is a multiple of the element size. It's the caller's responsibility. If this were a data structure that would be part of a library, maybe it would be appropriate to do the check for the caller. Then we would have to define an error code and make sure the caller checked the error code if there was a risk of misaligned elements. Since this is for an embedded project where all of the source code would be available to the developer and the number of developers that will be using the code is relatively small, it might be okay to forgo the check. There's no clear answer, and that creates tension.

Some other trade-offs that were made were that the queue doesn't allocate any memory—the caller is responsible for that, too—and the queue enforces that the head and tail only point to the same element if there are no elements in the queue. That means the queue can never be truly full. It will always advance the tail and remove an element if the head catches the tail.

With this queue, we could fairly easily create a log structure that uses the queue as an abstraction and offloads some of the complexity of logging so that we don't have to think about all of the details at once. It's clear that putting the code for handling the queue in one place is better than having it spread throughout the logging code, and it allows us to leverage the abstraction to get more work done, much like using a car to run errands.

The queue's functions also adhere to the Single Responsibility Principle (SRP), with each function doing one thing. The one possible exception is the Queue_dequeue function, which returns the number of elements left in the queue. It's a slight deviation from SRP, but it's very convenient when looping through the queue to know when the last element was just dequeued. Again, we see that there's tension about what the best design choice is.

What if there's a crash?

It's all well and good that we have logging to a circular buffer for our robot, but what if the code crashes and reboots? What if the robot starts running amok and we have to hit the shutdown switch? The queue only resides in memory, so in most failure cases, it's likely to be lost before we could read it. To make the log more permanent, we want to write it to non-volatile memory. Most embedded processors have internal flash, so we could use that flash or attach an external flash memory to store the log. Two problems still remain with this solution, our queue implementation doesn't know anything about writing and erasing flash or about initializing memory that may have some valid elements in it already.

Let's start with the problem of initializing the head and tail pointers for memory that might already have values in it. Erased flash memory always has a value of 0xFF (hexadecimal notation), so we can use that value to figure out where the head and the tail of the queue are. Starting from the base of the queue, (or anywhere within the queue, really) the tail will be the first non-erased element after an erased element, and the head will be the first erased element after a non-erased element. We can add this code to the Queue_init routine like so:

bool Queue_element_is_blank(QUEUE_PTR p_q, QUEUE_ELEM_PTR p) {
  QUEUE_ELEM_PTR p_end = p + p_q->elem_size;
  for (; *p == 0xff && p < p_end; p++) ;
  return (p == p_end);
}

void Queue_init(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_q_base,
                    uint32_t elem_size, uint32_t q_size) {
  p_q->base = p_q_base;
  p_q->head = p_q_base;
  p_q->tail = p_q_base;

  p_q->elem_size = elem_size;
  p_q->elem_count = 0;
  p_q->q_size = q_size;

  /*
   * head = 1st blank entry after a non-blank entry
   * tail = 1st non-blank entry after a blank entry
   */
  bool was_blank = true;
  for(QUEUE_ELEM_PTR p_elem = p_q->base;
      p_elem < p_q->base + p_q->q_size;
      p_elem += p_q->elem_size) {
    if (!Queue_element_is_blank(p_elem)) {
      if( was_blank ) {
        p_q->tail = p_elem;
      }

      p_q->elem_count++;

      was_blank = false;
    } else {
      if (!was_blank) {
        p_q->head = p_elem;
      }
      was_blank = true;
    }
  }
}

Adding this code to Queue_init shows another form of tension in writing functions. The two parts of the function, assigning values to the structure and finding the head and tail, could have been pulled out into their own functions and called from Queue_init. That would make Queue_init simpler because it would only be two function calls, and it would be much easier to understand at a glance. It may also be harder to introduce bugs because each step of the initialization process is isolated in its own function. Plus, a developer that only wanted to check how the structure is initialized or to look at the head-and-tail finding algorithm could focus on that piece of the code because it's in its own function.

On the other hand, keeping all of the initialization steps together at the same level of abstraction has its own advantages. It may be more complex, but not substantially so. The whole function is only about 30 lines of code. Having all of the code in one place makes it easier to see the entire behavior of the code at once. We don't have to jump back and forth between three different functions to see everything that it's doing. Seeing all of the details at once, as long as it's not too many, can make it easier to change the function, if needed, and see bugs that might be hidden between layers of abstraction. It's a difficult design choice, and every case will have its own conditions that tilt the balance one way or the other. In this case, it's probably best to keep all of the details together.

For the case of writing and erasing the flash in the Queue_enqueue function, things are a little different. The queue doesn't really need to know how to write and erase the flash, so those operations can be separate functions that are called when needed. However, the queue does need to know a little about the structure of the flash because normally flash memory locations can only be erased a block at a time, with a block being a range of addresses. To keep track of the tail, the queue will have to know the block size and advance the tail by that much whenever it erases a block. The new Queue_enqueue code looks like this:

int32_t Queue_enqueue(QUEUE_PTR p_q, QUEUE_ELEM_PTR p_new_elem) {
  QUEUE_ELEM_PTR p_write = p_q->head;

  Queue_advance_ptr(p_q, &p_q->head);

  // check if we need to erase things
  QUEUE_ELEM_PTR next_block = (p_q->head + FLASH_BLOCK_SIZE);
  bool tail_in_block = p_q->head <= p_q->tail && p_q->tail < next_block;
  bool erase = !Queue_element_is_blank(p_q, p_q->head);

  if (erase) {
    Flash_erase((uint32_t)p_q->head);
    if( tail_in_block ) {
      //need to move the tail to the next block
      p_q->tail = next_block - p_q->elem_size;
      Queue_advance_ptr(p_q, &p_q->tail);
    }

    uint32_t events_per_block = FLASH_BLOCK_SIZE/p_q->elem_size;
    p_q->elem_count -= events_per_block;
  }

  // now add element in the old location
  uint32_t ret = Flash_Write((uint32_t)p_write, p_q->elem_size, p_new_elem);

  if (ret) {
    p_q->head = p_write;
  } else {
    p_q->elem_count++;
  }

  return ret;
}

The function now returns an error code if the flash could not be written for some reason, and the FLASH_BLOCK_SIZE constant would be defined in the code for the flash. It would be nice if the queue didn't need to know about the flash because that's a different level of abstraction, and it's better to keep the entire function at the same level of abstraction. However, the head and tail management is bound up with the flash structure now, and it would probably add more complications and be more confusing if we tried to separate them. The other alternative, moving some of the head and tail management into the Flash_erase code, is even worse because the flash will most likely be used to store other things as well and mixing code for the queue into it would pollute the flash code.

In the end the compromise taken here isn't too bad because the function is still relatively short and understandable. Because it's an embedded system, not a general library, the implementation is already tied more closely with the hardware it's running on, and these kinds of compromises are made for the sake of code size and efficiency. A log in a different type of system would warrant a different choice, but then again, it may have a hard drive instead of flash, in which case the implementation would be totally different. Code should be designed for the context in which it is used.

This code is now a good starting point for implementing a logging function for our robot. The abstractions strike a good balance, and the benefits of using the queue with flash support likely outweigh the cost of understanding and maintaining it. It may not be as big of a jump as moving from a horse and buggy to a car, but programming is full of abstraction opportunities, and every well-designed abstraction helps get a little more work done for a little less effort.

Not Too Tight, Not Too Loose

We deal with abstractions all the time, and building abstractions on top of abstractions is how we make progress with technology. In software development it is especially apparent that abstractions are everywhere. The layers of abstraction within a program are thin, and they have a definite tension at the boundaries. Making the abstractions tighter can hide unnecessary details and improve code decoupling while making them looser can expose more interactions and improve code awareness. Good design balances the tension between those two concerns.

Optimizing Performance Requires an Open Mind

Performance optimization can be great fun. Finding new ways to make a code base run faster, have more throughput, or support more users is immensely satisfying. Pursuing optimizations can also lead you down plenty of dark alleys with disappointing dead ends. Figuring out which optimizations to pursue and avoiding the ones that end up making performance worse requires decent knowledge of the technology stack that you're using and an open mind to explore the code to find where it most needs to be sped up.

Silicon Vs. Grey Matter

Our intuition can easily lead us astray when trying to reason about what parts of a program are slow and should be the target of optimization. One reason that this is such a common problem is that the human brain processes things differently than a computer. While the brain is a massively parallel processing structure, the microprocessor is essentially a sequential processing machine. Sure, it does do some parallelization, but not nearly on the level of the brain, so the types of computation that are easy in each system are vastly different.

These large differences in computational models between the brain and the microprocessor lead to misconceptions in what we think would be the optimal algorithm for any given problem. To make up a simple example, consider what you would do if given a huge matrix of numbers and asked to manually change them all to zero to reset the matrix. Assume that it is a sparse matrix, so most of the numbers are already zero. Why would you do this manually? I don't know. It's a contrived example, so humor me.

Since it is a sparse matrix, the fastest way to clear the matrix is probably to find the non-zero values and change them to zero. The reason this algorithm can be stated so simply is because the "find non-zero values" operation is almost instantaneous for us. Our visual system is excellent at pattern matching, and we can easily pick out the non-zero values in a field of zeros.

The same is not true of a computer program. The same algorithm translated to a program would have to scan through every element of the array, check if it's zero already, and change it to zero if it's not. Checking if every element is zero is a waste of time for the computer since it has to touch every element anyway. It might as well simply write a zero to every memory location, and since the matrix is probably contiguous in memory, it could do this operation even faster by writing larger blocks of zeros at once. It's the same task, but the brain and the microprocessor each lend themselves to different optimal solutions.

These differences come up all the time in programming. It's common when processing a lot of values in a loop to only do a certain calculation on some of the values. If the combination of the filtering of values and the calculation on the values that pass the filter can be converted into one calculation on all of the values, the latter option is often faster than the former operation. Not only does it optimize for the type of calculations that are most efficient for a microprocessor, but it also removes a code branch inside of a loop. The resulting loop makes more efficient use of the hardware because more instructions can be in flight in the processor pipeline at once and there will be less data-dependent branch mis-predictions.

Know Your Tech Stack

All this talk about processor pipelines and branch prediction leads into another point. Getting good at recognizing where code performance can be improved is a skill that requires a good mental model of how computation is done at multiple levels of your technology stack. If you're programming in a web framework like Ruby on Rails on Django, you'll have a better idea of how to optimize your code if you know the underlying programming language better. Going further, knowing more about how compilers and interpreters work will enable you to see more optimizations.

You can keep drilling down to gain an even better understanding of how computation works in a computer. Assembly languages have quite different ways of doing computation than high-level languages, with multitudes of addressing modes and special instructions for controlling program flow and transforming data in interesting ways. Microprocessor architecture has all sorts of ways to speed up code as well, with deep pipelines, multiple FPUs (Floating Point Units) and ALUs (Arithmetic Logic Units), branch prediction, instruction trace caches, and cache-memory hierarchies. Computer Organization and Design, and Computer Architecture: A Quantitative Approach are good books to start exploring microprocessor architecture.

Although knowledge is power, knowledge can also be dangerous, so knowledge of the tech stack should be used wisely. Once you have a better idea of how computer computation works, it's easy to fall into the trap of trying to micro-optimize every line of code. I'm not sure why that's the case. Maybe after learning about all of those cool optimizations it's hard to resist attempting to do them by hand, but if learning about compilers and hardware architectures teaches you anything, it should teach you that tons of optimizations are being done to your code automatically.

Instead of spending your time trying to micro-optimize stuff that the compiler and hardware can do better, you should be focusing on algorithm complexity and domain knowledge, areas where those lower-level tools can't possibly help you. Knowing about the lower levels of abstraction can help you make better higher-level decisions, but you shouldn't use that knowledge as an excuse to attempt to duplicate what your tools are already doing for you.

Don't Fight The Compiler

I've inherited code bases from other embedded software engineers who did not trust GCC to compile their code correctly with optimizations turned on. My impression is that it is a common belief in the embedded software world that the only safe way to compile C or C++ code is to do it with optimizations turned all the way off. While it may be true that a particular code base works with optimizations turned off, and it crashes with optimizations turned on, that doesn't mean GCC optimizations aren't safe. GCC is a pretty well-proven tool, and if I had to put money on it, I would bet that GCC is correct and there's a runtime problem in the code rather than the other way around.

If code doesn't work or it's unstable with optimizations turned on, it's most likely because of a race condition or memory leak in the code, and compiling without optimizations is hiding the problem. Trust me, you don't want to run production code with optimizations turned off. It's like buying a Ferrari and never driving it over 20mph. You are sacrificing a tremendous amount of performance for the illusion of safety.

Consider a few of the things that the first level of GCC optimizations gives you: faster subroutine calls, branch prediction from probabilities, and register allocation. Register allocation alone is huge. Without it, the assembly code that's generated will read every variable in your code from memory, perform the operation in the expression being evaluated, and store the result back to memory in the variable specified on the left-hand side of the expression. Even temporary variables that are only used once to calculate intermediate results are assigned a memory location and values are loaded and stored to those locations.

Register allocation uses the processor's fastest memory to store intermediate values, and operations can be chained together using these registers as operands. Register allocation also enables the processor to do register renaming and use a much larger internal set of registers not available to the assembly language. Without register allocation, not enough registers get used to take advantage of register renaming, and the processor pipeline utilization is so restricted that not many instructions can be scheduled to execute at once. Why waste all of that power by turning off optimizations?

Other optimization levels add even more performance gains by doing a deeper analysis of the code to do things like reorder instructions, unroll loops, and inline functions. These are riskier optimizations for code that's not robust against race conditions, but the real problem that should be fixed in this case is the code. Higher optimization levels also are more likely to increase code size. The first level will almost certainly reduce code size as well as increase performance because it's primarily removing instructions and condensing code to make it faster. The higher levels start doing things that increase code size, and code size may be a real issue in some applications. In any case, if you're optimizing performance, you should definitely turn compiler optimizations on to some appropriate level. Otherwise, you're ignoring a huge performance gain for no good reason.

What about doing some of these optimizations by hand where they're needed most? Well, compilers and VMs (virtual machines used for interpreted languages and newer compiled languages like Java and C#) can do all kinds of intricate optimizations that would take us humans way too long to do in every place that they would help, and get it right every time. The compiler or VM writer has to get it right once—granted, for the more complex general case—and then the compiler or VM can use that optimization every place that it detects it to be applicable.
For example, Steve Yegge describes a great optimization that was figured out way back in 1994 in his Dynamic Optimizations Strike Back post. It's called type-feedback optimization, and what it does is during run-time the VM watches the type of the dynamic method calls. When a certain type is used in a method call in a loop, the VM assumes that type will be used the next time through the loop and inlines the method. If it was wrong, a guard instruction will catch the misprediction and recover, but if it was right, the program gets a nice speed-up.

That type-feedback optimization is one example. Compilers and VMs implement hundreds, if not thousands of optimizations, and they keep getting better and better all the time. Even if we could do these optimizations slightly better by hand, why would we try? The compiler can do so many more optimizations than we could even remember and do them on much larger code bases because it's automated. Trying to compete with that kind of scale is rather silly. We should be looking for higher-level optimizations that organize the overall structure of the code, and use the best algorithm for the task that we're trying to accomplish in the code. Then we're working with the compiler, leveraging the strengths of both us as programmers and the compiler as an automated optimization system, instead of trying to compete on a level that ends up being a waste of time.

Experiment With Reality

I've come this far without even mentioning that you should profile your code before optimizing. Profiling is hugely important because if you don't measure where your code is wasting time, you're much more likely to make the wrong optimizations based solely on intuition. The places where a program is slow can catch you by surprise, and the optimizations that fix it can be equally surprising (sometimes surprisingly easy).

At this point it's pretty much common knowledge that you have to measure before you optimize, but you still need to be careful in how you measure. Using microbenchmarks to measure the performance of small snippets of code before and after optimizations can be incredibly misleading. It's much better to measure the performance of code in a real application with a real environment and a real workload. Reality can be drastically different than microbenchmarks, and things like disk and network accesses, process interactions, and realistic data sets can have a huge impact on a real program while a microbenchmark may not even run into those issues.

Caching can also have profound effects on the run-time performance of code, and microbenchmarks may hide the fact that certain optimizations perform well in isolation but trash the cache for the rest of a larger program. Organizing data structures to take advantage of caching can drastically speed up a program, but be careful because different processors have different cache characteristics. You could be optimizing for a particular, narrow cache architecture. If a program will run on different cache architectures, profiling should be done on as many of those architectures as possible to make sure the code isn't being optimized into a corner.

Optimizations can make code brittle. Heavily optimized code becomes completely dependent on the context under which it was optimized, and if that context changes—because the processor or the compiler or the operating system changed—the optimizations can actually make code slower than if the optimizations weren't done at all. It's best to first make code clear, clean, and readable. Most of the time if the code is well-designed, it doesn't need to be optimized further. Making it clean is good enough. Most poorly performing code I've come across is also messy, verbose, ugly code. Cleaning it up has the dual benefits of making it easier to understand and improving its performance. Only optimize when performance becomes a problem, measure where it's a problem, and focus on improving the code that is actually a problem.

When you do optimize code, remember that your brain does computation much differently than a computer does so your intuition is probably wrong, your knowledge and understanding of the technology stack can greatly help you find optimizations, your compiler is your friend so you should work with it, and your profiler will give you the best answers if you measure reality. The best optimizations may not be obvious, but with an open mind, you can find them.