Choosing to Write

Every week I sit down and write an article for this blog, and every week it's challenging. Sometimes it's also scary because I'm not sure I'll write well enough or that the subject will be interesting enough. Sometimes it's also fun because I'm writing about something especially fascinating to me. But it's always challenging to put my thoughts down in writing, organized in a way that will be clear and understandable.

For the past two years I've committed to writing something every week, so I have to figure out what I can write about in a reasonable amount of time while still saying something meaningful. This is a hard task to do consistently, and trying to do it myself has given me great respect for the writers that do it for years and do it well. I try to spend about 4-6 hours a week on writing, enough to get into a topic, but not so much that I don't have time for anything else. I don't worry about length too much as long as I stay on topic without rambling.

Time is the main constraint because I have a lot of other things I want to do with my time. I don't want to spend it all hashing and rehashing articles, but I do want to make sure I'm delivering something every week. It would be too easy to sit on my articles week after week and edit them to death. I've pretty much posted every article I've written before I thought it was 100% perfect. I know I can always make more edits, say things differently, or expound on details. I'll never get an article to that perfect state, so I need to put a stake in the ground and ship it. Then I can start thinking about the next article. Each article is what it is, and I'll get better over time.

Because of the time constraint, I've learned that it's easiest to write about things that I'm already thinking about or working on. If I had to do a ton of research or other work before writing about something, it would be difficult to write with any competence and still get an article out once a week with time left over. That may seem obvious, but the pull to write about things that are slightly beyond my realm of expertise is strong.

Sometimes I'll let that desire get the better of me when I feel the topic is especially important or interesting. For example, I don't study economics professionally, but I do read about it a lot and the current trends in inequality are a pretty big deal so it was worth the extra time it took to write. I've also spread more outside the world of software development recently with excursions into Mathematics and Physics. These subjects are fascinating, and I learn more about them the best way I know how, by studying books, analysing problems, and now writing about what I've found and what my impressions are.

I've learned quite a lot through writing, and one thing I've learned that I didn't expect was that the articles that I enjoyed writing the most and think are the best writing I've done are not at all the most viewed posts. I could be totally off-base, and the posts I think are my best (like this one or this one) actually aren't. But I think something else is at play. I post on Hacker News and sometimes on Reddit.com, and nearly all of my most popular posts are ones that caught the wave on these news sites.

I find this interesting because from watching the front pages of these news sites, you may think that the best articles on any given day generally filter to the top. There is definitely filtering going on, and better articles head to the front page while worse articles drop off fairly quickly. But from the perspective of an individual blogger, the fact that some articles grab interest and others don't appears mostly random with a lot of luck involved. If the right set of people doesn't see a post at the right time, it will drop right off the new page without getting noticed. That same post might make it to the front page if the first people that saw it were different. That's probably the best behavior that can be asked for with these sites, where the front page holds articles of generally good quality, but not necessarily every good post gets voted up. Not that I'm whining about my posts not getting noticed—it doesn't really matter—I just find the process interesting and illuminating.

Another thing I find intriguing about my most popular posts is that one of them is a honeypot. It was a fluff post pointing to the other blogs I've read that I thought were exceptionally good. This one post attracts nearly all of the comment spam I get, and it's always in the form of "Love your blog. <link to random, generic IT company>" These comments are filtered out automatically, so I'm the only one that ever sees them. I didn't even have to set Blogger up that way, Google has it that way by default. Do the spammers realize this filtering is happening? Is the spam meant for me? Strange. I'm not even sure how many real readers look at this post and how much of it's popularity rank is due to the spammers.

One last thing I've learned is that focus is important. Yes, that's something I should have known already, but I learn it anew all the time. It's easy to get distracted by shiny things and go off on tangents. I even allow for that inevitability in the tag line of this blog. Every time I go off exploring some new thing, I find it difficult to keep making progress on the other things I want to get done.

I do think there's value in exploring and learning about stuff that's tangential to my main interests, but I need to temper those excursions so they don't take over all of my free time. Keep your eye on the prize, so to speak. That's why I find it funny looking back at my list of things I want to master in my lifetime. Eleven things is too much, and with hindsight I can easily eliminate four of them as simply being hobbies or things I won't really pursue in the future. That leaves seven things I want to master in my lifetime (to my personal definition of mastery), two of which I already feel that I've accomplished. That's much more manageable and might even leave room for new future interests.

It's useful to have exploratory periods where you're expanding your horizons to see what's out there, and then have consolidation periods where you concentrate on one thing for a while. It's a natural ebb and flow of learning that allows you to be aware of more subjects outside of what you normally do while still learning something deeply.

Next year I'm due for a consolidation period, so I want to narrow the topics I write about as well as the topics I learn about. This past year I let my writing topics go far and wide. Next year, I'm aiming to focus on software development, at least for the first part of the year. Partly, this refocusing is because writing and learning are so intertwined. I write about what I'm learning about because it's easier to pick topics that are interesting and fresh to me, and vice versa, I need to learn about what I'm writing about for obvious reasons. Learning and writing kind of drive each other. If I add focus to that combination, I can make better progress on the things I care about.

I've found that I enjoy putting things I'm learning into my own words in an attempt to explain them. I find great value in sharing knowledge, and I hope that some of the things I've written have sparked ideas in others. I could try writing more about stuff that would appeal to a wider audience, but that doesn't seem genuine to me and would probably show. I could write about stuff that I've known about for a long time, but I wouldn't be as interested in writing about stuff that's no longer challenging to me and that would show as well. I find that the best topics to write about are those that I'm dealing with and learning and doing right now. I sincerely thank you, my readers for taking the time to look at my articles and share your thoughts here and there. I can't wait to see what another year of blogging holds. Happy New Year!

All in Good Fun: Learning With LEGO Video Games

Since it's the week of Christmas, I thought I'd have some fun and review a bunch of video games that I've had a great time playing over the years: the LEGO video games. I first started playing the LEGO games a few years after my daughter was born, and now both her and her younger brother have been introduced to video games through the LEGO series. They're a great introduction to games for children because they're fairly easy and accessible, yet they're engaging enough for adults that you won't be bored to tears while playing them yourself. We love playing together, and it ends up being good quality daddy-and-kids time.

In these LEGO games, collecting everything and experiencing everything that they have to offer requires a fair amount of skill and determination, so parents will have plenty of challenges to keep themselves interested. Plus there's the awesomeness of playing as hundreds of characters from all of the classic stories that they've made LEGO games out of, from Obi-Wan Kenobi in Star Wars to Gandalf in Lord of the Rings. The cut scenes in the games are parodies of the most iconic scenes from the movies they're based on, and the fact that everything is acted out with LEGO characters and props made out of bricks is just hilarious.

It's a good, clean kind of fun, though, that's safe for kids. The worst things that ever happen involve LEGO people flying apart, and even then, the characters usually get up and go chasing after their loose body parts, making the scenes even more ludicrous. The games are also packed with plenty of puzzles to challenge young minds. The levels are mazes that need to be explored and navigated, which develops spacial intelligence, and like most video games, manual dexterity and hand-eye coordination are quickly improved as kids play more and get better.

The LEGO games also encourage teamwork and cooperation because at multiple points in every level both characters are required to work together to make progress. The levels are incredibly well-designed, and I've been amazed numerous times at the intricacy of some of the puzzles that stand in your way. The games even have money management features with their standard currency of studs that you collect by building and breaking things in each level. You have to make decisions about what to buy with your growing stash of studs to best help you get through levels more easily and collect more studs more quickly.

Finally, the LEGO games help kids develop goal-setting skills. There is so much to do in every game, and many paths to take. I let the kids choose what they want to accomplish, and they have to figure out how to achieve their goals. The games display plenty of statistics about how much of the various items and characters have been collected, so the kids can see what's still available and what they should go after next. It even helps them practice some basic math concepts like fractions and percentages. All around, the LEGO games are a great learning tool, and honestly, they're a blast to play.

Okay, enough of the learning stuff; let's get to the games.

Star Wars: The Complete Saga (November 2007)

All screen shots came from gamespot.com. Click to enlarge.

This is where it all began, with Star Wars. The first LEGO Star Wars game came out in March of 2005, covering Episodes 1-3. Then a second game came out covering Episodes 4-6. The Complete Saga combines those two games, so it's the one to get. It is awesome running around as a LEGO Jedi, swinging a lightsaber and pushing droids around with the force. Blowing stuff up with the bounty hunters' thermal detonators isn't too bad, either.

Star Wars set the foundations for how all of the LEGO games look and feel. The basic idea is to go through each level looking for minikits and power bricks while collecting as many studs as you can. If you collect all of the minikits in a level, you get a complete LEGO model of something iconic from the movie—like a vehicle or a prop—and a gold brick. You also get gold bricks for collecting enough studs and completing the levels. The gold bricks then unlock bonus levels for even more LEGO fun. The power bricks unlock power-ups like quick building, invincibility, and stud multipliers. All of these collectibles are hidden in puzzles throughout the levels, and merely completing the levels requires a fair amount of puzzle solving as well.

There's a ton of other stuff in the game, like pod-racing, bounty hunting, and level time trials. It takes about 20-30 hours to fully explore the game. Even though the graphics are showing their age, this game is still great fun to play. Come on, it's Star Wars, with LEGOs!

Indiana Jones: The Original Adventures (June 2008)

LEGO must have figured they had a good thing going by the time they released Indiana Jones. They decided to take the whole slap-stick, spoofed cut-scenes and run with it this time. Having all of the classic Indy scenes acted out with goofy LEGO characters that can't talk (they only grunt, hum, and mumble) is a riot.

The format is basically the same in this game as it was in Star Wars. Collect minikits, power bricks (now called parcels), and studs. The minikits build artefacts that unlock bonus levels, and the parcels unlock power-ups, like before. This time around there are no gold bricks, but they return in later games.

The levels in this game are great because of the huge diversity of characters. Indy has his whip that he can use to cross chasms and activate switches. There are "shorty" characters that can go through special tunnels. Other characters have wrenches, books, and shovels that are necessary for certain puzzles. And women characters can jump about twice as high as other characters to reach higher platforms. I would say there's more variety of characters than there was in Star Wars, and it really adds to the puzzle design of the levels. And you get to be Indiana Jones, which is awesome. Unless my daughter is playing. Then you have to be Satipo.

Batman: The Videogame (September 2008)

Batman is awesome, so of course they had to make a LEGO video game based on him. Robin is pretty cool, too, at least according to my son. When he's playing, I get to be Batman. In the cut-scenes Batman has essentially one grunt, and he uses it all the time, to full effect. LEGO took his demeanor from the latest films (you know, the Christian Bale Batman) and made it so extreme that it's hilarious. The contrast between him and the rest of the caste is perfectly done, and gives the game great character.

Once again there are no gold bricks in this game, but the power bricks make a return. The minikits and studs are still here, and power-ups and bonus levels are unlocked in the same way as Indiana Jones. The caste of characters is slightly smaller, but that's because Batman and Robin have a number of different suits they can wear. Each suit has its own special abilities, like gliding, setting mines, or my kids' absolute favorite, sucking up LEGO bricks with a vacuum suit.

Once you complete the main levels as Batman and Robin, you can go through them from a different perspective as the villains. As the villains, you progress through the same levels as you did with the heroes, but from the villains' point-of-view so everything is modified for their special abilities. It's great fun, and being Mr. Freeze is about as awesome as being Batman (or Robin, if you're four).

Indiana Jones 2: The Adventure Continues (November 2009)

It wasn't awesome enough the first time around, so Traveller's Tales (the game developers) decided to give Indiana Jones another go. In the previous games, the hub between levels was one of the main hang-outs from the story the game was based on. Star Wars had Mos Eisley Cantina, Indiana Jones had Barnett College, and Batman had The Batcave. These hubs had a few secrets in them, but the main part of the game was the levels.

In Indiana Jones 2, it's much different. Each story has it's own hub, and those hubs are packed with secrets, minigames, and bonus levels. Power bricks are also found in the hubs instead of the levels now, and once ten bricks of one color are collected, a parcel will drop in by parachute with a power-up that you can purchase. Characters also walk around the hubs and you can purchase those as well. Some of them, mostly villains, you have to fight first before you can purchase them.

The gold bricks also make their return, and they appear in the hub as you make progress through the levels. Once you get them all, you can build a major LEGO structure and unlock a super bonus level with plenty of fun stuff to build and destroy inside.

The interface has also improved with the addition of a split screen in two-player mode. It's not a normal split screen that's divided vertically or horizontally. Instead, it constantly adjusts depending on where the players are in relation to each other so that the split is perpendicular to the line between them. To make your way towards the other character, you head directly towards the middle of the split line, and when you get close enough, the split disappears. This is really, really awesome. Now you're not bound to a box on the screen when playing with someone else. You can explore where ever you want, which is especially helpful when playing with kids. You will still need to work together to solve many of the puzzles, but now you have so much more freedom.

Harry Potter: Years 1-4 (June 2010)

My kids love Harry Potter. So does my wife. Okay, I'll admit it, so do I. Being a wizard is awesome. They made some significant changes for this game, and I would say mostly for the better. The wizarding world has a lot of magic spells, so there's a spell wheel added to the interface where you can select which spell you want to caste. As you advance through the years, you learn more and more spells, and some wizards have special abilities. Harry has his invisibility cloak, Ron and Hermione have pets that help with certain puzzles, and dark wizards can blow up dark LEGO bricks.

Continuing the trend of Indiana Jones 2, the main hub of this game is Hogwarts Castle. You find the entrances to the levels as you explore the castle, and more of the castle opens up as you advance in the game. If you get lost (the castle is huge), Nearly-Headless Nick will be there to guide you to the next level. Don't rush through the castle, though because it holds plenty of secrets waiting to be discovered.

You do all of your shopping for spells, power-ups, and characters in Diagon Alley (where else?), and power-ups are now unlocked by finding parcels that you hand off to Hedwig to carry to the power-up shop. You get gold bricks for collecting studs, finding house crests, finishing levels, and saving students in peril. Once you collect enough, you unlock bonus levels for each year and a super bonus level once you've collected all 200 gold bricks. We're still working on it.

The graphics had gotten an upgrade for Indiana Jones 2, and they definitely got another one for Harry Potter. Lighting and water effects are much more realistic, and anything that's supposed to be LEGO pieces has a much more plasticy look. Animations are more fluid, too.

This is a great game—one of the kids' favorites—but I'll warn you. Try to play with one kid at a time until you unlock some extra characters because eeeveryone wants to be Harry Potter. Go figure.

Star Wars III: The Clone Wars (March 2011)

I don't actually have this game, yet. I, unfortunately, was under the impression that it wouldn't be much different than Star Wars: The Complete Saga, but after seeing how much better Indiana Jones 2 and Batman 2 are than the originals, I'm now sure I want to play Star Wars III. More saving the galaxy as a Jedi Knight, what could be more awesome than that?

Pirates of the Caribbean: The Video Game (May 2011)

Arguably, being a pirate is more awesome. This is probably my favorite of the LEGO games that we've played so far. Jack Sparrow as a LEGO figure just kills me. He's got the same flippity swagger from the movie taken to a hilarious extreme, and pretty much all of the characters are oozing with personality in this game.

The game play goes back to the LEGO roots somewhat, with an old-school sized hub for level selection, and you collect minikits and studs in the levels for gold bricks that unlock bonus levels and power-ups in the hub. Power bricks are nowhere to be found, though. Instead, they're power hats that you find in the hub, because, you know, pirates love hats, right?

The diversity of characters is incredible in this game. You've got Jack with his compass for finding hidden items in the levels, cursed pirates that walk under water, Davey Jones and his crew that can transport to different places at special barnacle spots, and Black Beard with his sword that can control black and red bricks. Then you have the normal assortment of shovel, blacksmith hammer, high-jump, shorty, and bomb-throwing characters. Oh, and there's a guitar player, mermaids, and a singer that can break glass. Then there are characters that combine different abilities, and I'm sure I'm forgetting some as well.

The puzzles and level design are fantastic. The cut-scenes are great. This is all around a seriously fun game. And when you think about it, who doesn't want to be a pirate?

Harry Potter: Years 5-7 (November 2011)

Since we haven't finished the first four years, I haven't gotten to this game yet. But I expect it to be at least as awesome as the first one. Just look at that screen shot—Harry, Ron, and Hermione flying a dragon through Gringots. It's gonna be good.

Batman 2: DC Super Heroes (June 2012)

I was floored when I saw the first cut-scene in Batman 2. The characters actually talk! No longer do the characters mumble and grunt (well okay, Batman still does). Now there's actual dialog, and it's well done. The gameplay is much different as well.

After the short reprieve of getting back to the basics with Pirates of the Caribbean, Batman 2 picks up the large central hub of the Harry Potter games and takes it to a whole new level with Gotham City. The game plays almost like GTA, with a large open world to explore. In it you find red power bricks, gold bricks, citizens in peril, extra villains, and the entrances to missions.

The gold bricks unlock extra characters this time, and there are lots of different super heroes and super villains to choose from. But you have to find them. Scattered around the city are various arches that can be built with gold bricks to reveal purchasable characters, and they are mostly hidden high up on the roof tops of buildings. The whole game is great for exploration and learning how to navigate in a complex environment. There's an overhead map that helps a little, but mostly the treasure hunting tasks depend on you figuring out how to get from where you are to some remote location with the multitude of tools you have available. It's a great learning experience.

The different suits are back, and better than ever. The vacuum suit is still popular with the kids, and this time it sucks up liquids instead of bricks. When the tank is full, you can shoot it at villains as a weapon or use it to solve puzzles or put out fires. The kids also love Robin's acrobat suit. It's great fun rolling around in the gerbil ball that comes with it, and since these are Robin's suits, that means I get to be Batman. Everyone's happy.

Lord of the Rings (November 2012)

I've only played the first level of Lord of the Rings so far, but it looks awesome. The graphics definitely got another upgrade, and the initial battle scene with Sauron had a ton of stuff going on at once. The action was a bit too intense for my daughter, so we may have to wait a half a year or so before really getting into this game. I'm looking forward to playing as all of these great characters. It's going to be awesome.

Marvel Super Heroes (October 2013)

They're pumping out LEGO games so fast now that I haven't been able to keep up. On top of that, Marvel Super Heroes is the first LEGO game available on the PlayStation 4, and I don't have one, yet. I can't wait to play as Iron Man and Magneto and The Hulk. Who am I kidding? I want to be all of them.

The LEGO Movie Videogame (February 2014)

If this game is anything like the movie, then it's going to be awesome. Look at that screen shot—Abe Lincoln in a flying rocket chair. Everything is awesome. That about sums it up.

The Hobbit (April 2014)

It looks like The Hobbit only covers the first two movies, and there should be more content for the third movie, which just came out a week ago. I would bet there will be either another game, or some downloadable content to add to this one to complete the story. It looks awesome, and I can't wait to play it.

Batman 3: Beyond Gotham (November 2014)

Hey, another Batman game. He's awesome, so why not? I hear this game is less about Batman and Robin and more about the entirety of the Justice League. They were heading in that direction with the last Batman game, so this is the logical continuation. I'm sure it will be…awesome!

What's Next?

Who knows! I've heard rumors about Ghostbusters, but if the release rate of the last few games is any indication, we won't have to wait long to find out. They've got their formula down, and every game they make tweaks and enhancements to keep making the games better. So far I'd have to say my favorites are Pirates of the Caribbean, Batman 2: DC Super Heroes, and Harry Potter, in that order. But all of these games are great, and they're a great learning experience for the kids, too. They encourage exploration, goal-setting, and decision-making. They help develop critical thinking skills and spacial intelligence with all of the great puzzles and level design in every game. And it is so much fun to play as all of these iconic characters in a world of LEGOs, as long as I can be Batman and the kids can be Harry Potter. Happy Holidays!

Inequality

As a follow-on to my review of two economics books on inequality, I'd like to spend some time analysing the problem in more depth. It is an incredibly complex problem, and it's very easy to forget the big picture and get lost in minute details. As a programmer, I deal with complex problems all the time, trying to find the best solution in a field of non-optimal trade-offs. Sometimes it feels like trying to find the least-worst solution, and that seems to be the case with inequality as well. Every potential solution has its own set of issues, and even defining the problem is difficult and open to endless debate. Regardless of the difficulties involved, let's see what the problem space looks like.

Defining Inequality

Inequality means many different things to different people. In this context we'll deal primarily with financial or economic inequality, so that narrows the idea somewhat—but not much. Bill Gardner over at The Incidental Economist has a great deconstruction of the different ways of thinking about equality, so we'll use his framework of answering three questions to define equality.

Equality among whom? Throughout America's history the idea of equality has expanded to encompass more and more people. Equality first applied only to white male landowners. With much blood, sweat, and tears it has grown to include women, all races, and the poor and middle class, at least on paper. (Racial inequality is still a huge problem in the U.S., and it's closely intertwined with economic inequality.) Immigrants are still excluded in many ways, and global equality is not in the forefront of most people's minds.


It's reasonable to expect that most people would think first about their own country. Not that it's right in a moral sense, but it's hard to believe that most people will find the energy or good will to care about the world's poverty while inequality is growing so much in the U.S., with more people struggling to make ends meet. In some ways we need to put our own house in order before we can expect our citizens to look outward. However, we should still pursue international aide whenever we can, if for no other reason than it is also in our own best interests.

Equality of what? Should we strive for equality of outcomes, opportunity, or well-being? There may be other dimensions, but these are the big three. Trying to create equality of well-being has the problem of who decides what constitutes well-being. Different people have different ideas about how they want to live, and who is to say what's right? There is also the problem of free-loaders. Not everyone wants to work, so how do we encourage everyone to be productive members of society?

Equality of outcomes and opportunity are hard to separate because the economic outcomes of parents heavily influences the economic opportunities available to their children. We may be able to come up with programs to improve opportunities for children—subsidized or even free post-secondary education, better public secondary and elementary education, school lunch programs, free preschool and day care, prenatal care, etc.—but it's hard to know where to start and where to stop. In reality, outcomes and opportunity cannot be separated, and programs to improve one will necessarily impact the other.

Why do we want equality? Is it equality for its own sake, is it to give everyone the wherewithal to survive, or is it because everyone deserves a basic level of respect as a human being? Mark Thoma gives an excellent reason why we should all want equality:

...people who, because of their incomes, cannot participate fully in society are poor. A child getting enough to eat, and with clothes to wear, who cannot afford the toys needed to be part of the group of kids in the neighborhood is socially isolated and socially disadvantaged (we don't want to play at your house because you don't have a TV, you can't come with us because you don't have a bike, you didn't get my text message about baseball practice being moved?, etc., etc., etc.). Giving people, children in particular, what they need to participate in the society around them is an important element of how successful they will be in the future. It helps to determine their ability to give back to society as fully participating adults. ...

Having more successful, participating adults would benefit all of us. In addition to requiring less support from everyone else, they could contribute more to the productivity and advancement of our society.

In the end, we could want equality because of our own self-interest. Having a more equal society increases everyone's safety and improves everyone's opportunity. We would all have a better chance of being a successful entrepreneur or productive employee, not just because everyone would be better educated and healthier, but because more people would have the means to afford more products and services. One of the reasons Henry Ford paid his assembly line workers good wages was so that they could afford to buy his cars. We should remember that powerful idea.

What causes inequality?

The basic opposing views on what causes inequality are that it is because of structural issues or individual decisions. Structural causes would be things like technology displacing workers, globalization moving jobs overseas, differences between regions of the country, or variations in the quality of education. I have a hard time believing that technology is having any more of an effect on inequality than it ever has in the past. The technologies being created today may be new, but the fear that technology will leave a vast swath of the population unemployed has followed every major technological advancement. In every case we found that there was more work to fill the gaps, and at the same time, we were able to enjoy more leisure. This time is probably no different. We just don't know what future work will look like, yet.

As for the other structural causes, globalization does effect some jobs, like manufacturing, but not others, like retail and most services. Over time the living standards of foreign workers will increase to the point where their wages and the increased costs of shipping, logistics, and management will increase the incentives for domestic manufacturing. On the other hand, regional and educational inequality pose real, persistent issues that should be addressed.

Individual causes of inequality are generally thought to result from poor decision making, laziness, or some combination of the two. This kind of reasoning smacks of a blame-the-victim mentality, but it is a fairly commonly held view, especially by certain out-spoken politicians.

Chris Dillow has an interesting analysis of structural vs. individual poverty, and he discusses why it is hard to separate them, much like equality of outcomes vs. opportunity. His comments stem from a debate between Matt Bruenig and Noah Smith on the same topic, and Smith has a great post with a number of thought experiments dealing with structural and individual inequality. In particular, he theorizes about what it would mean if everyone stopped working:

Suppose everyone, acting independently and on their own, just stopped doing any sort of work. The entire world would instantly fall into poverty. But would that poverty be "structural" or "individual"? Note that if you're the one person who wants to work in a society of people who do no work at all, you're not going to be very productive, because you can't specialize. So in the case where no one works, no individual's behavior can hoist him or her out of poverty. So that's "structural", right? But if you simultaneously persuaded a large fraction of people to work, then large numbers of people would climb out of poverty. So is the poverty "individual"?

He makes a strong case for both structural and individual causes of poverty playing a role simultaneously, and he also brings up another major cause—randomness. It is certainly the case that some people are poor for no other reason than a run of bad luck. It's not their fault. It's not society's fault. Sometimes bad things happen, and that's a completely legitimate factor that needs to be taken into account.

Another way to think about Smith's 100% unemployment thought experiment has to do with the economic theory of the marginal product of labor (MPL). MPL is the incremental output that one additional employee produces, and the theory suggests that each employee should be paid according to their marginal product. This idea is usually used to justify paying CEOs of large corporations exorbitant salaries and reducing labor costs for unskilled workers as much as possible. But suppose that "everyone" in Smith's thought experiment is everyone at a particular company, and the one person that tries to work is the CEO. In that case the CEO isn't going to be very productive on his (or her) own, and in fact his productivity in the company is partly (mostly?) a result of the rest of the company's employees. If that is true, then why is the pay difference between the CEO and the lowest paid employee so extreme? This argument may be good justification for more balanced wage structures in large corporations and a reduction in rent-seeking behaviour on the part of the CEO.

Rent-seeking is a major source of inequality in many ways. Companies building monopolies or oligopolies is a form of rent-seeking. Patent war chests and litigation are a form of rent-seeking. Aggressive lending using sub-prime mortgages, excessive student loans, or high-rate credit cards are all forms of rent-seeking. In every case, the person or company benefiting from the arrangement is making more profit than they could have in a purely free market, and so they are getting more than their marginal product as compensation. The people on the wrong end of these deals end up poorer as a result, and inequality grows.

How can we reduce inequality?

First, it should be noted that reducing inequality doesn't mean creating a society with perfect equality. We already know such a scheme can't work because there are no incentives to work hard, to innovate, or to take risks—all necessary components of a working economy. We are no where near that extreme on the equality/inequality scale, though. We can certainly make substantial progress towards more economic equality without endangering America's entrepreneurial spirit. Reducing inequality will most likely improve incentives, giving more people the opportunity to take a risk and pursue their dreams.

How do we reduce inequality? One of the first ideas that normally comes to mind is to make changes to the tax system. This discussion on Hacker News, for example, spends a lot of time debating the pros and cons of different taxation ideas. Things like the EITC and the minimum wage currently do a lot to keep families above the poverty line, but there is a problem at the income threshold where these benefits disappear. There is a disincentive to make more than the threshold because the lack of benefits is equivalent to a very high marginal tax rate, and households that are barely over the threshold may actually bring home less income than they did before.

Other ideas like a negative income tax or a basic income for everyone are appealing because they do not have a threshold where a high marginal rate kicks in. These policies improve incentives to work harder and to move up the income scale. The disadvantage is that we would need to decide how to pay for such a policy, especially for a basic income, and people that don't need it are generally opposed to paying for it.

On the other end of the income scale, there is the question of out-sized tax benefits to the wealthy. The share of entitlement benefits is fairly well distributed between the bottom, middle, and top income households, with about 10 percentage points more going to the bottom and 10 percentage points less going to the top, as shown in the following figure:

On the other hand, the share of tax expenditures is grossly skewed towards the top income households, creating a massive tax benefit for the group of people that probably need it the least:

Most of the difference in the share of tax expenditures comes from the mortgage interest, state, local, property, real estate, and charitable contribution tax deductions and the fact that capital gains and dividends are taxed at a lower rate than earned income. Hedge fund managers especially benefit from the lower capital gains rate because they can claim most of their income as capital gains through the carried interest loophole. Finding ways to equalize the distribution of tax expenditures would go a long way towards reducing inequality.

Other effective solutions tend to fall under one of two categories. They either improve security or opportunity. Things that improve people's economic security include better health care and health insurance so that we have a healthier work force and getting sick isn't catastrophic, unemployment insurance or work sharing programs to protect workers in recessions, and retirement insurance so people are not destitute in their old age. Basically, any type of insurance that is widely available and spreads life's risks over the whole of society will help people deal with the rough spots in life. We have some form of each of these types of insurance today, but there is certainly room for improvement.

The opportunity category includes a wide range of things that give people the skills and resources to succeed, if they have the motivation and put in the effort. Quality, affordable education for everyone from elementary school through a college degree or vocational program would greatly improve people's opportunity to succeed. Getting money and special interests out of politics, and setting term limits on members of Congress would tilt the balance of power away from the extremely wealthy and back towards everyone else so government programs and policies would benefit citizens more broadly instead of serving narrow interests and private corporations. Full employment would increase workers' bargaining power and improve competition in the labor force so wages would become more fair and at least some forms of rent-seeking would be more difficult.

These solutions are certainly broad and general, and the details of implementing programs and policy changes that would bring about the reduction of inequality get complicated very quickly. The intent here is to show what the landscape looks like and what paths are likely to get us to a more equal society. There is no one outstanding cause of inequality. It results from a combination of structural problems, individual decisions, and random chance. Likewise, the solutions to the problem of inequality are widely varied. We need to address the distribution of income and taxes, the security of individuals when crisis strikes, and the opportunities available to everyone to achieve a better life. We are at a point in our history where inequality is becoming so extreme that it is damaging our ability to function and dragging all of us down. Turning that trend around and improving equality would go a long way to restoring trust in our institutions and each other.

Econ Book Face Off: The Price of Inequality Vs. House of Debt

In the spirit of my recent diversions from Tech Book Face Offs, I'm going to venture into the domain of economics with an Econ Book Face Off. I don't want to get too far into the field of economics and its ugly cousin, politics, on this blog, but I do think that inequality is one of, if not the major issue of our time, and it's worth spending a post or two talking about it.

We should all be taking a critical look at the state of our economy (this applies to many countries, not only the United States, even thought that is the country I'm focusing on) and how we want it to evolve over the next few decades. Being that our country is a democracy, I truly believe every citizen does have a voice, and together we can make a difference if we raise our voice strongly enough, in spite of all of the corporate money and special interests that are clogging the halls of power. To enact the change that will be most beneficial to all citizens, we need to know what the true issues are, and how to solve them effectively.

Like any software project, reducing inequality requires sharp analysis of context and trade-offs. The evidence is quite clear that inequality has been increasing for decades, and we have reached a point where the top 1% of income earners are bringing in as large of a percentage of total income as they were on the eve of the Great Depression.

Fred the Oyster at commons.wikipedia.org [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

We've already been through a Great Recession that has caused great hardship for so many people, and the weak recovery has left the bottom 90% of people essentially no better off than they were before the crisis while the wealthy have recovered their losses and then some. To try to better understand what has been happening to the distribution of wealth, what is causing the relentless divergence of income and wealth, and how to make progress in reversing this trend, I did what I always do. I picked out a couple of books to read.

VS.

The Price of Inequality

Joseph Stiglitz is an excellent writer on economic issues, right up there with the likes of Paul Krugman and Robert Reich in his clear, understandable way of writing about economics so that the average person can easily get everything he's talking about. I thoroughly enjoyed this book because it hits on so many pertinent aspects of the inequality issue we're dealing with in a way that makes sense. In doing so, Stiglitz reveals how complicated inequality is and how there's no silver bullet for fixing it. This book is packed with great analysis and reasoned ideas for making our society more equal and fair without damaging the qualities that define us as a nation of innovators and entrepreneurs.

I can't really do the book justice in one short review, so I'll attempt to hit the most important points that made it resonate so strongly with me. Know that there is a ton more great material than what's represented here. Stiglitz starts off, as all economics books do, by defining the problem we're dealing with, and he puts it in the context of what a desirable economic system should be:

Some individuals will work harder and longer than others, and any well-functioning economic system has to reward them for these efforts. But this book shows that both the magnitude of America's inequality today and the way it is generated actually undermine growth and impair efficiency. Part of the reason for this is that much of America's inequality is the result of market distortions, with incentives directed not at creating new wealth but at taking it from others.

These ideas really set the tone for the book. Fighting inequality is not about trying to redistribute income arbitrarily or waging class warfare. It's about finding ways to eliminate the distortions that are already present in our economy and to make the system more efficient so that everyone who contributes can get their fair share of our national product and technological advancements. If we can do this, then everyone can share in the growth, and better outcomes can be had by all. However, in order to achieve this goal, the wealthy and empowered will have to let go, somewhat. Stiglitz is careful but forthright as he makes his case:

[T]he rich do not exist in a vacuum. They need a functioning society around them to sustain their position and to produce income from their assets. The rich resist taxes, but taxes allow society to make investments that sustain the country's growth. When little money is invested in education, for lack of tax revenues, schools do not produce the bright graduates that companies need to prosper. Taken to its extreme—and this is where we are now—this trend distorts a country and its economy as much as the quick and easy revenues of the extractive industry distort oil- or mineral-rich countries.

In many ways it is in the best interests of the rich to help everyone and make these investments in education to improve the workforce instead of exploiting it for short-term gain. The importance of improving the labor environment in reducing inequality cannot be understated. If there were better support and protections for labor, it would lead to more job loyalty, less churn in employment and the corresponding training costs, and higher productivity, which would create a positive cycle that is beneficial to both the employers and employees. Stiglitz raises the issue of fair treatment of workers numerous times in the book, and he also focuses on the closely related issue of trust:

I have emphasized how the country has to act together, cooperatively, if the country's problems are to be solved. Government is the formal institution through which we act together, collectively, to solve the nation's problems. Inevitably, individuals will differ in their views of what should be done. That's one of the reasons that collective action is so difficult. There needs to be compromise, and compromise has to be based on trust: one group gives in today, in the understanding that another does in another year. There must be trust that all will be treated fairly, and if matters turn out differently from how the proponents of a measure claim it will, there will be change to accommodate the unexpected circumstances.

Here he is talking about trust of the government and democracy, which has been damaged by the catering to special interests and other corrupt practices that permeate our politics. The loss of trust can also be seen in the workforce, reducing worker's loyalty and morale, and it is clearly an issue with how the public now views the financial sector. Banks especially have lost trust by actively destroying it. If any other industry designed products that take advantage of the consumer the way that the banks did, I would imagine those companies would quickly fail, but we seem to be beholden to the banks in ways that don't allow us to let them fail.

People tend to generalize and pick a primary source of failure as either the government or the markets, but the truth is both are susceptible to corruption and failure. Both are also capable of working efficiently and fairly when set up well. The natural reaction when things don't work well, and instead look broken, is to throw up your hands in frustration and throw out the trash, but that doesn't fix anything. Something is going to fill the void left by crippled government or over-regulated corporations, and the public is going to have less control over it than what they have now. Neither the government nor private companies will ever be perfect, but we should always work to make them better instead of blindly trying to make them smaller.

Stiglitz takes a pleasantly rational and balanced approach to the problem of rising inequality, and the ways that we can work to fix it through the channels of our government and private business. He ends with a chapter collecting and summarizing all of the various ways we can attack the problem from his discussions throughout the book. The solutions range from broad economic solutions of market fairness, tax reform, education, and public investment to immediate labor market improvements and household debt relief to political reform in campaign finance, voting, and media coverage. He elegantly addresses a huge, complex economic problem with an appropriately broad, multifaceted set of solutions. I highly recommend The Price of Inequality to get a real sense of what's at stake with this issue of inequality that affects us all.

House of Debt

Whereas The Price of Inequality covered the full gamut of issues surrounding inequality, House of Debt, by Atif Mian and Amir Sufi, takes a different approach. The focus was squarely on the housing crisis of the Great Recession and how the out-of-control lending and run-up of private debt before the crisis resulted in a severe, protracted recession that left our country even more divided than before the recession began. While the Great Recession cannot possibly explain the trend in inequality that has been going on for decades, it is a major contributor to our current predicament and is indicative of the larger issue of how the rich get richer and the poor get poorer.

The authors start out by clearly explaining how the benefits and risks of mortgage debt can be very asymmetric for the lender and borrower. Under most circumstances, the lender has a very safe income stream for the money they lent, backed by an asset that in normal times is very unlikely to lose value. In contrast, the borrower must pay interest on the money they borrowed while taking on most of the risk if the asset loses value. The borrower's equity is the junior share of the investment, so the borrower will lose everything before the lender will lose anything. If the borrower doesn't walk away in the event of going underwater on the mortgage, they end up losing more than they invested while the lender doesn't lose a thing.

The housing bubble was fueled by the lending boom, and the authors did a great job of showing empirically why this was the case. The consequences of this lending are well-known:

Eager to create and sell profitable securities, lenders extended credit to so many marginal borrowers that they reached a point at which they lent to borrowers so credit-unworthy that they defaulted almost immediately after the loan originated. Once the defaults began to rise, the entire game unraveled, and levered losses kicked in.

Much of the sub-prime lending can be explained by rising inequality. As the rich got richer and were looking for more investments for their growing wealth, sub-prime lending increased to meet the demand. The authors remind us that:

We are more than willing to blame "irresponsible home owners" who stretched to buy houses. But the house-buying binge was only possible given the aggressive lending behavior by banks. 

They go through an explanation of how debt is a necessary catalyst for inflating asset price bubbles, using a thought experiment of a set of optimists who will buy assets at a higher price because they believe the assets will increase in value, and a set of pessimists who enable the optimists by lending the money needed to buy the assets at the higher price. Even though the pessimists don't believe the assets are worth the higher price, they lend the money to the optimists because they believe the loans are safe. Without the pessimists lending money, the optimists couldn't inflate the bubble.

When the bubble finally bursts, the optimists lose their equity, but the pessimists still have their loans that they feel they deserve to collect. The pessimists did take some of the risk in loaning money, though, so they should share in some of the downside as well. There is precedent for this idea of lenders sharing the downside risk, and the authors relate some policy choices that occurred in the aftermath of the Great Depression:

When the United States went off the gold standard in 1933, the dollar was worth far less in gold than it had been. As a result, creditors all wanted to be paid the original amount back in gold. But the gold clause in debt contracts was abrogated by Congress. This meant borrowers could pay back in dollars that were worth far less in real terms than what they had borrowed. …

Interestingly, the effects of the gold-clause abrogation were quite positive for both borrowers and lenders. As Kroszner demonstrates, both equity prices and bond prices rose when the Supreme Court upheld the congressional action. In other words, debt forgiveness actually made creditors better off. It is likely that we would have reached a similar outcome during the Great Recession had the government more aggressively facilitated the restructuring of household debt. [emphasis theirs]

They argue convincingly that it is not necessary that wealthy creditors get bailed out while struggling debtors are left under a mountain of debt. They also focus on the best ways to share the downside of a financial crisis equally between creditors and debtors, and why other policies, even fiscal policies, don't work quite as well:

[T]he most effective fiscal stimulus would be one in which the government taxes creditors and provides the funds to debtors. But why do we need fiscal stimulus? Mortgage cram-downs in bankruptcy, for example, would accomplish the same goal without getting taxation involved.

Fiscal policy is an attempt to replicate debt restructuring, but it is particularly problematic in the United States, where government revenue is raised from taxing income, not wealth. The creditors whom the government should tax tend to be the wealthiest people in the economy, which is why they are able to lend to borrowers.

The household debt problem has been a major driving force for rising inequality, and House of Debt is a great book that shows why it's a problem, how it works to enrich creditors at the expense of debtors, and what we can do to make the system more fair, more stable, and more efficient. It's another book on inequality that I highly recommend.

Reducing Inequality

The Price of Inequality is a great book for getting a broad view of inequality, its problems, and possible solutions. It covers most aspects of the issue, and Stiglitz is an effective and eloquent writer. House of Debt takes a more focused look at one major component of inequality, household debt, and what we can do to fix it. Mian and Sufi write clearly and thoroughly about the various arguments and concerns surrounding this sensitive topic. Both books were excellent at what they tried to accomplish.

There are certainly many other books on inequality that I could have read. In particular, why didn't I read Thomas Piketty's Capital in the Twenty-First Century? I would like to, and from what I hear, it's exceptionally well researched and bound to be a classic economic work. But honestly, I'm a bit intimidated by it, and I believe these two books, especially House of Debt, are much more approachable for the casual but concerned reader.

It's important to spread the knowledge about how serious a problem inequality is becoming. It creates a destructive cycle of unemployment and poverty leading to depression and helplessness, which leads to poor education and crime, which leads to more unemployment and poverty. It's a very hard cycle to break once it becomes rooted in a society. I think the public is waking up to these issues and starting to pay attention.

Once enough people are paying attention, we need to decide what we want the future to look like. Do we envision having a desperate underclass that struggles to survive and a disconnected wealthy class that misleads and exploits the masses, or do we want a future where everyone benefits from our technological advancements and leads healthy, happy lives with fulfilling work and enjoyable leisure? I believe the latter is entirely possible, if we decide that's what we want, and we do what it takes to get there.

Are Patents Still Useful?

Patents used to have a well-defined role in the economy and product innovation. That is no longer the case in some technological fields, like software. Now it seems like patents have been usurped and distorted by large corporations to protect their monopolies and stifle innovations that might threaten their business. The small entrepreneur that was originally supposed to be protected is now at a major disadvantage when forced to compete with huge patent war chests and highly-paid corporate lawyers. The patent system is in dire need of change, especially because of how technology is advancing and how execution is trumping ideas in the online world of SaaS.

The Purpose of Patents

Originally, patents were an important way to protect patent holders from competitors that would steal their product ideas and put product clones on the market without needing to go through potentially expensive R&D efforts. Copying product ideas was a way to try to get a free lunch—easy for the thief, hard on the inventor, and quite damaging to the economy.

If the government didn't try to prevent this copying of ideas, then there would be no incentive for businesses to invest in R&D. Any business that did so would be spending an awful lot of money to develop a product that anyone could turn around and produce without the initial R&D costs, once the product was on the market and was reverse engineered.

With a patent system in place, the original inventor has some time to sell a product before others are allowed to copy it. The patent holder isn't given exclusive rights to produce the invention, but he or she can attempt to enforce the patent when someone has potentially violated it within the term of the patent (usually 20 years in the US). This system would seem to resolve the problem of unscrupulous competitors, but it creates its own set of problems.

The Problem with Patents

The most obvious problem with the patent system is that it created the devious monster known as the patent troll. Some people try to set up companies with a business model of licensing and litigation based on patents that they have no intention of designing into products themselves. They come up with ideas and file patents for them with the most general, wide-reaching claims they can get. Then they try to find small companies (and sometimes large companies) that maybe violate those patents, and send the victims letters that threaten lawsuits if they don't pay licensing fees for the patents they are allegedly infringing on. The aggressor in this situation is the patent troll.

One example of a high-profile company engaging in these tactics is Rambus, Inc. They develop and license high-speed memory interfaces for microprocessors and DRAM, and they've spent plenty of time over the last 15 years in lawsuits with all kinds of semiconductor companies from Samsung to Micron to Nvidia. Instead of making memory products, they design and document memory interfaces and file lots of patents so they can then go after the companies that actually make real products.

The only reason this kind of business model can work is because it's not required to produce the invention to be issued a patent for it. If the original intent of patents was to protect the inventor from theft of her ideas until she had sufficient time to bring a product to market, then this use of patents to license technology instead of building it is a gross distortion of how patents should be used. It wastes resources in court and discourages companies from developing new technologies.

Another related problem with patents happens when companies build up huge war chests of patents and then threaten other companies with lawsuits for patent infringement whenever and wherever they can. The classic example of this problem is when IBM got Sun Microsystems to license some of their patents by threatening to sue over some weak patent infringement claims. It was thinly veiled extortion on the part of IBM's lawyers. Sun paid up because IBM had thousands of patents at their disposal, and IBM could easily outlast Sun in court.

Timothy B. Lee has a great article in the Washington Post on this problem of large patent holders, and he summarizes a reform proposal that may have made some progress on solving this problem. The article's a year old, but still quite relevant. If anything, patent war chests have gotten bigger, and are acting more towards stifling innovation.

Companies are in an arms race to build up a large enough patent portfolio to protect themselves from other companies, even if they have no intention of suing for patent infringement themselves. The danger is that once a company has this large collection of broad patents and a small competitor comes along, the temptation to sue to protect their business is too great.

Patent war chests end up making it very difficult for small companies to innovate because they don't have the resources to make sure they aren't violating any patents or to pay licensing fees and litigation costs if they do. This doesn't mean that small companies should be allowed to blatantly violate patents, but so many patents are either too generic, frivolous, or obvious, or they are already invalid due to prior art that was not considered at filing. There should be an easy, inexpensive way to cleanse the system of these worthless patents that only serve to entrench the old guard and restrict the progress of new ideas.

The Irrelevance of Patents

While arguably patents are still useful for companies that make physical products because of the development and manufacturing time it takes to bring them to market, software companies—especially SaaS companies—are under very different constraints. A modern software startup can go from square one to launching an internet service to an exponentially growing user base in a matter of months. The speed at which these companies need to advance to stay ahead of the competition makes the normal 20-year term of a patent look like an eternity.

In fact, many companies that make physical products can move nearly as fast as SaaS companies because of the rapid prototyping capabilities that are now available for circuit boards, plastics, and other types of hardware. If companies are depending on 15-year old patents to protect their business instead of developing new technologies to compete in the here and now, they're probably not going to last much longer, and their patents are a frictional force on their competitors without actually improving competition. Maybe having a shorter term for patents that can be implemented rapidly, 5 years let's say, would go quite a way towards fixing this problem.

Another reason patents can be so restrictive is that many of them are obvious solutions to the problems you would encounter when developing a product. Especially with software, solutions that are novel and unique are quite rare. Most of the time you're building on top of a massive stack of other technologies, extending it a little more in obvious ways to solve your specific problem. Very few people are inventing new data structures and algorithms that would actually warrant a patent.

Finally, patents are ideas, and ideas are worthless. It doesn't matter how many patents a company holds. The real value of those patents is in the products that company builds and how well they execute in the market. Google, Amazon, Facebook, Twitter, and all the other big internet service companies probably have thousands upon thousands of patents, but at the end of the day, they didn't get where they are because of their patents. They got where they are because of how well they executed as businesses. So why do they have so many patents? They probably feel like they have to, to protect their investments of time and money in their ideas. In reality their patents may be irrelevant at best, and extra baggage that's holding them back at worst. Tesla decided to get rid of all this baggage, and hopefully help the entire auto industry develop EVs faster in the process, by open-sourcing all of their patents.

Exactly how much are these patent war chests holding back the advancement of technology, I wonder. Patents definitely had their uses in the past of protecting a company's ideas while they brought their products to market. Now that products are being developed so quickly, and standing still can be the death knell of a company, patents should change to reflect this new environment. If patents are still useful for software companies, they probably won't be for much longer.

The Importance of Analysis

Assumptions get us into trouble. They can lead us astray, taking us down paths that may look familiar but end in wrong answers and unforeseen problems. Ferreting out the assumptions you're making when solving a problem and validating them against solid evidence is an big part of good analysis. We all have personal biases, and these biases show up in the assumptions we make about problems. Resisting these biases and taking a hard, reasoned look at the problem is both a skill and an art with many aspects to keep in mind.

As an engineer, I analyze and solve problems everyday. These problems are usually pretty narrow and specific to what I'm designing at work, so they don't make good examples for talking about analysis. But the general principles I use are valid for pretty much any problem out there. I've found that examples of these principles of analysis abound in the economics issues facing the US today, so to give them more meaning, we can link each of these principles to a major economic issue. The intent here is not to solve the country's economic problems in one article, but to get a sense of how analysis can be used to understand big, real-world problems that everyone is likely to be aware of. Then you can apply those principles to the more specific problems that you deal with on a daily basis.

Context

To even hope to begin solving a problem, first you must understand the problem. No problem in the real world happens in a vacuum, and neither should our decisions about how to solve those problems. Context is the information about the problem that should keep you from blindly adhering to a pet solution. We all have solutions that we fall back on because it's what we know, but just because a solution might have been right in one situation, doesn't mean it will be right in all situations. Even if the conditions look fairly similar, something might have changed to make the previous decision wrong in the current context.

The obvious economic example of the need for context is the current state of the Federal short-term interest rate, which is stuck at basically zero and has been for the past six years. In normal times, such a policy would have resulted in soaring inflation, and indeed, many knowledgeable people predicted this outcome. But, as Paul Krugman keeps trying to tell us, we are not living in normal times. Current context trumps general rules for how the economy behaves. The housing bubble that burst in 2008 left a lot of consumers in mountains of debt and a lot of businesses without customers. That set of circumstances turns most of macroeconomics on its head, so to make progress on the solutions that will fix the economy, one must first understand the context of the problem we're dealing with now. Keep that in mind the next time you're trying to solve a problem that you think you've solved before.

Trade-offs

Once you know the context, it's time to understand the most important aspects of the problem and how they relate to each other. Some aspects of a problem will be in direct opposition to each other. Other issues form a group that cannot all be optimized at once. If you could optimize everything with no downsides, there would be no problem, right? There are trade-offs inherent to any problem, and optimizing those trade-offs within the context that defines your problem requires sharp judgement.

Taxes are a good example of a problem with trade-offs at multiple levels. How much to tax is an issue. Should we tax more and provide more government services, or should we tax less and cut government spending? Who to tax is an issue. Should we tax everyone at the same rate, or have a progressive tax system? Should we tax certain people less, like people who have children, have a mortgage payment, or donate to charity? What to tax is an issue. Should we tax income, capital, wealth, consumption, bad habits, use of services, or all of the above? Why we tax is even an issue. Should we tax to raise revenue, redistribute income, or regulate businesses?

There are tons of trade-offs when figuring out a tax system, and these are just the big ones. No wonder tax systems are a mess. At least knowing and understanding the trade-offs can help guide you to a better solution to the problem. You can also rest easier knowing that your programming problems are no where near as complex as the US tax system.

Magnitude

When you're analyzing a problem, be sure to consider what orders of magnitude you're working with for different aspects of the problem. If you're considering different options whose benefits are of completely different magnitudes, you're going to want to know that. Similarly, if a particular option has negative consequences that are of a different magnitude than the benefits, you're going to want to know that, too. Being able to judge the magnitude of different options and their consequences enables you to make much better decisions.

Take the minimum wage debate. Some people claim that raising the minimum wage will raise prices, negating the increased wages, but it's a question of magnitude. Sure, doubling everyone's salary in the country is likely to double prices for everything, kicking off severe inflation. That's assuming that such a thing would even be possible, let alone desirable, but raising the minimum wage will only raise some fraction of worker's wages. Some prices may go up, but they'll go up much less than the worker's wages that were at the old minimum, making it an overall win for them financially while not making it that much harder on the rest of us.

When taking the magnitudes of change into account, raising the minimum wage looks much more benign with regards to price levels. The same holds true in many software development problems, especially when it comes to performance optimization. Adding what seems like a significant amount of processing could still result in better performance if it eliminates communication to disk or over a network. Knowing the magnitudes involved when deciding on trade-offs is essential.

Correlations

Interpreting a correlation as causation is a special kind of assumption, and like assumptions, these interpretations can get you into trouble. Having two things be correlated simply means that they move together, either in the same direction or opposite directions. A correlation doesn't say anything about which variable is dependent on the other, or if another invisible variable is involved and forming a more complicated dependence with the other two.

As an example, studies have found poverty to be correlated with single parenthood. As a result, various people advocate for programs that promote marriage as a solution to poverty. That assumes that single parenthood causes poverty, but it could very well be the other way around. If that is the case, then marriage promotion programs aren't going to be very successful at reducing poverty.

Understanding correlations will help you focus your attention on the things that will have the largest impact on the problem you're trying to solve. It's worth the time to make sure your interpretation of the correlation is accurate, and that you're moving the right levers to fix your problem.

Interconnections

Beyond correlations, there are also the general interconnections within your problem domain to consider. Changing something in one place could have far-reaching and unforeseen effects on seemingly unrelated areas of the problem domain due to poorly understood connections that run through the domain. Making the effort to understand the interconnections within your domain can give you much better insight into the issues at play, even if you can't be confident of how things will play out. You can at least have an idea of what to expect and be more prepared if things go wrong.

Take the recent drop in oil prices, for example. At first glance falling oil prices seems like a great thing for most people, but predicting how changing oil prices will effect the economy is more complicated than you would expect due to the web of connections that oil prices have within our economy. Lower oil prices may make oil production uneconomical for companies in Texas, Louisiana, and North Dakota, increasing unemployment in those areas. If this is a strategy of companies in the Middle East to put US oil companies out of business, then it could increase our dependence on foreign oil in the long run. It can also increase our consumption of a resource that's becoming more limited while reducing the incentive to develop alternatives like solar energy and EVs, not to mention the increased pollution that would result. When the wells run dry faster than anticipated, we may be in for a shock as gasoline prices skyrocket and we're caught unprepared.

I'm not predicting that this scenario will actually come to pass, but the interconnections are there to make it possible. Being aware of that, and at least analyzing and preparing for such a possibility would be a prudent thing to do. The same reasoning goes for problems in any domain. You should at least be aware of the interconnections, even if the future is uncertain, so that you can make contingency plans and be prepared to handle undesirable situations if they come up.

Irreducible Complexity

Sometimes you want to simplify a situation, to make your problem easier to deal with, but you can't. Some problems are too complex, have too many interconnections, and have intricate dependencies that cannot be decoupled. These complex problems require sufficiently complex solutions. If you try to simplify the solution too much, it will no longer solve the problem, and in some cases can actually make the problem worse.

The US health care system is one such beast. If anything has irreducible complexity, it's health care. Between patients, the insured, the uninsured, insurance providers, care providers, medical device manufacturers, pharmaceutical companies, and the government, there are thousands of businesses, millions of people, and trillions of dollars involved. A system of this scale is bound to waste some money, but it is pretty well documented that the US wastes much more than it should on health care. Any system that attempts to reduce the waste in US health care is going to be complicated. Even the first step of figuring out where all of the waste is in the system is complicated. It's no wonder the ACA is so complicated. The real question is, does the complexity of the ACA address the irreducible complexity of the US health care system?

When analyzing a problem and trying to simplify it in order to develop a tractable solution, it's important to be aware of when you've hit the point of irreducible complexity. Every problem has a complexity threshold, and if you oversimplify the problem, the solution is not going to fit. Part of good analysis is identifying where the point of irreducible complexity is, and solving for that level of complexity.

FUD

At some point during the analysis of a problem, especially as the problem gets more complicated, you may encounter FUD (fear, uncertainty, and doubt). Someone—or many people—starts thinking, what if this solution doesn't work? What if it has unintended consequences? What if it blows up in our faces? We shouldn't do it! It's too risky!! Look, because of this thing right here, the whole system is going to collapse, and the world will end!!! At this point you need to take a deep breath and calmly analyze if these claims have any merit. Maybe the solution is too risky for various reasons, but the correct decision cannot be reached through the emotions of FUD.

One recent example of FUD was the semi-successful NASA mission to land the space probe Philae on a comet. While not exactly an economic example, plenty of money was spent on the mission with significant risks to success because of FUD. The mission was successful in that Philae did indeed land on the comet, but due to a couple of malfunctions with its landing gear, it bounced around and came to rest at the base of a cliff, making the solar panels it was equipped with ineffective as an energy source. If Philae had used plutonium-238—an entirely safe, non-weapons-grade radioisotope—as its power source, the fudged landing wouldn't have mattered, and Philae would have had enough power to take all of the measurements NASA intended as the comet approached the sun.

The problem is that plutonium-238 has the word "plutonium" in it, so enough people are afraid of it without understanding it and realizing how benign it is. If reasonable analysis could have been used in this case to overcome the FUD, we may have gained much more insight into the properties of comets, greatly increasing our knowledge of another piece of the universe. Try not to let FUD get in the way of your analysis of potential solutions to your problems.

Counterexamples

One way to counteract FUD is to find counterexamples. Are there actual examples that dispute the opposition's claims? If your opponents are claiming that a solution can't work because of some general principle, finding a counterexample can cause their argument to collapse and force everyone to look at the context and trade-offs at hand instead of speaking in generalities.

As an example, let's turn back to taxes. Americans generally believe raising taxes on the middle class and the poor reduces work incentives, resulting in higher unemployment. If you don't get to keep as much of your hard-earned money, then what's the point in working so much, right? (Bonus: can you see the logical flaw in this argument?) We can actually test this theory because there are plenty of countries with much higher taxes than the US. Do we actually see reduced employment in countries with higher taxes? Scandinavian countries are a glaring counterexample to this theory, so there must be more in play than the general claim that high tax rates equals lower employment.

Counterexamples are good for deciding whether or not general rules apply in particular situations. If counterexamples exist that disprove the general rule, then you should look more closely at the context of the problem at hand to decide what the best solution will be.

Counterfactuals

Counterfactuals are similar to counterexamples, except that you're not looking for a real example to disprove a general rule; you're doing a mental comparison of what happens to a system in the presence or absence of something. It's very easy to take a one-sided view that something does (or doesn't) work without looking at the opposite situation. Doing the comparison leads to much more sound analysis and better insights.

A good example of the benefit of counterfactuals comes from the Obama stimulus package in early 2009. The economy was still cratering when the ARRA went into effect, and unemployment peaked at 10%, well above the estimates coming out of the Obama administration when they were promoting the predicted benefits of the stimulus. It's very easy to conclude that the stimulus didn't work, but that ignores what would have happened without the stimulus. Yes, 10% unemployment was way above the administration's estimates, but the estimates were just wrong. That doesn't mean the stimulus didn't work. It means without the stimulus, unemployment would have been much higher. In fact, nearly all economists responding to a recent survey agreed that the stimulus did reduce unemployment.

Counterfactuals are very important when analyzing a problem. Adding and removing different parts of a system and comparing the expected outcomes is a great way to gain a better understanding of trade-offs and interconnections. Such an exercise may show you that even though a particular solution didn't fix the problem, it helped, and it may be worth looking at extending the solution instead of discounting it.

Alternative Solutions

An analysis is never complete if you only look at one possible solution. Having a collection of alternative solutions to compare and choose from is critical to good analysis. Coming up with alternatives really brings together all of the other points listed above. The best solutions will fit the context of the problem and optimize the trade-offs for your particular goals, taking the magnitudes of various effects into account. The correlations, interconnections, and complexity of the problem domain will bring out different solutions and highlight the strengths and weaknesses of them. Alternative solutions will include counterexamples and counterfactuals to help combat FUD. In the end, if you've done your analysis well, you'll have a good selection of alternatives to choose from, and you'll be more confident of deciding on the best solution.