The Importance of Analysis

Assumptions get us into trouble. They can lead us astray, taking us down paths that may look familiar but end in wrong answers and unforeseen problems. Ferreting out the assumptions you're making when solving a problem and validating them against solid evidence is an big part of good analysis. We all have personal biases, and these biases show up in the assumptions we make about problems. Resisting these biases and taking a hard, reasoned look at the problem is both a skill and an art with many aspects to keep in mind.

As an engineer, I analyze and solve problems everyday. These problems are usually pretty narrow and specific to what I'm designing at work, so they don't make good examples for talking about analysis. But the general principles I use are valid for pretty much any problem out there. I've found that examples of these principles of analysis abound in the economics issues facing the US today, so to give them more meaning, we can link each of these principles to a major economic issue. The intent here is not to solve the country's economic problems in one article, but to get a sense of how analysis can be used to understand big, real-world problems that everyone is likely to be aware of. Then you can apply those principles to the more specific problems that you deal with on a daily basis.

Context

To even hope to begin solving a problem, first you must understand the problem. No problem in the real world happens in a vacuum, and neither should our decisions about how to solve those problems. Context is the information about the problem that should keep you from blindly adhering to a pet solution. We all have solutions that we fall back on because it's what we know, but just because a solution might have been right in one situation, doesn't mean it will be right in all situations. Even if the conditions look fairly similar, something might have changed to make the previous decision wrong in the current context.

The obvious economic example of the need for context is the current state of the Federal short-term interest rate, which is stuck at basically zero and has been for the past six years. In normal times, such a policy would have resulted in soaring inflation, and indeed, many knowledgeable people predicted this outcome. But, as Paul Krugman keeps trying to tell us, we are not living in normal times. Current context trumps general rules for how the economy behaves. The housing bubble that burst in 2008 left a lot of consumers in mountains of debt and a lot of businesses without customers. That set of circumstances turns most of macroeconomics on its head, so to make progress on the solutions that will fix the economy, one must first understand the context of the problem we're dealing with now. Keep that in mind the next time you're trying to solve a problem that you think you've solved before.

Trade-offs

Once you know the context, it's time to understand the most important aspects of the problem and how they relate to each other. Some aspects of a problem will be in direct opposition to each other. Other issues form a group that cannot all be optimized at once. If you could optimize everything with no downsides, there would be no problem, right? There are trade-offs inherent to any problem, and optimizing those trade-offs within the context that defines your problem requires sharp judgement.

Taxes are a good example of a problem with trade-offs at multiple levels. How much to tax is an issue. Should we tax more and provide more government services, or should we tax less and cut government spending? Who to tax is an issue. Should we tax everyone at the same rate, or have a progressive tax system? Should we tax certain people less, like people who have children, have a mortgage payment, or donate to charity? What to tax is an issue. Should we tax income, capital, wealth, consumption, bad habits, use of services, or all of the above? Why we tax is even an issue. Should we tax to raise revenue, redistribute income, or regulate businesses?

There are tons of trade-offs when figuring out a tax system, and these are just the big ones. No wonder tax systems are a mess. At least knowing and understanding the trade-offs can help guide you to a better solution to the problem. You can also rest easier knowing that your programming problems are no where near as complex as the US tax system.

Magnitude

When you're analyzing a problem, be sure to consider what orders of magnitude you're working with for different aspects of the problem. If you're considering different options whose benefits are of completely different magnitudes, you're going to want to know that. Similarly, if a particular option has negative consequences that are of a different magnitude than the benefits, you're going to want to know that, too. Being able to judge the magnitude of different options and their consequences enables you to make much better decisions.

Take the minimum wage debate. Some people claim that raising the minimum wage will raise prices, negating the increased wages, but it's a question of magnitude. Sure, doubling everyone's salary in the country is likely to double prices for everything, kicking off severe inflation. That's assuming that such a thing would even be possible, let alone desirable, but raising the minimum wage will only raise some fraction of worker's wages. Some prices may go up, but they'll go up much less than the worker's wages that were at the old minimum, making it an overall win for them financially while not making it that much harder on the rest of us.

When taking the magnitudes of change into account, raising the minimum wage looks much more benign with regards to price levels. The same holds true in many software development problems, especially when it comes to performance optimization. Adding what seems like a significant amount of processing could still result in better performance if it eliminates communication to disk or over a network. Knowing the magnitudes involved when deciding on trade-offs is essential.

Correlations

Interpreting a correlation as causation is a special kind of assumption, and like assumptions, these interpretations can get you into trouble. Having two things be correlated simply means that they move together, either in the same direction or opposite directions. A correlation doesn't say anything about which variable is dependent on the other, or if another invisible variable is involved and forming a more complicated dependence with the other two.

As an example, studies have found poverty to be correlated with single parenthood. As a result, various people advocate for programs that promote marriage as a solution to poverty. That assumes that single parenthood causes poverty, but it could very well be the other way around. If that is the case, then marriage promotion programs aren't going to be very successful at reducing poverty.

Understanding correlations will help you focus your attention on the things that will have the largest impact on the problem you're trying to solve. It's worth the time to make sure your interpretation of the correlation is accurate, and that you're moving the right levers to fix your problem.

Interconnections

Beyond correlations, there are also the general interconnections within your problem domain to consider. Changing something in one place could have far-reaching and unforeseen effects on seemingly unrelated areas of the problem domain due to poorly understood connections that run through the domain. Making the effort to understand the interconnections within your domain can give you much better insight into the issues at play, even if you can't be confident of how things will play out. You can at least have an idea of what to expect and be more prepared if things go wrong.

Take the recent drop in oil prices, for example. At first glance falling oil prices seems like a great thing for most people, but predicting how changing oil prices will effect the economy is more complicated than you would expect due to the web of connections that oil prices have within our economy. Lower oil prices may make oil production uneconomical for companies in Texas, Louisiana, and North Dakota, increasing unemployment in those areas. If this is a strategy of companies in the Middle East to put US oil companies out of business, then it could increase our dependence on foreign oil in the long run. It can also increase our consumption of a resource that's becoming more limited while reducing the incentive to develop alternatives like solar energy and EVs, not to mention the increased pollution that would result. When the wells run dry faster than anticipated, we may be in for a shock as gasoline prices skyrocket and we're caught unprepared.

I'm not predicting that this scenario will actually come to pass, but the interconnections are there to make it possible. Being aware of that, and at least analyzing and preparing for such a possibility would be a prudent thing to do. The same reasoning goes for problems in any domain. You should at least be aware of the interconnections, even if the future is uncertain, so that you can make contingency plans and be prepared to handle undesirable situations if they come up.

Irreducible Complexity

Sometimes you want to simplify a situation, to make your problem easier to deal with, but you can't. Some problems are too complex, have too many interconnections, and have intricate dependencies that cannot be decoupled. These complex problems require sufficiently complex solutions. If you try to simplify the solution too much, it will no longer solve the problem, and in some cases can actually make the problem worse.

The US health care system is one such beast. If anything has irreducible complexity, it's health care. Between patients, the insured, the uninsured, insurance providers, care providers, medical device manufacturers, pharmaceutical companies, and the government, there are thousands of businesses, millions of people, and trillions of dollars involved. A system of this scale is bound to waste some money, but it is pretty well documented that the US wastes much more than it should on health care. Any system that attempts to reduce the waste in US health care is going to be complicated. Even the first step of figuring out where all of the waste is in the system is complicated. It's no wonder the ACA is so complicated. The real question is, does the complexity of the ACA address the irreducible complexity of the US health care system?

When analyzing a problem and trying to simplify it in order to develop a tractable solution, it's important to be aware of when you've hit the point of irreducible complexity. Every problem has a complexity threshold, and if you oversimplify the problem, the solution is not going to fit. Part of good analysis is identifying where the point of irreducible complexity is, and solving for that level of complexity.

FUD

At some point during the analysis of a problem, especially as the problem gets more complicated, you may encounter FUD (fear, uncertainty, and doubt). Someone—or many people—starts thinking, what if this solution doesn't work? What if it has unintended consequences? What if it blows up in our faces? We shouldn't do it! It's too risky!! Look, because of this thing right here, the whole system is going to collapse, and the world will end!!! At this point you need to take a deep breath and calmly analyze if these claims have any merit. Maybe the solution is too risky for various reasons, but the correct decision cannot be reached through the emotions of FUD.

One recent example of FUD was the semi-successful NASA mission to land the space probe Philae on a comet. While not exactly an economic example, plenty of money was spent on the mission with significant risks to success because of FUD. The mission was successful in that Philae did indeed land on the comet, but due to a couple of malfunctions with its landing gear, it bounced around and came to rest at the base of a cliff, making the solar panels it was equipped with ineffective as an energy source. If Philae had used plutonium-238—an entirely safe, non-weapons-grade radioisotope—as its power source, the fudged landing wouldn't have mattered, and Philae would have had enough power to take all of the measurements NASA intended as the comet approached the sun.

The problem is that plutonium-238 has the word "plutonium" in it, so enough people are afraid of it without understanding it and realizing how benign it is. If reasonable analysis could have been used in this case to overcome the FUD, we may have gained much more insight into the properties of comets, greatly increasing our knowledge of another piece of the universe. Try not to let FUD get in the way of your analysis of potential solutions to your problems.

Counterexamples

One way to counteract FUD is to find counterexamples. Are there actual examples that dispute the opposition's claims? If your opponents are claiming that a solution can't work because of some general principle, finding a counterexample can cause their argument to collapse and force everyone to look at the context and trade-offs at hand instead of speaking in generalities.

As an example, let's turn back to taxes. Americans generally believe raising taxes on the middle class and the poor reduces work incentives, resulting in higher unemployment. If you don't get to keep as much of your hard-earned money, then what's the point in working so much, right? (Bonus: can you see the logical flaw in this argument?) We can actually test this theory because there are plenty of countries with much higher taxes than the US. Do we actually see reduced employment in countries with higher taxes? Scandinavian countries are a glaring counterexample to this theory, so there must be more in play than the general claim that high tax rates equals lower employment.

Counterexamples are good for deciding whether or not general rules apply in particular situations. If counterexamples exist that disprove the general rule, then you should look more closely at the context of the problem at hand to decide what the best solution will be.

Counterfactuals

Counterfactuals are similar to counterexamples, except that you're not looking for a real example to disprove a general rule; you're doing a mental comparison of what happens to a system in the presence or absence of something. It's very easy to take a one-sided view that something does (or doesn't) work without looking at the opposite situation. Doing the comparison leads to much more sound analysis and better insights.

A good example of the benefit of counterfactuals comes from the Obama stimulus package in early 2009. The economy was still cratering when the ARRA went into effect, and unemployment peaked at 10%, well above the estimates coming out of the Obama administration when they were promoting the predicted benefits of the stimulus. It's very easy to conclude that the stimulus didn't work, but that ignores what would have happened without the stimulus. Yes, 10% unemployment was way above the administration's estimates, but the estimates were just wrong. That doesn't mean the stimulus didn't work. It means without the stimulus, unemployment would have been much higher. In fact, nearly all economists responding to a recent survey agreed that the stimulus did reduce unemployment.

Counterfactuals are very important when analyzing a problem. Adding and removing different parts of a system and comparing the expected outcomes is a great way to gain a better understanding of trade-offs and interconnections. Such an exercise may show you that even though a particular solution didn't fix the problem, it helped, and it may be worth looking at extending the solution instead of discounting it.

Alternative Solutions

An analysis is never complete if you only look at one possible solution. Having a collection of alternative solutions to compare and choose from is critical to good analysis. Coming up with alternatives really brings together all of the other points listed above. The best solutions will fit the context of the problem and optimize the trade-offs for your particular goals, taking the magnitudes of various effects into account. The correlations, interconnections, and complexity of the problem domain will bring out different solutions and highlight the strengths and weaknesses of them. Alternative solutions will include counterexamples and counterfactuals to help combat FUD. In the end, if you've done your analysis well, you'll have a good selection of alternatives to choose from, and you'll be more confident of deciding on the best solution.

Caring for Our Raw Materials

I greatly respect the Pragmatic Programmers, Andy Hunt and Dave Thomas, and I have always enjoyed reading their stuff. I was perusing their publishing website recently, and came across a nice article about how we should develop ourselves as programmers because we are the raw materials that go into the construction of software. In their own words:

We are the only raw material of consequence in software development. Oh sure, the process will involve a few keyboards and mice, some compilers, database products, and myriad office supplies, but they’re all completely secondary. Contrary to popular myth, we don’t write software on computers. We don’t write software in programming languages, integrated development environments or case tools, whiteboards, or 3×5 cards.
We write software in our heads.

They go on to describe a number of ways to prepare ourselves to be good developers:

• Use the right tool for the job, and know how to use the tools.
• See the software you're writing within the larger context in which it will be used.
• Own up to our mistakes, and offer solutions, not excuses.
• Communicate what we're doing, how we're stuck, and where we plan to go next.
• Learn, not just about new technologies, but also about our own abilities and shortcomings.

That's the summary of an already quick article, and they're all excellent points. The article may have been short, but it left me thinking about what we can do to get the most out of our own mental capacity. These five points cover how a good developer approaches software development—what quality raw materials look like—but there's another aspect of these raw materials that's equally important. How do we take care of our raw materials?

Learn Three Ways

All of those aspects of a good developer are things that need to be learned. In software we learn about tools, we learn problem domains, we learn how to problem solve, we learn to communicate, and we learn how to learn. Each of these topics is an article, or an entire book, unto itself, but let's briefly explore the last one, learning how to learn.

Learning how to learn is a lifelong endeavor, if you keep working at it. I have never been better at learning than I am now, but next year I hope to be better. I'll have picked up more knowledge that will act as a foundation for still more knowledge. I'll also know myself better and what learning methods work best for me.

When I was in high school, I thought learning meant studying. I would immerse myself in books of all subjects and work through plenty of assigned problems in math and science. While this method of learning does give you loads of practice with the concepts, it doesn't provide much context for using what you've learned in real world applications. That's probably why math teachers so commonly get asked, "When will we ever need to use this stuff in real life?!"

I still study a lot, and I can't read enough books to satiate my appetite. But in college I learned the value of actually doing things in addition to studying. You can study until you're blue in the face, but you don't really know something until you've tried to use it in a real project. Take circuits for example. You can read all about Ohm's law, how op amps work, negative feedback, stability, PSRR, and all that stuff. Once you've read about it all, do you know how to build an amplifier circuit for conditioning a signal from a pressure sensor? Almost certainly not. You should have some idea of how to do it, but you'll learn a great deal more about circuit design by actually doing it.

Over the last couple years I've found that studying and doing are still only part of the whole learning picture. The last part of the picture is teaching. Teaching something to others requires an entirely new understanding of the topic you're trying to teach. I've constantly come up against this issue when writing my more technical blog posts. I thought I knew statistics pretty well, but that series was still pretty challenging. I looked up a lot of things that I use all the time to make sure I got them right. Part of the difference is that when you're using knowledge to do something, you have your own understanding of things, and that works to get the job done. When you need to turn around and teach that knowledge to others, you need to think about how to convey that knowledge in a clear enough way that a reasonable number of people with widely varying backgrounds have a chance of getting it. That's a hard thing to do, and you learn a ton by doing it.

Attempting to teach, even in the small way that I'm doing it on this blog, has given me an even greater respect for all of the people that do it as a full-time career. It's also helped to show me that learning is so much more than studying. Learning by studying, doing, and teaching gives you a much deeper understanding of a subject than reading books alone. With that complete picture of learning, we can move on to the other ways to care for our raw materials as programmers.

Body and Mind

Our minds are not disconnected from the physical world, able to operate at full efficiency regardless of how we treat our bodies. We can't expect to do all of this learning and thinking and problem solving while ignoring the rest of our well being. Our minds are intimately connected to our physical selves, and how well we feel greatly affects how we think. There are at least four aspects of taking care of ourselves that directly impacts our ability to be effective programmers.

First, get enough sleep. This point should be obvious, but just because it's obvious doesn't mean it's easy. The average person should get about 7-8 solid hours of sleep a night. When you're busy working on multiple projects at work and at home while taking care of and taking part in the rest of what life has to offer, that kind of sleep goal is hard to meet. When you do get enough sleep, the return on that investment is worth it. You think more clearly. You're better able to deal with the minor stresses that happen throughout your day. Your problem solving abilities greatly improve. If you skimp on sleep to try to get more done in a day, you're building up a debt that you're going to have to pay sooner or later, and it's counter productive in the mean time. Get enough uninterrupted, quality sleep so you can be a rock star instead.

Second, eat well. Your mind needs fuel to think, and the right kinds of foods will help improve your performance. Loading up with lots of greasy, fatty foods and sugary soft drinks will either make you feel dull and sluggish or energized for a short time before crashing hard. A good balance of fruits, veggies, complex carbs, and protein will do a much better job of keeping you humming along, thinking at full capacity all day long. Oh, and don't even thinking about skipping breakfast. It's what kick-starts your day. Don't skip lunch, either. And don't eat dinner too close to bed time. You'll sleep better.

Third, exercise regularly. Whenever I exercise, I feel much more alert and better at taking on problems, and that feeling lasts for a good day or more. On the other hand, when I skip a workout, I definitely feel down the next day. I hate skipping workouts. A good workout doesn't have to bring you to your knees. In fact, it shouldn't. A good workout should make you feel rejuvenated, and a simple 30 minute walk each day, whenever you can fit it in, could do that for you. It helps you breathe more deeply, take in more oxygen, and get your blood flowing to help you think better. Regular exercise also helps you sleep better.

Finally, relax. We are not machines. We can't expect to go, go, go all the time without taking a break. Leisure time is important, too. It gives your brain time to rest and recharge, and you'll remember some of the other good things in life, besides programming. Taking some time off can also help you solve some of those programming problems you've been struggling with. Stepping away from a problem, and letting it simmer in the back of your mind, will often lead to sudden realizations and new insights. When you come back to that mind-bending problem after a good vacation, you may be surprised by a moment of clarity where everything falls into place. The problem that you were working so hard on before may seem to solve itself. Additionally, taking time to relax will reduce your stress level and, you guessed it, help you sleep better.

While we do write software in our heads, and preparing ourselves as the raw materials of software development is largely a mental process, We should take a very holistic view of how to care for our raw materials. We are not just brains sitting in jars, building complex programs through pure thought. Our minds are inseparably linked to our bodies, and our thought processes are highly dependent on how our bodies feel. To learn all we need to learn to be effective programmers, we need to take care of ourselves through quality sleep, good nutrition, regular exercise, and some R&R. With that, I think it's time for bed. Sweet dreams.

Physics Book Face Off: Hyperspace Vs. The Elegant Universe

I've always been interested in physics. It's the subject that tries to answer the ultimate question of how the universe, and everything in it, works at its most fundamental level. I took a few physics courses in college, but I started to shy away from the subject after taking modern physics and having it go way over my head. I had a hard time grasping the concepts at the time, but recently, like with mathematics, I've been thinking about getting back into studying it more.

To kick off that activity, I started with two popular physics books that may be a little outdated, but should still have plenty of relevant, intriguing material on what has happened in the field post-Einstein. Both books are by prominent string theorists. The Elegant Universe was written in 1999 by Brian Greene, a professor of physics and mathematics at Columbia University. Hyperspace was written five years earlier in 1994 by Michio Kaku, a professor in theoretical physics at The City University of New York.

VS.

The Elegant Universe: Superstrings, Hidden Dimensions, and the Quest for the Ultimate Theory

From what I've read, there's a huge debate going on in theoretical physics right now about whether or not string theory is the future of how we will understand the universe, or if it's a dead end that will never produce meaningful predictions about how the universe works. I'm certainly not qualified to make any judgements about this debate, but I still believe that the investigations of string theory have merit because the exploration of ideas has value in and of itself. String theory has also made significant contributions to both mathematics and physics by developing new mathematical constructs and bringing various far-flung ideas between the two subjects together under one framework.

That, however, is not the point of this book. The point is to give the reader a basic understanding of what string theory is about and how it affects our idea of how space-time works. Brian Greene is an excellent writer, and he does a great job of conveying his ideas in a way that non-theoretical physicists can understand. He starts out with a detailed description of Einstein's theories of Special and General Relativity and how they change our concept of the flow of time and the structure of space. Then he leaves the expanse of space to describe the main features of the very small particles of quantum mechanics. He wraps up this introductory material by explaining how these two sides of the universe—the very large and the very small—are incompatible when viewed within the confines of relativity and quantum mechanics. The two fields even come in direct conflict when trying to calculate what happens inside black holes or during the Big Bang.

This conflict is what string theory attempts to resolve. The rest of the book describes what sting theory is, how it can combine relativity and quantum mechanics into one overarching Theory of Everything, and goes into a number of issues that the theory must address before it can be considered valid. Towards the end of the book, Greene gets caught up in generalities and doesn’t do as good of a job relating the physics he's describing to everyday reality. He talks about strings wrapping around curled up dimensions and branes covering tears in the fabric of space. It's very hard to visualize what he's talking about and what implications it has for the behavior of space-time, but maybe the vagueness betrays the fact that no one really understands what's going on here, yet.

His other explanations are quite good, and reading the book generated tons of questions in my mind about how the concepts he was describing could be extended. For example, when he was describing how it's known that gravity travels at the speed of light, he goes through a thought experiment about what would happen to the planets if the sun suddenly exploded. The gist of the explanation is that the planets would not immediately leave their orbits because it would take time for the change in gravity to reach each planet.

As I was reading, I wondered what would happen if instead of exploding, the sun disappeared entirely, just winking out of existence (never mind how that might physically happen). Would it still take time for the change in gravity to reach the planets? At first I struggled with this idea because it seemed to me that in the first case of the sun exploding, the change in gravity would move along with the remnants of the sun, so the fact that gravity would be limited to the speed of light wasn't surprising. The matter that was traveling outward from the blast would be limited by the speed of light, and the force of gravity would change based on what happened to the matter as it sped outward. What was surprising was that, according to Einstein, even if the sun just disappeared, the planets would still take time to notice the absence of the sun's gravity because the sun was warping the space around it and the planets were following the curve of space in their orbits. The speed at which space would flatten out in the absence of the sun would happen at the speed of light.

Another great part of the book was the discussion of how to visualize higher dimensions. The concept of extra dimensions beyond three is especially hard to grasp because we experience the world in three dimensions, and we have no reference for what a fourth dimension (let alone a tenth dimension) would be like. One way to think about this—the way the book describes—is to imagine how a being in a one dimensional world would see a two dimensional object, and then work your way up to higher dimensions. Another way, that the book doesn't go into, is to think about how we already see our three dimensional world as a two dimensional projection. Our eyes actually see in 2D, and we build 3D models of objects in our minds. Similarly, we can project 4D objects into 3D spaces with computer simulations to try to get a better idea of what they are. One common example is the tesseract, or four dimensional cube. Here is a video showing what it looks like to rotate and unwrap a tesseract:

Even after watching the video, it's still really hard to visualize what's going on, but every way of trying to imagine higher dimensions adds a bit more to our understanding.

Understanding higher dimensions is a big part of string theory because in higher dimensions there is more room to unite all of the forces and fundamental particles into one theory. Greene does a great job describing the issues and implications with string theory, as well as much of the physics history that has led up to it. It was a great read that gave me a much better understanding of what string theory is all about, and I'm looking forward to reading more of his books.

Hyperspace: A Scientific Odyssey Through Parallel Universes, Time Warps, and the 10th Dimension

Hyperspace is another book about string theory, although the focus here is much more on higher dimensions than the details of vibrating strings and how they interact. Michio Kaku also spends a lot of time on the history of physics, and the older I get the more I appreciate the context that history provides. It will also be interesting to see how his ideas change in later books because of the impact of new technologies that have come into play since this book was written. He talks a fair amount about the possibility of the universe ending in the Big Crunch, but since the Hubble Telescope has been in service and our measurements of the cosmic microwave background (CMB) radiation have gotten better, we have pretty much ruled that out as a possibility. The LHC has also added many new discoveries to physics research and our understanding of quantum mechanics that would impact string theory as well.

Despite its age, this book was a fascinating read. Kaku explores so many interesting topics, including time travel, the tenth dimension, worm holes, and the future of our civilization. Like Greene, he attempts to describe how to visualize higher dimensions, and the additional perspective is helpful. He has an easy conversational style that clearly conveys his ideas, and I especially enjoyed his discussion of how civilizations are predicted to develop:

A Type I civilization is one that controls the energy resources of an entire planet. This civilization can control the weather, prevent earthquakes, mine deep in the earth’s crust, and harvest the oceans. This civilization has already completed the exploration of its solar system. A Type II civilization is one that controls the power of the sun itself. This does not mean passively harnessing solar energy; this civilization mines the sun. The energy needs of this civilization are so large that it directly consumes the power of the sun to drive its machines. This civilization will begin the colonization of local star systems. A Type III civilization is one that controls the power of an entire galaxy. For a power source, it harnesses the power of billions of star systems. It has probably mastered Einstein’s equations and can manipulate space-time at will.

We are currently a Type 0 civilization, and when put in this context, it's pretty clear that the most important advances we can make right now are in energy production. Making progress in new computing devices, robotics, and transportation are still important, of course, but we're not going to get anywhere until we dramatically increase the amount of energy available to us.

I also wonder if the process of advancement to a Type I civilization would go faster if more resources were allocated to it. We chronically underfund space exploration and basic research. However, Kaku makes an interesting argument that technological advancement should not outpace social development or else we'll be in critical danger of self-annihilation. We still need to figure out how to function productively as one world-wide civilization instead of a collection of nation-states in constant conflict. We will continue to be on the edge of destruction until we solve the political, social, and environmental problems that we're dealing with. Technology that's too advanced for our social structures will only make things worse.

Putting our sociopolitical issues aside and turning back to string theory, Kaku goes beyond the normal assertions that it has the potential to unify our separate models of the universe, from the Standard Model to the four fundamental forces. He thinks it also has the potential to unify much of the separate fields of mathematics:

One consequence of this formulation is that a physical principle that unites many smaller physical theories must automatically unite many seemingly unrelated branches of mathematics. This is precisely what string theory accomplishes. In fact, of all physical theories, string theory unites by far the largest number of branches of mathematics into a single coherent picture. Perhaps one of the by-products of the physicists’ quest for unification will be the unification of mathematics as well.

This seems like a pivotal accomplishment for our civilization, when and if it occurs. The book is packed with high-flying ideas like this that make you think and wonder about what is possible in our future. Kaku strikes a good balance between explaining the history of physics and its future potential. Like The Elegant Universe, I thoroughly enjoyed reading this book, and I look forward to more of Michio Kaku's books.

Wish I Would Have Read These Sooner

 It would have been very useful to have read books like these in college while taking difficult math and physics courses. It would have helped give context to the things I was learning, and motivated me to pursue a deeper understanding of things I struggled with. I remember having a really difficult time conceptualizing things like black body radiation and the Schrödinger equation at the time, and Kaku's example of students not understanding the implications of an exam problem with an intriguing application sounded eerily familiar to me: 

In the autumn of 1985, on the final exam in a course on general relativity given at Caltech, Thorne gave the worm-hole solution to the students without telling them what it was, and they were asked to deduce its physical properties. (Most students gave detailed mathematical analyses of the solution, but they failed to grasp that they were looking at a solution that permitted time travel.)

If I had read books like these in college, I would have had a much better grasp of the general concepts of physics, and some of the things that flew over my head may have found a better place to stick instead. I'm sure I would have been much more motivated to understand the complex equations involved if I would have known that books like these existed (and read them). I'm definitely motivated now. Every aspiring physics major, and even hobbyists, should take a look at these books to see what wonders and paradoxes the universe holds.

Idea Management

Ideas are a dime a dozen. On any given day I come up with multiple ideas for half a dozen different projects at home and at work. If I'm reading a good book, it will generate even more ideas, and I may want to remember or write about them at some later date. When you're thinking about hard problems, ideas just happen, and they can happen anywhere at anytime. Sometimes I feel overwhelmed with ideas, and I have to organize them.

One project in particular that I need to manage is this blog. When I started nearly two years ago, I came up with a list of ideas that I thought would get me through the first six months of writing one article per week. This is now my 95th post, and my list of ideas to work from has grown to 56. (At that rate, I should be well stocked with ideas until the heat-death of the universe.)

I've tried many different tools for managing these ideas, and they each have their pros and cons. I started out with a simple outlined list in Google Docs. I tried using mind maps. I even took a look at Basecamp. Finally, I settled on Trello, which seems to fit my preferred workflow pretty well. Here's a blow-by-blow of how I came to that conclusion and why Trello is working better for me than any other tools I've tried.

The Outline

• Topic 1
• Idea 1
• Sub Idea 1
• Sub Idea 2
• Idea 2
• Idea 3
• Sub Idea 1
• Sub Idea 2
• Sub Idea 3
• Topic 2
• Idea 1
• Idea 2

The outline is the classic organizational tool. I learned in high school, as did many people, that when planning out a writing assignment, the outline is a great way to get your thoughts organized and to flesh them out. You can list your main points, sub-points for each main point, links to references, quotes, and detailed thoughts that you know you want to hit in your prose. You can go as deep as you need to go, and it's all laid out nicely on the page or screen so you can see your whole argument at once.

If your outlines are on the computer, it's fairly easy to reorder ideas and modify them, allowing you to shape your argument as you see fit. There are software programs made specifically for outlining that make this even easier. I decided to go with Google Docs for my blog idea list because I already use it, it's a straightforward, low-tech solution, and it's available anywhere I have access to the internet.

It's really important to be able to access these outlines almost anywhere, because that's where I get ideas—anywhere. I could carry a notebook for those times that I'm away from my main computer, but then I lose the flexibility of a digital document. Besides, Google Docs is so much more convenient. I can edit them anywhere and not worrying about keeping track of a physical object. It's very rare that I don't have access to Google, and in those situations where I don't, I can jot a note on a piece of paper, shove it in my pocket, and transfer it the next chance I get. I think I just talked myself out of ever buying a Moleskine.

Google Docs isn't all good when it comes to keeping track of lots of blog ideas, though. Its main drawback is that it doesn't scale. If I want to keep all of the ideas in one document, and I'm expanding on a number of them while taking notes on a couple books that I'm reading at the same time, it gets unwieldy rather quickly. On top of that, once I'm done writing about a topic, I have to delete the notes for that topic or at least move them to another document so they don't clutter up my list of ideas with ones I've already written about. That's not efficient.

Additionally, some of my notes for books will go on for multiple pages, which makes it difficult to get an overview of all of the topics I have to choose from when I want to pick one to write about. I could create a separate document for each topic, but then I lose the ability to organize the topics the way I want, and it becomes much more difficult to move ideas from one topic to another. In the end Google Docs was not working out so I went in search of alternatives.

The Mind Map

I was turned on to mind maps by Kathy Sierra, who passionately advocated for them on her blog. The idea is compelling, so I gave mind mapping a try by using them to take notes for half a dozen different books. What I found was that it didn't really fit in with my workflow, and it ended up being a lot more work than it was worth. For highly visual people, mind mapping may provide more benefit, but I guess I'm not one of those people. I didn't get much more insight into the books from looking at notes in mind map form than I do from looking at them in outline form.

In the end outlines and mind maps are two equivalent ways to represent the same thing, but in different formats. For me, the mind map had a number of downsides in addition to those already plaguing outlines. First, the only decent software I could find at the time was desktop software, which severely limited my access to the maps. Today, I would probably use something like mindmup.com, which looks really slick. It saves maps to Google Drive, makes nice-looking maps like the one above, and it's free.

Good cloud software eliminates the availability issue, but there are other issues. For instance, I read most of my books on a Kindle, and highlighting and taking notes is built into the Kindle software. I can do all of the note-taking I need without having to leave the book, everything is automatically copied up to Amazon's cloud, and then I can copy and paste my saved notes and quotes directly from Amazon's cloud to my blog. Building a mind map is a lot of extra work when the Kindle makes note taking so easy and seamless.

That's not all. The biggest problem is that like outlines, mind maps don't scale well. It's hard to get a good overview of a ginormous mind map, and when all of my notes and quotes are added to a mind map, it gets pretty big. We're still only talking about a mind map for a single book, too. It doesn't even make sense to build a mind map with a book's worth of notes as one topic in a larger map of all of my blog ideas, so then I'm back to managing multiple maps for different things. Not ideal. Even a map of only my blog ideas doesn't work very well because they're all spread out on the map and scanning through them is not easy.

Mind maps were not working out, so I moved on to another project organization tool.

Basecamp

Basecamp.com aims to be a simple project management tool that allows you to collaborate with your team through discussions, to-do lists, and shared files and documents. It's a great tool for working on team projects, and I thought maybe it would also work well for managing blog ideas. The core useful feature in this case is the to-do list. As the only one working on the blog, I wouldn't have much use for the discussions, files, or documents.

The to-do list can actually be a number of different to-do lists, each with their own title. This format essentially creates a two-level outline where the titles could be the different topics I want to write about, and the bullets would be the main points for each topic. Each bullet point can also have a comment thread attached to it, which could be used to expound on the corresponding points, if need be.

You may have noticed that I'm talking about Basecamp as if I might use it, not that I did use it. That's true. I evaluated Basecamp and Trello together and decided to go with Trello, but Basecamp has a number of advantages over outlines and mind maps that make it a good option to consider, especially if you're working on a project with other people.

First, as an online app, Basecamp is available everywhere. It also has smartphone and tablet versions for easy project management on those devices. Second, it looks like it should scale pretty well. The to-do lists are structured to keep things short and sweet with links to separate comment pages, so lists of dozens of blog ideas should be manageable. Finally, it's dead simple. There aren't many extra features, so there's basically no learning curve and working with it is very responsive and intuitive. If I wasn't using Trello, I'd probably be using Basecamp.

Trello

Trello.com is a project management tool done in the spirit of agile software development. You can create a series of boards that correspond to projects, and on each board you can create cards that represent topics or tasks for that project. Each card can contain a description, colored labels, checklists, a due date, attachments, and comments. Cards are organized into columns with their own titles.

This system is incredibly flexible, and it fits perfectly with what I want to do with my blog ideas. Having the cards organized into columns makes a board equivalent to a two-level outline in a very compact format that's easy to reorganize. I can make columns for different categories of topics that I want to talk about, and I have one column on the far left labeled "Up Next" to hold the next topic that I'm preparing to write on. Within the other columns I can sort the cards so the higher priority topics are at the top, and I can instantly see which categories have the most topics ripe for developing. Dragging the cards around to organize them is a snap.

Looking at a board, I have a great overview of all of the ideas there, but I don't get overwhelmed with details. To drill down into one idea, I can click on that card to bring up all of the information on that idea, and I can review it, edit it, and add to it quickly and easily. I use the description to expand on the general idea in the title of the card. The checklists are handy for holding the main points I want to hit, and the comments work well for notes that don't fit well in the checklists. Links can be used anywhere, and they just work, making external references easily accessible.

If I'm reading a physical book instead of a Kindle book, I'll even take all of my notes on one card. Even if it is a Kindle book, I'll put a few main points I want to address in a checklist and then link to the notes and highlights page for that book on my Amazon account. Once I'm done writing about a topic, I can archive the card so it's off the board but still available if I need it. Seamless, I tell you.

Trello beautifully addresses my three main criteria for a great idea management tool. It's an online app with Android and iOS versions, so it's available everywhere. It's wonderfully scalable while staying compact—I could manage hundreds of ideas without getting overwhelmed. And it fits well with my organizational style, giving me a visual format that I can work with quickly and intuitively.

Trello works so well for my blog ideas that I've been using it to keep track of my other work and personal projects as well. It works equally well for tracking project features, with features grouped by those you need to start, those you're currently working on, those you need to test, and those that you may or may not do in the far future. I'm not sure how Trello would work for collaborating with a team. It was definitely built for that purpose, but it may not scale too well that way. It probably depends on how well it's used by the whole team. For personal tracking, though, Trello is a big win for me.

Outlines Vs. Mind Maps Vs. Basecamp Vs. Trello

In summary, there are many ways to manage a big collection of ideas. The tools I covered here are by no means the only options, but they represent a good cross section of what's out there. Outlines are a basic tool that everyone should know about. They can be done in any text editor or word processor, and can represent a set of ideas in a straightforward, linear way. However, editing and organization can become a burden when the number of ideas gets too large, and it's difficult to maintain a history of completed ideas. Mind maps are equivalent to outlines, but are represented in a more visual format that works well for spacial learners. Maps can require more work to create and maintain than an outline, and compared to other tools, they get too big to manage rather quickly.

Whereas outlines and mind maps are general tools with many different implementations—you can even use pen and paper to draw them out if need be—Basecamp and Trello are more specific tools built as dedicated applications. Basecamp is closer to an outline with its to-do lists, and it has additional features for collaboration. Trello is closer to a mind map, but is much more compact and easier to organize in comparison. Personally, I find Trello to be the easiest to use and the tool that best fits my style. It's been working great for me so far, and the more I use it, the better it gets. I wouldn't completely discount the other options, though. Each one might be the right choice in certain situations, and different solutions are going to work better for different people. The best thing to do is experiment and find out what works best for you.

Math Book Face Off: Everyday Calculus Vs. How Not To Be Wrong

I'm deviating slightly from my normal Tech Book Face Off. I think it's kind of fun, so expect more deviations like this in the future. It's been a long time since I've done some serious math studying, and I wanted to get started again by dipping my toes in the shallow end of the field. I also wanted to find a good, entertaining popular math book that deals with real-world applications of mathematics and the development of mathematical thinking. My books of choice for this venture were Everyday Calculus by Oscar E. Fernandez and How Not To Be Wrong by Jordan Ellenberg, both of which came out earlier this year. Let's see how they tackle the problem of speaking math to the masses.

VS.

Everyday Calculus

I was really expecting to enjoy this book. Unlike most high school kids, I enjoyed learning calculus. It just made sense to me. After all, derivatives are essentially subtraction taken to an extreme, and integrals are addition taken to an extreme. Things get complicated when you actually have to derive equations using limits or when you get into proving theorems in calculus, but that's true of math in general. The basic ideas of calculus are fairly straightforward.

Fernandez did a decent job of bringing that across, but something about the book didn't sit well with me. I thought his treatment of concepts was too superficial, and he never went beyond the trivial exploration of applications of calculus to everyday life. The general format of the book was to walk through a day in the life of Fernandez from waking up to going to bed and examine things like the sleep cycle, coffee, and television from the perspective of calculus. The concept was okay, but I just couldn't get into it. I wanted to see some more substantial analysis than what was there, and I thought that should have been possible while still keeping the book accessible to the wider audience that it was intended for.

As an example, in chapter 3 he shows how the derivative can be used when calculating the efficiency of driving a car with different rates of acceleration, but he assumed that the efficiency of the car itself was constant when calculating fuel consumption for a given trip. It would have been more interesting—and more realistic—to model the efficiency of the car dependent on acceleration and integrate over the resulting curve to figure out a more accurate fuel consumption for the trip. Then he could have showed how much of an impact the acceleration rate had on fuel economy. Granted, he didn't cover integration until later in the book, but still, he could have revisited it. I would have loved to see a more in depth analysis of that.

Some parts of the book did pique my interest. I thought the equation for sustainability analysis was pretty cool, and I enjoyed contemplating the questions that came to mind when Fernandez was describing how integration was developed. It took 2000 years to fully develop the concepts and structure of calculus, which makes it seem like it was very difficult to do. Indeed, the use of limits and infinitesimals was rejected as a valid method of calculation for a long time, and Isaac Newton had to overcome a lot of resistance to his ideas. Now calculus is routinely taught to millions of students around the world, and it's a basic requirement for many fields of study.

It's amazing to me that calculus has gone from a field that took two millennia to develop to the point that even a few people could understand and use it, to something that a significant portion of the population is expected to know. Are we getting smarter as a species, or is it more a matter of us collectively standing on the shoulders of giants? Is it inherently easier to learn something once it is already known? That's likely, considering that once an idea is discovered or developed, it can be shaped and refined until it is much more easily accessible to more people. It's also interesting to think that we're getting smarter, not that we're all smarter than Newton, but on average we might be smarter than the average person a thousand years ago. It would be hard to determine if such a trend had to do with anything more than better hygiene, nutrition, and education. It's still fun to think about, though.

In any case, Everyday Calculus had a few interesting nuggets, but overall I didn't really enjoy it. However, it was a quick read, and it did make me want to get out my old calculus and numerical methods textbooks, so it wasn't a total loss.

How Not To Be Wrong

Despite its pretentious title, this book was full of awesome. With chapter titles like "Everyone Is Obese" and "Dead Fish Don't Read Minds," I knew I was in for an entertaining read, and I was not disappointed. Ellenberg does a great job covering a number of real-world issues from a mathematical perspective, and clearly explains the logic and reasoning behind many of the mathematical methods that are used in analysis.

The book had a definite focus on probability and statistics, which makes a lot of sense when talking about how to make money off of a poorly designed lottery or how the link between smoking and lung cancer became undeniable as the evidence mounted against cigarettes. Ellenberg periodically brought other fields of mathematics into the discussion, such as linear algebra and non-Euclidean geometry, but he always circled back to statistics. It's fitting since statistics is probably the most applicable field of mathematics to our everyday life, and it's something everyone would benefit from knowing more about. So many of our personal experiences that guide our intuition can lead us to the wrong conclusions when we try to extrapolate them into broader contexts, and statistics provides the tools to correct our thinking.

For example, Ellenberg goes into a discussion on Big Data and how when you're filtering for some particular type of person that's a very small percentage of the population (his example is terrorists, but it could be any small group), it doesn't matter how accurate your filter is. Even if it's 99% accurate in filtering out people not in the group you want to detect, you're going to end up with a lot of false positives because 1% of hundreds of millions of people is still millions of people. Then he lays out the argument for the right to privacy:

You might well think that Facebook would never cook up a list of potential terrorists (or tax cheats, or pedophiles) or make the list public if they did. Why would they? Where's the money in it? Maybe that's right. But the NSA collects data on people in America, too, whether they're on Facebook or not. Unless you think they're recording the metadata of all our phone calls just so they can give cell phone companies good advice about where to build more signal towers, there's something like the red list going on. Big Data isn't magic, and it doesn't tell the feds who's a terrorist and who's not. But it doesn't have to be magic to generate long lists of people who are in some ways red-flagged, elevated-risk, "people of interest." Most of the people on those lists will have nothing to do with terrorism. How confident are you that you're not one of them?

It doesn't matter if innocent people should have nothing to hide, which is one of the arguments the people make that are trying to create these lists and provide Security For All. Innocent people should not live in fear that their government will wrongly implicate them in criminal activity, especially without their knowledge via these secret lists. It's not really improved security when a large number of citizens are in danger of wrongful incrimination by their government.

Ellenberg also talks eloquently about education in mathematics, both from his perspective as a professor teaching students the importance of practicing calculation and learning how to think mathematically, and his experience as a student learning about hard work:

The cult of the genius also tends to undervalue hard work. When I was starting out, I thought "hardworking" was a kind of veiled insult—something to say about a student when you can't honestly say they're smart. But the ability to work hard—to keep one's whole attention and energy focused on a problem, systematically turning it over and over and pushing at everything that looks like a crack, despite the lack of outward signs of progress—is not a skill everybody has. Psychologists nowadays call it "grit," and it's impossible to do math without it. It's easy to lose sight of the importance of work, because mathematical inspiration, when it finally does come, can feel effortless and instant. I remember the first theorem I ever proved; I was in college, working on my senior thesis, and I was completely stuck. One night I was at an editorial meeting of the campus literary magazine, drinking red wine and participating fitfully in the discussion of a somewhat boring short story, when all at once something turned over in my mind and I understood how to get past the block. No details, but it didn't matter; there was no doubt in my mind that the thing was done.

I can totally relate to this type of problem solving. I experience it all the time when I have a hard design problem I'm working on, and it's constantly turning over in the back of my mind. I call it bedtime debugging.

This passage also made me think about another aspect of studying that I pretty much avoided in high school and college. As I get older, I become more interested in the history of math and science, the lives of the great thinkers that discovered and developed the ideas we use today, and the difficulties involved in the process of discovery. This kind of knowledge gives context to the theorems and laws and ideas of math and science. Understanding where these ideas came from can give you a greater appreciation for the humanity involved in our development of knowledge, that it didn't all come about in the perfect form that it's presented in classrooms and textbooks. There was intense struggle, debate, and uncertainty behind it all, so you can take comfort in the fact that your own struggles are normal. The pursuit of knowledge has always been a battle.

Clearly, this book has made me think about some deep topics, and there were many more instances like these of discussions in the book sending me off thinking on wild tangents. I really enjoy books that do that, and Ellenberg was especially good at it. He's also an exceptional writer. He has a way of starting off a topic, developing it for a while, and then summarily dropping it on the floor to describe some other thing. At first you wonder what just happened, but you keep on reading because this new topic is also well written and interesting. Eventually, he'll pick the first topic up off the floor and tie it in with his current explanation, and everything becomes clear. Every once in a while he'll also drop little quips about old topics into the current discussion to keep you on your toes. It was completely engaging, and I burst out laughing a number of times while reading particularly witty parts. My wife probably thought I was crazy, laughing at a book about math like that. It truly was a treat to read.

Surrounded By Math

We are surrounded by math in our daily lives. It permeates everything we experience, and it can explain a lot about why things happen, if we're only willing to pay attention and understand. I went looking for a book that would capture the pervasiveness of mathematics in the real world, and I definitely found what I was looking for. While Everyday Calculus fell flat, and I couldn't get into it, How Not To Be Wrong delivered the goods. It has everything I want in a good popular math book. It's well written, engaging, and above all, it made me think hard on a number of topics. That always makes for a satisfying read. If you're in the mood to see how mathematics, and statistics in particular, shape our world and our understanding of it, go read How Not To Be Wrong. I can't recommend it enough.

Everyday Statistics for Programmers: Nonlinear Regression

Last week I talked about how to figure out if two variables in your data set are correlated, and the week before I talked about fitting a trend line to your data. What happens if you know your data is correlated, but the relationship doesn't look linear? Is there anything that can be done to estimate the trend in the data if it's not linear? Yes, depending on what the data looks like, we can either transform the data to make it linear, or do a polynomial regression on the data to fit a polynomial equation to it.

We'll take a closer look at data transformations, and then briefly cover polynomial regression. The idea with data transformations is to somehow make your data linear. There are a few data trends that this will work for, and luckily, these types of trends cover most of the nonlinear data that you're ever likely to see. These trends fall into the categories of exponential, power, logarithmic, and reciprocal relationships.

The exponential relationship is probably the most common of these, so lets go through an example of how to transform a set of data that exhibits an exponential trend. Say you've started a new website, and you're measuring the number of active users on your site each week. Your usage data might look something like this:

The linear trend line is there to show that the data is not really linear. It's kind of linear, but in the middle of the scatter plot the data bows below the line and at the end of the plot it looks to be taking off above the line. One would assume that to get a better idea of what might happen in the near future, using a linear trend line will underestimate the site's performance. To get a better idea, we can use linear regression, but first we want to transform the data to make it more linear. In this case that means taking the logarithm of the y-values of the data, which produces the following scatter plot of the transformed data:

The transformed data definitely looks more linear, as the trend line running right through the scatter plot shows. You can do a linear regression on it in exactly the same way as any other linear regression, ending up with the m (slope) and b (offset) coefficients. However, we want the coefficients for the original exponential relationship, which is of the form

y = α*exp(ß*x)

Since we took the logarithm of y, the form of the linear equation that we found the coefficients for is

ln(y) = ln(α) + ß*x

This looks just like the linear equation y = m*x + b, so we can conclude that α = exp(b) and ß = m. Now that we have the coefficients of the exponential trend line, we can plot it against the original data like so:

Pretty slick. The exponential trend line fits the data much better, and we can merrily be on our way, extrapolating future site growth (but not too far in the future, of course). One thing to be careful of, though. Notice how the data points get more spread out later in time? That's characteristic of many exponential trends, but not all. If the data points look evenly spread out over the entire range of the data (in statistical terms this would be stated as constant variance over the range), then transforming the data will cause the smaller values to be more heavily weighted in the linear regression. A more appropriate analysis to do in this case, if you need the additional accuracy, would be a true nonlinear regression using a nonlinear solver.

A similar procedure to the logarithmic transform for exponential trends works for a power relationship of the form

y = α*x^ß

The difference is that the logarithm of the x-values must also be calculated to give the data a linear form of

ln(y) = ln(α) + ß*ln(x)

The original coefficients are found the same way as they were for the exponential relationship, with α = exp(b) and ß = m. That wasn't too difficult; just a slight tweak was needed to the data transformation.

The last two forms of logarithmic and reciprocal relationships are easier to calculate because only the x-values need to be transformed with a ln(x) and a 1/x operation, respectively. The linear coefficients are the same between the transformed and original data, so once the linear regression is done on the transformed data, the work is pretty much done.

That's all well and good as far as nonlinear regression goes, but what happens if your data looks like this:

This data doesn't fit any of the relationships discussed so far. It's an example of a quadratic relationship that also shows up fairly often in different types of data, especially economic data. When you see data like this, you can assume that there is some kind of tradeoff involved, and you probably want to optimize for the area on the scatter plot where the data peaks. To figure out exactly where that peak is, you need to find an equation for the trend line, and to do that, you need polynomial regression.

Before we get into polynomial regression, also note that data with a quadratic relationship doesn't have to look like the scatter plot above. It can also be flipped and shifted so that it looks more like an exponential relationship. The data would have a growth characteristic in this case, instead of a trade-off characteristic. It's often hard to tell the difference between quadratic and exponential growth, especially if the data doesn't span enough time. A trend line might show a misfit if the data increases much faster than a quadratic function. In that case, the exponential fit is definitely the way to go. Otherwise, you'll have to resort to your knowledge of the domain and use your judgement as to which relationship better describes the data.

Cubic relationships can also show up in data from time to time, but they are much more rare. Having data with polynomial relationships larger than third order are more of a warning sign than anything. If you're trying to fit a tenth-order polynomial to your data, you may want to rethink your analysis or question your data. You may be moving out of the realm of statistics and into the realm of signal processing.

Getting back to the problem at hand, how do we find the best fit quadratic curve for the data set shown above? Basically, you can construct a linear equation with matrices and solve for the coefficients. That actually makes polynomial regression a form of linear regression, but it's done on nonlinear data. The math is more involved than we'll get into here, but there is a nice implementation of the algorithm in Ruby that I found on a great programming algorithm website, called rosettacode.org. Here's the code adapted to the Statistics module I've been building:

require 'matrix'

module Statistics
  def self.polyfit(x, y, degree)
    x_data = x.map { |xi| (0..degree).map { |pow| (xi**pow).to_f } }
 
    mx = Matrix[*x_data]
    my = Matrix.column_vector(y)
 
    ((mx.t * mx).inv * mx.t * my).t.to_a[0]
  end
end

The first three lines of the polyfit() method are building up the X and Y matrices, and then the last line of the method does the actual calculation to produce the coefficients. If you call this method with the (x,y) data from the scatter plot above and set the degree to 2, you'll get the three coefficients out for the equation:

y = a*x^2 + b*x + c

If you then plot this quadratic equation as the trend line for the scatter plot, you get a nice fit of the data:

That wraps up the everyday ways of doing regression on nonlinear data. Doing nonlinear regression with transformations on the data or using a polynomial regression should cover the vast majority of nonlinear data that you would come across, and the analysis is fairly straightforward once you decide on which type of curve to fit.

I hope you've enjoyed this miniseries on statistics, and that it proves useful for your everyday data analysis. Happy estimating!

Everyday Statistics for Programmers: Correlation

Last week I went through the basics of linear regression, and I touched on correlation but didn't get into it. That's because correlation may seem fairly simple, but the simplicity of the calculation disguises a vast field of complexity and logical traps that even professional researchers, statisticians, and economists routinely fall into. I'm not claiming to know more than these people that live and breathe correlation every day, but the issues are well known. Knowing the issues involved can raise our general awareness of when certain conclusions are warranted and when they're going too far.

I Think, Therefore I Correlate

One of the reasons why correlation is so deceptive is that we see correlations all the time. A correlation is basically a relationship between two things. If two separate things grow or increase or change at the same time, they are correlated. If one thing increases and another thing decreases at the same time, they are also correlated.

Our pattern-matching brains are wired to detect this behavior. It's how we've survived and developed throughout our history. When our distant ancestors would eat something and then get sick, they recognized that they shouldn't eat that plant or animal anymore. When they would eat or drink certain things when they were sick, or they would apply different plants to wounds, and then get better, they recognized that as well and passed on that information as early medicine. When they started planting seeds and developed early agriculture, they paid attention to what practices would result in better crop yields. All of these things are correlations, and more often than not, seeing those correlations could mean the difference between life and death. It's ingrained in us.

Even though seeing correlations is a good thing, it's also very easy to develop false correlations. Our ancestors used to believe comets were bad omens. The coming of a comet meant that something terrible was about to happen. Many ancient cultures believed (and some still do) that various traditions, like dancing, could bring the rains so their crops would grow. There are innumerable examples of other false correlations different cultures have believed at different times, and plenty survive to this day. We even collect them as old wives' tales and pass them on half-jokingly to future generations.

Have you ever heard the joke about the guy who was walking down the street in a big city, flapping his arms wildly up and down? Another man walks up to him and asks why he's doing that. The first guy, still flapping his arms, says, "To keep the alligators away." The second guy says, "But there aren't any alligators here." The first guy responds, "See, it's working!" That's false correlation.

These false correlations come about when two things happen at the same time, and someone attaches too much significance to the fact that those two events just happened to coincide. The person witnessing the events doesn't realize that there could be some other explanation for what they experienced, and he goes on to convince enough other people that these two things are related that it becomes common knowledge.

Correlate Wisely

When we're looking at data and calculating correlations, or even reading studies done by others, we need to be careful to not be fooled. It's harder than it sounds. Correlations are easy to calculate, and many things can seem correlated when they're not, either by coincidence or because something else that you're not paying attention to is systematically causing the correlation that you're observing. It may be all the more difficult to recognize a false correlation because you want to believe that it's there because of internal biases or external influences. On top of that, your brain is wired to see the pattern even if it's not the right explanation, and you have to fight against this instinct to get to the truth.
You should first decide whether the correlation makes any sense at all. Is there a plausible mechanism that could explain the connection? Are the two things related in any way that would make the correlation reasonable? Is there anything that would directly contradict the correlation that you're seeing? Is it extraordinary that these two things are correlated? Remember, extraordinary claims require extraordinary evidence.

If the correlation is plausible, then you might start thinking about which of the variables in the correlation is dependent on the other. While you can do this, you must be careful because this line of thought leads to another big way that correlations are used incorrectly. If you've read enough studies that use statistics, sooner or later you'll hear the phrase, "correlation does not imply causation." There are other analysis tools that can get closer to proving the claim that A causes B, but correlation is not one of them. All correlation says is that A and B are related in some way, nothing more. To make stronger claims about dependence, more work will be involved.

It may seem obvious that one thing causes the other in the system you're studying, but it may still be possible for causation to run in the other direction or an entirely different cause may explain why both of the variables being measured are moving together. One recent, high-profile example of this issue occurred after the financial crisis of 2008. A couple of economists, Carmen Reinhart and Kenneth Rogoff, set out to measure the relationship between government debt and GDP growth. They found a negative correlation, meaning that as government debt went up, a country's GDP growth tended to go down, and they concluded that government debt was causing the slow down in GDP growth.

It turned out that there were some errors in their data and calculations, so the correlation was actually much weaker than they thought, but even before that there was a big debate about which way causation actually ran. It's fairly easy to argue that for many of the countries in the study, they suffered lower GDP growth or even negative growth that drove up their government debts, instead of the other way around. In reality causation ran in both directions, and it was highly context dependent by country. Some countries started with high government debt before going into recession, and others went into recessions that ballooned their debt. Correlation couldn't show any of these details.

Correlation In Practice

The ease with which correlation analysis can go wrong doesn't mean correlation isn't useful. It is very useful as a starting point for further analysis. Correlations can show if a relationship actually exists so you can investigate it further, or if you're barking up the wrong tree and need to look elsewhere. So how do we calculate a correlation, given a set of data with X-values and Y-values for two different variables that we want to compare? Here it is, implemented in Ruby once again:

module Statistics
  def self.r(x, y)
    s_xy(x, y) / (Math.sqrt(s_xx(x))*Math.sqrt(s_xx(y)))
  end
end

I defined the methods s_xy() and s_xx() last week for calculating the r-squared value of a regression analysis, so at this point we're getting pretty far along in statistical equations, using earlier calculations to construct new equations. In fact, this correlation coefficient is closely related to the r-squared value. All you have to do is square it and that's what you get, which is why it's referred to as the r value in the code above.

Both the correlation coefficient and the r-squared value give an indication of how strongly linear a data set is, but the correlation coefficient has one property that the r-squared value doesn't have. It can be negative, and a negative correlation coefficient happens when one variable increases while the other decreases, i.e. the scatter plot shows a negative-trending slope. So if the correlation is +1 or -1, the data is perfectly linear with a positive or negative slope, respectively.

If the correlation is close to zero, then the data is either uncorrelated or it might have a nonlinear relationship—you have to check the scatter plot to decide. The closer the correlation is to +/-1, the more linear the relationship is, but to get a better idea of that linearity, you should square the correlation coefficient. This operation gives you the r-squared value, which shows the percentage of the variation in the data that is due to linearity.

Go Forth And Correlate

If you take away anything from this discussion, remember that correlation is quite useful if used wisely, but it is also incredibly easy to misuse. Correlation does not imply causation, no matter how much you want to believe that causation runs a certain way in your data. Unless there is a known mechanism that explains why one variable depends on the other, you need to perform other analyses before you can claim causation. If your data has a time component to it, there are a number of time series analyses and cross-correlation analyses that could uncover more details in the data related to causation. Other types of data require other more advanced statistical techniques to figure out causation. In any case, examining the correlation of the data is a good place to start, and will generally point you in the right direction.

We're getting close to the end of this miniseries on statistical tools. Next week I'll wrap up with a couple ways to analyze data that's not linear, but looks like it could fit some other type of curve.