Who hasn't wished for a library that neatly solves the programming problem that is currently staring you in the face? When you're down in the dirt, gearing up to tackle some gnarly problem that seems like it should have a general solution, it's only natural to wish, or even expect there to be a library that addresses your need. Once you have a workable solution that you think is elegant and useful, don't you think about releasing it for the world, or at least your company, to fulfill other programmer's wishes for self-contained solutions to their problems? Be careful what you wish for.
This post is probably not going to be about what you're thinking right now. I'm not going to relate some harrowing tale about a software library I picked up off the street, hoping it was going to solve all of my problems, only to find that it was more trouble than it was worth. Honestly, I haven't had many experiences like that. Either I've had amazing luck finding exactly what I need, or I have low expectations and work through the problems with the libraries I find as they come up. No, this post is about the way the software library marketplace has grown exponentially over the years and how one type of complaint about libraries has morphed into another, seemingly without notice.
When I was first learning to program, the languages I learned had small standard libraries. C has a pretty manageable standard library. C++ also has a small standard library by today's standards, even though it includes most of the C standard library as well. You could also include the standard template library, still keep all of the C++ library functionality in your head, and recall it at will to solve your programming problems. Early Java also had a tractable standard library.
Admittedly, my early view of the world of programming languages was somewhat limited, but the world was definitely smaller back then. One of the most common complaints I heard was that the world was a little too small. We needed more good libraries and frameworks for building applications—prepackaged plug-and-play code that would solve all of our problems and make development easy and painless. Sure, there were some isolated examples of application development frameworks out there, like MFC for Windows development in C++. But we wanted more, much more, and we weren't afraid to ask for it.
Now I'm not in any way knocking the fact that we wished for all of these libraries and frameworks, or that we expected them to solve all of our problems. It was and is entirely reasonable, even virtuous, to not want to reinvent the wheel over and over again. We should package up solutions to solved problems and make them part of the foundation that we then build on top of. I wouldn't have expected or wanted it to happen any other way, but I don't think many programmers realized that it would happen the way it did.
Today Java 8 has over 4,200 classes in the standard library public API. The .NET 3.5 framework has nearly 10,000 public classes—the latest version I could find numbers for. The .NET 4.5 framework has well over that number, I'm sure. No single person could possibly know everything that's in the standard libraries of these languages, and that's not even counting all of the contributions available from open source libraries.
It's a bit harder to get stats on open source projects. Ruby, a language with a fairly tractable standard library, has 6,272 gems available on RubyGems.org. Take into account that gems are generally a larger unit of measure than classes and the gems available at RubyGems.org are only a subset of all available gems, and it is clear that there are a lot of open source libraries available for Ruby. The same goes for all of the currently popular languages, especially JavaScript. Because of the nature of JavaScript—ease of releasing projects and use on both the client and server sides of the browser—the number of libraries available is becoming truly astounding. Looking only at GitHub, there are over 320,000 active JavaScript repositories! Of course some (possibly many) of these repositories are mis-categorized, and many more of them aren't actually reusable libraries, but other varieties of projects. Even so, any way you slice that number, it's big, and growing fast.
The point here is not to get an accurate measure of the number of libraries available for different languages, but to show that in any language you use, that number is absolutely massive. The game has fundamentally changed from knowing everything that's available and how to use most of it to searching for what you need when you need it and evaluating whether or not you can use it (90% of it is crap, as per Sturgeon's Law). This shift makes some people very uncomfortable.
You see, back when we were complaining about the lack of good libraries and frameworks for building awesome applications and solving all of the mundane problems of software development, this is not what we wanted. We wanted elegant, high-performance tools that perfectly fit the problems we were trying to solve, not a mountain of crap to sift through for a few choice pearls. I posit that the former was never going to happen, and the latter is the best we could have hoped for.
Part of the reason is purely because that's the nature of software development and the web. When anyone can post a library on the web for others to use, you're going to end up with a ton of stuff that nobody needs or is of decidedly low quality. To get the fraction of really good stuff, we need to put up with a lot of mediocre, useless stuff.
The other part of the reason is that perfect software libraries are one of Steve Yegge's Nonesuch Beasts. They don't exist. You can't build one. It's impossible, and here's why. Any library that's generally applicable to a wide range of projects necessarily needs to solve problems in a general way. On the other side of the coin, you likely require a specific solution to a narrow problem, and a large collection of projects like yours will have a wide range of narrow problems that are slightly different.
What we end up with is a large problem set (the projects) on one side and a large solution set (the libraries) on the other side with a lot of thin connections running between them. One project will use a part of one library in a particular way, and another project will use a the same library in a different way or a different library altogether. There's an enormous mismatch, and you can't have both a general solution for many problems and a perfect solution for each problem. Abstractions are leaky. Although we can keep making things better, and I think we have much better tools than we did a decade ago, that mismatch will always exist. Ruby on Rails is one example among many excellent web development frameworks. Python and R have some great data analysis tools. JavaScript continues to impress me with the libraries available for adding all kinds of features to your web application. The solutions are there if we know where to look and how to evaluate them.
When programmers complain about how hard it is to find the right library for their needs and how everything out there is crap, I think they're actually complaining about needing to think. What they want is to be able to go on Google and type in, "I need a JavaScript library for x," be taken to a GitHub page with an easy-to-install library, and be able to use the library intuitively. No reading documentation, comparing options, or analyzing performance required. Sometimes we can get pretty close to this experience. Some libraries are very good and the choice is clear, but most of the time software development is still hard. The choices are vague and complicated. Nothing may be the right fit, and we need to develop our own solution. Programming still requires plenty of thought.
I count that as a good thing. If programming was simple, and all we had to do was plug together a bunch of libraries we found on GitHub, it would get unbearably boring. Besides, if we truly solved the software library problem, it would mean we were no longer innovating and pushing the frontier of software. Researching options and piecing them together to build novel solutions, basically creative problem solving, is an inherent part of our job. It is fundamental to what we do. Without that difficulty, most of us would probably be out of a job. We wished for more and better libraries. Now we're getting them, and we're wishing that we didn't have to think so hard about when and how to use them. We should be careful what we wish for. What if we actually get it?
Face It, You Can't Predict the Future
We programmers are a funny bunch. We think we can predict the future. No, really, we do. Every time we add a layer of abstraction or a level of indirection or a complicated inheritance hierarchy just in case, we're trying to predict the future. Those three words—just in case—should be closely associated with two other words in our minds—make work. In reality, that's what we're doing most of the time. We're making work for ourselves instead of solving the problem at hand and finishing the work that needs to be done.
I fall into this trap of trying to predict the future all the time. I have to stop myself when my mind goes off on tangents, dreaming up Rube Goldberg architectures to handle all kinds of imagined circumstances that may or may not ever happen. Sometimes it's easier to architect than to work on edge cases. Sometimes it's more interesting to rough out structures for potential future issues than to implement the nuts and bolts of features that need to get done now. Sometimes it's fun to over-engineer and over-think a problem.
Other times we need to get stuff done. During those times (honestly, it's most of the time, isn't it?) how do we stay on track instead of heading off into the over-engineered weeds?
For starters, if you're writing some extra structure into your code, just in case you need it, don't. You should be able to hear yourself saying it. "What if we need to support more than one type of gizmo that talks to the thingamajig? I'll make a thingamajig-gizmo bridge interface so that we can easily implement many types of gizmos, just in case." You better be sure that either the number of gizmo types is really more than one (and you will definitely use more than one as part of the requirements of the product), or the use of the gizmo will be strewn over so much of the code that it creates substantial risk should you ever need to replace it. If neither of these things is true, maybe it's better to put your head down and implement the gizmo instead, just in case you want to ship your software.
It's worth it to carefully consider your options any time you want to abstract away part of your system, especially the core parts of your system. Will you really ever switch views? databases? libraries? Ask yourself, are you making an all-singing all-dancing platform, or are you making a targeted product that real people are going to use to solve real problems, hopefully soon.
Maybe you think you are making a platform, but remember, most of today's platforms didn't start out that way. They started out as products that people found useful and eventually grew into platforms. You can't address every single concern up front, and if you try, you'll never ship. You'll be spending all kinds of time working on abstract interfaces to loosely hook up all of your code to…um…save time. I'm not sure how that works, exactly. Only a few of those abstractions will be needed. Pick a view. Pick a database. Pick your libraries. Make some decisions and ship your product.
Speaking of platforms, platforms need the feature of being able to scale. 'Scale' can mean a lot of different things. It can mean millions of users per month. It can mean tens of thousands of transactions per day. It can mean thousands of requests per second. For a team that's still working on their product, it mostly means the load the software should be able to support, just in case it gets big fast. Face it, not everyone is Twitter or Slack. Not even Twitter was Twitter when they started, and they had all kinds of scaling problems as they got huge. It didn't matter, and they dealt with the scaling problems in real time.
If you're engineering your software to scale before you need to, don't. There are too many things you should be doing to make your product better so that you actually have problems with scaling later. Don't waste time on scaling before it's a problem. If you waste enough time, scaling may never be a problem because your product never ships.
So far I've been talking about programming in the large, but let's turn to programming in the small. Programming in the large is all about architecture and scaling. Programming in the small is all about refactoring and optimizing. Refactoring is another one of those slippery words like scaling. It can mean whatever you want it to mean, so everyone loves it and loves doing it. I love refactoring, too. What does it mean? I'm not sure, but I love it.
I think it means change. I try not to change code unless it markedly improves its readability or performance, but mostly readability. When writing code, the primary goal should be to make it as clear and concise as possible. A piece of code should express its intent as obviously as the language allows. I find that clear, concise code also performs better than verbose, flexible code most of the time. Making code shorter tends to make it faster, to a point. And 90% of the time shorter code has good enough performance while being more clear.
Another reason why performance isn't an issue for most code, and you can get away with focusing on clarity, is that most code doesn't need to run fast. The bulk of most code bases is there to setup the system, provide configuration options, or is I/O bound, waiting for user input or slower processes.
One of my previous projects was writing code for an embedded system that did real-time signal processing at a sample rate of up to 1 MHz for four input channels. Functions could be attached to each of the channels for calculating things like running averages, frequencies, and peak-to-peak amplitudes. The more functions that could run simultaneously, the better. Certain sections of code in this system had to run as fast as they possibly could, so those sections were written in the specialized assembly language of the DSP processor we used. To make this code run even faster, a lot of things were set up and calculated ahead of time so that as few things as possible needed to be recalculated during run time. All of this setup and configuration code didn't need to be fast, so it was written as clearly as possible. That made it much easier to change as requirements changed, and I could implement new features within days of them being proposed because I could look at most of the code and immediately understand how it worked.
If performance is a problem and if you have measured and found where the code is slow, then, and only then, should you optimize the code in question. Since the code is already as clear and concise as you can make it, optimizing will mean making the code more complicated in some way. Maybe you need to use a more complex algorithm. Maybe you need to use more obfuscated low-level functions or assembly language. Maybe you need to add caching or memoization that muddies up the code. That's why optimizations should only be done where they're necessary, and the goal should still be to make the optimized code as clear as possible.
Beyond premature optimizations, we can also have premature refactorings in the name of DRY (Don't Repeat Yourself). DRY is a fundamental principle of good software development from Andy Hunt and David Thomas in The Pragmatic Programmer. (Excellent book. You should read it, if you haven't already.) But the DRY principle can be abused if you try to compartmentalize every operation into its own function, just in case that operation may be used again in some other place.
The problem with making every operation a function is twofold. First, you probably can't predict how the operation will be subtly different in other cases, so the function interface will likely be wrong. Second, doing this fractures your code base to the point where it's hard to find or understand anything. It actually makes it more likely that you'll have code duplication because you'll have hundreds of tiny functions that all do similar things, and you don't even realize it because you can't remember what they all are. If the original code adheres to DRY, leave it alone until code duplication actually occurs. Once it does occur, then refactor.
Obviously, the over-arching theme here is don't do something just in case it may be required in the future. If you do things just in case, you'll have spent huge amounts of time writing code that will mostly go unused. That time would be better spent thinking more deeply about the real problems and writing code to address those problems. Make sure the code is clear and concise and the architecture closely fits the problem you're currently trying to solve. Wait to scale until scaling becomes the problem you need to solve, and only optimize the code that needs to run fast. Face it, you can't predict the future. Instead, deal with the present.
I fall into this trap of trying to predict the future all the time. I have to stop myself when my mind goes off on tangents, dreaming up Rube Goldberg architectures to handle all kinds of imagined circumstances that may or may not ever happen. Sometimes it's easier to architect than to work on edge cases. Sometimes it's more interesting to rough out structures for potential future issues than to implement the nuts and bolts of features that need to get done now. Sometimes it's fun to over-engineer and over-think a problem.
Other times we need to get stuff done. During those times (honestly, it's most of the time, isn't it?) how do we stay on track instead of heading off into the over-engineered weeds?
For starters, if you're writing some extra structure into your code, just in case you need it, don't. You should be able to hear yourself saying it. "What if we need to support more than one type of gizmo that talks to the thingamajig? I'll make a thingamajig-gizmo bridge interface so that we can easily implement many types of gizmos, just in case." You better be sure that either the number of gizmo types is really more than one (and you will definitely use more than one as part of the requirements of the product), or the use of the gizmo will be strewn over so much of the code that it creates substantial risk should you ever need to replace it. If neither of these things is true, maybe it's better to put your head down and implement the gizmo instead, just in case you want to ship your software.
It's worth it to carefully consider your options any time you want to abstract away part of your system, especially the core parts of your system. Will you really ever switch views? databases? libraries? Ask yourself, are you making an all-singing all-dancing platform, or are you making a targeted product that real people are going to use to solve real problems, hopefully soon.
Maybe you think you are making a platform, but remember, most of today's platforms didn't start out that way. They started out as products that people found useful and eventually grew into platforms. You can't address every single concern up front, and if you try, you'll never ship. You'll be spending all kinds of time working on abstract interfaces to loosely hook up all of your code to…um…save time. I'm not sure how that works, exactly. Only a few of those abstractions will be needed. Pick a view. Pick a database. Pick your libraries. Make some decisions and ship your product.
Speaking of platforms, platforms need the feature of being able to scale. 'Scale' can mean a lot of different things. It can mean millions of users per month. It can mean tens of thousands of transactions per day. It can mean thousands of requests per second. For a team that's still working on their product, it mostly means the load the software should be able to support, just in case it gets big fast. Face it, not everyone is Twitter or Slack. Not even Twitter was Twitter when they started, and they had all kinds of scaling problems as they got huge. It didn't matter, and they dealt with the scaling problems in real time.
If you're engineering your software to scale before you need to, don't. There are too many things you should be doing to make your product better so that you actually have problems with scaling later. Don't waste time on scaling before it's a problem. If you waste enough time, scaling may never be a problem because your product never ships.
So far I've been talking about programming in the large, but let's turn to programming in the small. Programming in the large is all about architecture and scaling. Programming in the small is all about refactoring and optimizing. Refactoring is another one of those slippery words like scaling. It can mean whatever you want it to mean, so everyone loves it and loves doing it. I love refactoring, too. What does it mean? I'm not sure, but I love it.
I think it means change. I try not to change code unless it markedly improves its readability or performance, but mostly readability. When writing code, the primary goal should be to make it as clear and concise as possible. A piece of code should express its intent as obviously as the language allows. I find that clear, concise code also performs better than verbose, flexible code most of the time. Making code shorter tends to make it faster, to a point. And 90% of the time shorter code has good enough performance while being more clear.
Another reason why performance isn't an issue for most code, and you can get away with focusing on clarity, is that most code doesn't need to run fast. The bulk of most code bases is there to setup the system, provide configuration options, or is I/O bound, waiting for user input or slower processes.
One of my previous projects was writing code for an embedded system that did real-time signal processing at a sample rate of up to 1 MHz for four input channels. Functions could be attached to each of the channels for calculating things like running averages, frequencies, and peak-to-peak amplitudes. The more functions that could run simultaneously, the better. Certain sections of code in this system had to run as fast as they possibly could, so those sections were written in the specialized assembly language of the DSP processor we used. To make this code run even faster, a lot of things were set up and calculated ahead of time so that as few things as possible needed to be recalculated during run time. All of this setup and configuration code didn't need to be fast, so it was written as clearly as possible. That made it much easier to change as requirements changed, and I could implement new features within days of them being proposed because I could look at most of the code and immediately understand how it worked.
If performance is a problem and if you have measured and found where the code is slow, then, and only then, should you optimize the code in question. Since the code is already as clear and concise as you can make it, optimizing will mean making the code more complicated in some way. Maybe you need to use a more complex algorithm. Maybe you need to use more obfuscated low-level functions or assembly language. Maybe you need to add caching or memoization that muddies up the code. That's why optimizations should only be done where they're necessary, and the goal should still be to make the optimized code as clear as possible.
Beyond premature optimizations, we can also have premature refactorings in the name of DRY (Don't Repeat Yourself). DRY is a fundamental principle of good software development from Andy Hunt and David Thomas in The Pragmatic Programmer. (Excellent book. You should read it, if you haven't already.) But the DRY principle can be abused if you try to compartmentalize every operation into its own function, just in case that operation may be used again in some other place.
The problem with making every operation a function is twofold. First, you probably can't predict how the operation will be subtly different in other cases, so the function interface will likely be wrong. Second, doing this fractures your code base to the point where it's hard to find or understand anything. It actually makes it more likely that you'll have code duplication because you'll have hundreds of tiny functions that all do similar things, and you don't even realize it because you can't remember what they all are. If the original code adheres to DRY, leave it alone until code duplication actually occurs. Once it does occur, then refactor.
Obviously, the over-arching theme here is don't do something just in case it may be required in the future. If you do things just in case, you'll have spent huge amounts of time writing code that will mostly go unused. That time would be better spent thinking more deeply about the real problems and writing code to address those problems. Make sure the code is clear and concise and the architecture closely fits the problem you're currently trying to solve. Wait to scale until scaling becomes the problem you need to solve, and only optimize the code that needs to run fast. Face it, you can't predict the future. Instead, deal with the present.
Don't Fool Yourself
The first principle is that you must not fool yourself and you are the easiest person to fool. -Richard P. FeynmanHave you ever fooled yourself? Maybe a better question is, do you remember the last time you fooled yourself? You've worked all day or all week on a problem, and you think you've come up with a reasonable, if not elegant, solution. You show it to someone you trust—a coworker or a friend maybe—and she immediately sees the flaw in your logic or proposes a dead simple alternative that you never thought of. Face-palm.
I fool myself all the time, and I'm not ashamed to admit it. Those face-palms are the times when real learning happens, if you're willing to examine where you went off track and why the better solution works. Then you can generalize it to other problems. I came across one fascinating post by Ivan Yurchenko a few months back that perfectly illustrates how you can fool yourself on multiple levels when you're excited about something and want it to be more than it is.
The post is about calculating the Nth Fibonacci Number in O(log n) time, and it caught my attention because I thought that was impossible. To be clear, I don't want anyone to think I'm trying to take down Ivan or his post in any way. That's not the intent here. I enjoyed reading his post, and it made me think hard about algorithms, benchmarking, and optimization. It's a great post to learn from, and I'm sharing some of the insights I had. Getting back to Fibonacci Numbers, why did the claim of O(log n) time bother me?
Question Your Assumptions
The Fibonacci Numbers are a common sequence of numbers that are generated by starting with 0 and 1 (or 1 and 1, depending on context and personal preference) and continually adding the last two numbers to get the next number in the sequence. The first 10 Fibonacci Numbers are
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...Using a loop, we can write a simple program that calculates the Nth Fibonacci Number. Ivan does this, and here is his version in Python:
class IterationFibonacci:
def __init__(self):
pass
def get_number(self, n):
"""Get nth Fibonacci number (n is int, non-negative)."""
if n == 0:
return 0
a = 0
b = 1
c = 1
for i in range(n-1):
c = a + b
a = b
b = c
return c
where
is the golden ratio, and
Normally, a closed form expression signals that we can compute a result in constant time, but take a closer look at that expression. The n for the Nth Fibonacci Number that we're trying to find is an exponent in two places, and φ and ψ are irrational numbers. Taking the nth power of a number requires n-1 multiplications. That makes the time complexity of this closed-form expression O(n). We could be smarter about the multiplications and split them up into a binary tree of multiplications. That would make the time complexity O(log n).
Can we do better? How about using a power function provided by a standard library? It turns out that the run time complexity of a power function is dependent on the precision that you need in the result, and once you start looking at the precision that you need for Fibonacci Numbers, you start to see the real problem that we're dealing with. The Nth Fibonacci Number will have on the order of n bits in binary form. The 100th number has 69 bits. The 1,000th number has 694 bits. the 10,000th number has 6,942 bits.
It takes a little while for the series to get going, but after a while every subsequent number requires another bit to calculate and store it. That growth means that a power function needs n bits of precision to compute the Nth Fibonacci Number exactly, so it will have O(n) time complexity, assuming you already know the golden ratio to the required number of bits outright. That's the best we can do. Returning to the simple iterative algorithm above, the addition inside the for loop doesn't take constant time. It takes O(n) time because there are O(n) bits to add together. That was the incorrect assumption we were making, and the algorithm actually has O(n^2) time complexity.
That brings us to the matrix multiplication algorithm that Ivan presented from The Art of Computer Programming. It only has O(log n) complexity if you assume that all multiplies can be done in constant time, but the numbers being multiplied are on the order of O(n) bits long so the complexity is actually O(n log n). Knowing that calculating Fibonacci Numbers means working with ginormous numbers is a good thing to keep in mind when figuring out the complexity of a real implementation. The theory may be making assumptions that can't be met on a physical computer.
Pay Attention to What You're Testing
Moving on from complexity theory to benchmarking, Ivan compared the simple iterative algorithm above to the matrix multiplication algorithm, and the results showed that the run time of the iterative algorithm grew linearly and the matrix multiplication algorithm run time appeared to stay constant. There were a couple wrinkles in his tests, though.
The test procedure consisted of doing test runs of each algorithm, calculating 10,000 Fibonacci Numbers around n where n was chosen pseudo-randomly in the range n±5. He did these 10,000-iteration runs over the range of n from 10 to 1010 in increments of 10. Here's the code:
def benchmark():
random.seed()
for around in range(10, 1010, 10):
ifib = IterationFibonacci()
mfib = MatrixFibonacci()
ti = 0
tm = 0
count = 10000
for _ in range(count):
n = random.randint(around-5, around+5)
t = time.time()
ifib.get_number(n)
t1 = time.time() - t
ti += t1
t = time.time()
mfib.get_number(n)
t2 = time.time() - t
tm += t2
print("{0}\t{1}\t{2}".format(around, ti, tm))The second issue arises when we look at how the matrix algorithm is implemented. Another optimization was added to the matrix algorithm to speed it up more. In the part of the algorithm that decomposes n into powers of 2 and calculates the matrices for those powers of 2, the matrices are calculated once and saved in a hash table so that they can be recalled the next time they're needed instead of calculating them again. This optimization is called memoization, and it muddies the waters a bit because we're no longer comparing two algorithms, we're comparing one algorithm without an optimization and another one with an optimization. Here's the matrix algorithm with memoization:
class MatrixFibonacci:
A = [[1, 1],
[1, 0]]
def __init__(self):
self.__memo = {}
def __multiply_matrices(self, M1, M2):
"""2x2 matrices multiplication"""
a11 = M1[0][0]*M2[0][0] + M1[0][1]*M2[1][0]
a12 = M1[0][0]*M2[0][1] + M1[0][1]*M2[1][1]
a21 = M1[1][0]*M2[0][0] + M1[1][1]*M2[1][0]
a22 = M1[1][0]*M2[0][1] + M1[1][1]*M2[1][1]
r = [[a11, a12], [a21, a22]]
return r
def __get_matrix_power(self, M, p):
"""Calculate matrix power (p is a power of 2)"""
if p == 1:
return M
if p in self.__memo:
return self.__memo[p]
K = self.__get_matrix_power(M, int(p/2))
R = self.__multiply_matrices(K, K)
self.__memo[p] = R
return R
def get_number(self, n):
"""Get nth Fibonacci number (n is int, non-negative)."""
if n == 0:
return 0
if n == 1:
return 1
# Decompose to powers of 2,
# i.e. 62 = 2^5 + 2^4 + 2^3 + 2^2 + 2^0 = 32 + 16 + 8 + 4 + 1.
powers = [int(pow(2, b))
for (b, d) in enumerate(reversed(bin(n-1)[2:])) if d == '1']
# Less pythonic: http://pastebin.com/h8cKDkHX
matrices = [self.__get_matrix_power(MatrixFibonacci.A, p)
for p in powers]
while len(matrices) > 1:
M1 = matrices.pop()
M2 = matrices.pop()
R = self.__multiply_matrices(M1, M2)
matrices.append(R)
return matrices[0][0][0]To compare the two algorithms directly, we can remove the memoization from the matrix algorithm. Even so, running up to n=1,000 doesn't show a substantial difference between the two algorithms. To get a better sense of their difference, we can run out to n=10,000, which results in run times like this:
Keep Looking for a Simpler Way
If a solution seems complicated, it probably is. There is often a simpler approach that will accomplish the same goal more clearly and in less code. The above matrix algorithm is pretty complicated, and a few of the commenters on the original post came up with some nice improvements. One that I particularly liked from Juan Lopes was dramatically shorter, and it's reproduced here with a few edits to put it in the same format as the previous algorithms for running in the test bench:
class MatrixFibonacciShort:
def __init__(self):
pass
def mul(self, A, B):
return (A[0]*B[0] + A[1]*B[2], A[0]*B[1] + A[1]*B[3],
A[2]*B[0] + A[3]*B[2], A[2]*B[1] + A[3]*B[3])
def fib(self, n):
if n==1: return (1, 1, 1, 0)
A = self.fib(n>>1)
A = self.mul(A, A)
if n&1: A = self.mul(A, self.fib(1))
return A
def get_number(self, n):
if n==0: return 0
return self.fib(n)[1]
I run into this problem all the time where I think long and hard about the best way to express some algorithm only to have someone else (or a Google search) show me a much more elegant solution. It's always great to learn about another clever way to solve a problem, and over time these solutions build up your repertoire. But every time I'm attacking a new problem—and it's good practice to always spend some time reasoning about solutions yourself—I basically assume that whatever solution I come up with, I'll find something better on the internet.
It is so true that the easiest person to fool is yourself, and these mindsets are a good way to prevent at least some foolishness. First, always question your assumptions. Assumptions are the blind spots that hide obvious mistakes. Second, make sure that when you're comparing ideas, you know exactly what you're comparing and you are comparing exactly what you should to reach a valid conclusion. Finally, assume that whatever solution you come up with, a better one exists, especially if your current solution seems too complicated. Then you have to decide if further optimizations are worth pursuing, or if your current solution is good enough.
Tech Book Face Off: Growing Object-Oriented Software, Guided by Tests Vs. Agile Testing
It's been nearly eight months since I've done a Tech Book Face Off. That is way too long. Granted, I had been reading plenty of other stuff—math books, physics books, and econ books—but it's high time that I got back to some software development books. One area where I thought there was still somewhat of a hole in my understanding was automated testing, so I picked out a couple books on testing that I thought would be good: Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce and Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and Janet Gregory.
I'm not going to lie. Getting through these books was kind of a slog. I had hoped to learn lots of great new insights on how to go about automating tests and working testing into the software development process without going crazy. I found out that pretty much everything I needed to know was already covered in other books I had read, especially Clean Code and Agile Principles, Patterns, and Practices in C#, two books I've reviewed previously.
While I have some misgivings about panning anyone's work because these authors obviously put tons of time and effort into their books, I don't want to mislead any readers about the value that I got out of these books. After all, these reviews are supposed to give my assessment of whether a pair of books is helpful or not. Sometimes both books are good and complement each other nicely, sometimes one is clearly better than the other, and sometimes I don't find either one very useful. I try to describe those opinions as respectfully and honestly as I can while providing some of the reasoning behind those opinions. With that in mind, let's take a look at these two books on agile testing.
The premise of this book is something I totally agree with. In many ways software grows and evolves over time. Freeman and Pryce nicely described three distinct ways that a code base grows and changes through refactoring:
The first part of the book was much like these excerpts with clear, well thought out discussions of TDD (Test-Driven Development). The second part of the book was an extended example of developing an internet auction bidding program with Agile testing principles, and that is what I had a hard time with. It became increasingly difficult to stay interested as the example went on.
There is something to be said for working through a non-trivial example to better show how a process would work in a more realistic project, but the example that they used seemed too complicated to me. Maybe it was because I haven't written in Java since college and I'm not familiar with any Java frameworks. However, I hadn't done much programming in C# before reading Agile Principles, Patterns, and Practices in C#, and I didn't have any problems with the examples used in that book. In this Java example, the authors seemed to pick a bunch of graphics and networking frameworks that required some prior familiarity to understand what was really going on. For all I know, they are common Java frameworks, but using them served to narrow the audience that could follow along. The complexity of the example and the prior knowledge required obscured the message that the authors were trying to get across.
Once the complexity of the application increased, the big ideas tended to get buried in the details of working through the code. The main point that the authors kept coming back to was that encapsulating related functionality enabled programmers to understand the code quickly and change it easily in one place. This is programming nirvana, but it was hard to see how that encapsulation developed through all of the details of their code samples.
Another issue that came up was that after a couple of tasks were completed, it wasn't clear that it was useful to continue with the example. New concepts and benefits of TDD were no longer being demonstrated; we were just plowing through the rest of the example using the same process we already know about. The danger with a long example is that the focus changes from teaching concepts using a demonstration to completing the example for its own sake. It's a difficult balance that I'm well aware of. You start off with good intentions, but as you get further into the example, it takes on a life of its own. After a while you pull back and wonder how you got where you are, and are you really saying what you mean to say anymore. The overarching ideas have gotten lost in the details of the example. In this case, the goal should have been to show how software can grow efficiently while using TDD, not the gritty details of how to implement an auction sniper.
Even though the extended example was a drag, the authors still had some nice insights into practical testing:
Overall, Growing Object-Oriented Software, Guided by Tests was okay, but I can't recommend it. There are much better books out there, and this one didn't add much to my understanding of TDD. If you're an experienced Java programmer that happens to work with OpenFire, Smack, Swing, and the other libraries that they used in their example, you may get more out of it. Otherwise, I would pass this one by.
After getting through the last book, I was hoping for something a little more engaging with Agile Testing. This book was written by a couple of testers who have worked on a number of Agile teams and gave their perspective on how testers could fit into the Agile process after the programmers on the team have adopted TDD. Crispin and Gregory make the case that there are many other types of testing that still have a place in improving software quality, and I strongly agree. Automation with unit and functional testing only scratches the surface of the testing that needs to be done for a product, and having dedicated testers on the team that offer their unique perspective and talents is invaluable.
I did find some insights that resonated well:
The book was organized into five parts:
Some of their points were surprising as well. For example:
Later in the book the authors talk about the difficulties of bringing new testers up to speed on a project, and they make this recommendation:
The book did contain a few pleasantly surprising nuggets, though. Near the end when the authors were discussing defect tracking software, they claimed:
All in all, I can't recommend this book either. I pretty much heard it all before and it has been presented much better in the other books I've already mentioned. Maybe if you're a tester that's completely new to the Agile process, you'd find Agile Testing much more useful, but I'd hope you could find a more concise resource. As for programmers, The Pragmatic Programmer and Robert Martin's books are more engaging and are all you're going to need.
I set out to fill what I thought was a gap in my understanding, and found out that other books I read had already covered what I needed to know. I probably didn't need to read either of these books. It was too much reading about testing and not enough practicing. You only need to read a little bit about testing to understand the benefits and how to go about it, and then it's best to spend the time experimenting with it. Clean Code and Agile Principles, Patterns, and Practices in C# are more than enough on the subject of testing for developers.
 | VS. |  |
I'm not going to lie. Getting through these books was kind of a slog. I had hoped to learn lots of great new insights on how to go about automating tests and working testing into the software development process without going crazy. I found out that pretty much everything I needed to know was already covered in other books I had read, especially Clean Code and Agile Principles, Patterns, and Practices in C#, two books I've reviewed previously.
While I have some misgivings about panning anyone's work because these authors obviously put tons of time and effort into their books, I don't want to mislead any readers about the value that I got out of these books. After all, these reviews are supposed to give my assessment of whether a pair of books is helpful or not. Sometimes both books are good and complement each other nicely, sometimes one is clearly better than the other, and sometimes I don't find either one very useful. I try to describe those opinions as respectfully and honestly as I can while providing some of the reasoning behind those opinions. With that in mind, let's take a look at these two books on agile testing.
Growing Object-Oriented Software, Guided by Tests
The premise of this book is something I totally agree with. In many ways software grows and evolves over time. Freeman and Pryce nicely described three distinct ways that a code base grows and changes through refactoring:
Breaking out: When we find that the code in an object is becoming complex, that’s often a sign that it’s implementing multiple concerns and that we can break out coherent units of behavior into helper types. … Budding off: When we want to mark a new domain concept in the code, we often introduce a placeholder type that wraps a single field, or maybe has no fields at all. As the code grows, we fill in more detail in the new type by adding fields and methods. With each type that we add, we’re raising the level of abstraction of the code. Bundling up: When we notice that a group of values are always used together, we take that as a suggestion that there’s a missing construct. A first step might be to create a new type with fixed public fields—just giving the group a name highlights the missing concept. Later we can migrate behavior to the new type, which might eventually allow us to hide its fields behind a clean interface, satisfying the “composite simpler than the sum of its parts” rule.I find that a lot of my software design progresses in these ways. The architecture of a system emerges over many iterations, step by step. This constant flux is a characteristic of Agile development in particular, and it can be an issue for teams and management that are new to it. The authors make some good points about that initial discomfort as well:
[I]ncremental development can be disconcerting for teams and management who aren’t used to it because it front-loads the stress in a project. Projects with late integration start calmly but generally turn difficult towards the end as the team tries to pull the system together for the first time. Late integration is unpredictable because the team has to assemble a great many moving parts with limited time and budget to fix any failures. The result is that experienced stakeholders react badly to the instability at the start of an incremental project because they expect that the end of the project will be much worse.The goal with Agile development is to not end up in crunch time at the end of the project, but instead spread the chaotic stuff more evenly over the full length of the project, making it more consistent and manageable in the long run.
The first part of the book was much like these excerpts with clear, well thought out discussions of TDD (Test-Driven Development). The second part of the book was an extended example of developing an internet auction bidding program with Agile testing principles, and that is what I had a hard time with. It became increasingly difficult to stay interested as the example went on.
There is something to be said for working through a non-trivial example to better show how a process would work in a more realistic project, but the example that they used seemed too complicated to me. Maybe it was because I haven't written in Java since college and I'm not familiar with any Java frameworks. However, I hadn't done much programming in C# before reading Agile Principles, Patterns, and Practices in C#, and I didn't have any problems with the examples used in that book. In this Java example, the authors seemed to pick a bunch of graphics and networking frameworks that required some prior familiarity to understand what was really going on. For all I know, they are common Java frameworks, but using them served to narrow the audience that could follow along. The complexity of the example and the prior knowledge required obscured the message that the authors were trying to get across.
Once the complexity of the application increased, the big ideas tended to get buried in the details of working through the code. The main point that the authors kept coming back to was that encapsulating related functionality enabled programmers to understand the code quickly and change it easily in one place. This is programming nirvana, but it was hard to see how that encapsulation developed through all of the details of their code samples.
Another issue that came up was that after a couple of tasks were completed, it wasn't clear that it was useful to continue with the example. New concepts and benefits of TDD were no longer being demonstrated; we were just plowing through the rest of the example using the same process we already know about. The danger with a long example is that the focus changes from teaching concepts using a demonstration to completing the example for its own sake. It's a difficult balance that I'm well aware of. You start off with good intentions, but as you get further into the example, it takes on a life of its own. After a while you pull back and wonder how you got where you are, and are you really saying what you mean to say anymore. The overarching ideas have gotten lost in the details of the example. In this case, the goal should have been to show how software can grow efficiently while using TDD, not the gritty details of how to implement an auction sniper.
Even though the extended example was a drag, the authors still had some nice insights into practical testing:
Some TDD practitioners suggest that each test should only contain one expectation or assertion. This is useful as a training rule when learning TDD, to avoid asserting everything the developer can think of, but we don’t find it practical. A better rule is to think of one coherent feature per test, which might be represented by up to a handful of assertions. If a single test seems to be making assertions about different features of a target object, it might be worth splitting up.I like the pragmatic nature of this recommendation. Dogmatically splitting up tests so that each has a single assertion only serves to extend your test time and bloat your test code. When a set of assertions focus on one feature and have the exact same setup and tear down, they should be combined into one test.
Overall, Growing Object-Oriented Software, Guided by Tests was okay, but I can't recommend it. There are much better books out there, and this one didn't add much to my understanding of TDD. If you're an experienced Java programmer that happens to work with OpenFire, Smack, Swing, and the other libraries that they used in their example, you may get more out of it. Otherwise, I would pass this one by.
Agile Testing: A Practical Guide for Testers and Agile Teams
After getting through the last book, I was hoping for something a little more engaging with Agile Testing. This book was written by a couple of testers who have worked on a number of Agile teams and gave their perspective on how testers could fit into the Agile process after the programmers on the team have adopted TDD. Crispin and Gregory make the case that there are many other types of testing that still have a place in improving software quality, and I strongly agree. Automation with unit and functional testing only scratches the surface of the testing that needs to be done for a product, and having dedicated testers on the team that offer their unique perspective and talents is invaluable.
I did find some insights that resonated well:
If you’ve worked in the software industry long, you’ve probably had the opportunity to feel like Lisa’s friend. Working harder and longer doesn’t help when your task is impossible to achieve. Agile development acknowledges the reality that we only have so many good productive hours in a day or week, and that we can’t plan away the inevitability of change.This is, of course, one of the key arguments for adopting an Agile process, and you'll see similar statements in any Agile book you read. Most of the book was much like this quote—reasonable and well thought-out—when taken in isolation, but taken as a whole, it was excessively verbose and redundant. The authors were constantly discussing the benefits of automated testing, the use of this or that tool during the testing process, or how you really don't want to make an iteration a mini-waterfall process. I felt like they could have easily covered the same amount of material in less than half of the 576 pages they took to do it.
The book was organized into five parts:
- Introduction to Agile Testing
- Organizational Challenges in moving to Agile Testing
- The Agile Testing Quadrants and the different types of testing that makes up each quadrant
- Automation of any testing that warrants it
- An Iteration in the Life of a Tester and how testers fit into an Agile development environment
Some of their points were surprising as well. For example:
“AADD,” Agile Attention Deficit Disorder. Anything not learned quickly might be deemed useless. Agile team members look for return on investment, and if they don’t see it quickly, they move on. This isn’t a negative characteristic when you’re delivering production-ready software every two weeks or even more often.I wrote one small note on my Kindle when I highlighted this quote, "definitely sounds like a flaw to me." This attitude struck me as having a cavalier disregard for deeper thought and more exploratory learning, especially for difficult problems. Maybe it works when you're implementing yet another CRUD app, but some software requires careful research and design to make something that's going to have a big impact. My interpretation of this statement is that I shouldn't be doing Agile Testing if I'm developing ground-breaking software, but I don't think that's really true. I think Agile promotes examining the trade-offs before making a decision on how to proceed, and sometimes the right decision is research and learning. AADD isn't really defensible.
Later in the book the authors talk about the difficulties of bringing new testers up to speed on a project, and they make this recommendation:
This conversation sparked the idea of a FAQ page, an outstanding issues list, or something along that line that would provide new testers a place to find all of the issues that had been identified but for which the decision had been made not to address them.Ideas like this should be questioned as to whether or not they scale and if they add unnecessary complexity to the process with too many tools. I can't imagine such a thing scaling well at all, and anytime you create documentation that you'll never use personally, it will not be adequately maintained. I find that keeping these kinds of lists up-to-date is nearly impossible when they're not part of some other process that the maintainer does find useful.
The book did contain a few pleasantly surprising nuggets, though. Near the end when the authors were discussing defect tracking software, they claimed:
If your team is working closely with the programmers and is practicing pair testing as soon as a story is completed, we strongly recommend that you don’t log those bugs as long as the programmer addresses them right away.It's an interesting difference from bug tracking software proponents, and such pragmatic practices can eliminate a lot of tedious bookkeeping.
All in all, I can't recommend this book either. I pretty much heard it all before and it has been presented much better in the other books I've already mentioned. Maybe if you're a tester that's completely new to the Agile process, you'd find Agile Testing much more useful, but I'd hope you could find a more concise resource. As for programmers, The Pragmatic Programmer and Robert Martin's books are more engaging and are all you're going to need.
More Than Enough Testing
I set out to fill what I thought was a gap in my understanding, and found out that other books I read had already covered what I needed to know. I probably didn't need to read either of these books. It was too much reading about testing and not enough practicing. You only need to read a little bit about testing to understand the benefits and how to go about it, and then it's best to spend the time experimenting with it. Clean Code and Agile Principles, Patterns, and Practices in C# are more than enough on the subject of testing for developers.
The Ivory Tower of Software Development and Messy Reality
What is the best way to learn something that is complex enough and broad enough that you will spend a lifetime mastering it? Some people believe the way to learn something is to first study the fundamentals and proper techniques, and only then move on to doing or creating things. Other people take a different approach and introduce new students to something simple, fun, and engaging right away to capture their interest and imagination before getting into the more technical aspects of the subject.
It's a question of a purist approach or a pragmatic approach—the ivory tower or the open market. This dichotomy has a strong presence in software development, but it exists almost everywhere.
One example of this purist-pragmatist divide occurs when learning to play an instrument. Let's say you wanted to play the piano. You wouldn't have to go to lessons with too many different teachers before you would recognize different beliefs about teaching methods.
The purist approach would involve learning all of the proper techniques first. New student would learn scales, chords, finger position, and a bunch of other fundamentals, and then they would drill and drill and drill until they could do these things perfectly without thinking. They would do all kinds of exercises completely outside the context of real music, and then when they were ready, they could begin playing classical pieces. This is the ivory tower of perfection. Mistakes are eliminated quickly, bad habits are never allowed to take root, and the student is always on the straight and narrow path to musical virtuosity.
The pragmatic approach is a bit different. Instead of starting with technique, the students will start playing songs right away. The songs start out simple, to be sure, and proper technique is taught within the context of the songs that use those techniques. New skills are learned as needed to play new songs, and the focus is on playing real music. The belief is that the required skills will be learned in time; it doesn't have to all be done up front. Certain techniques may not be learned until much later and students may spend more time in areas that they enjoy. The students that want to become much better musicians and challenge themselves more will naturally put in the extra effort to master the more difficult techniques because the music is driving them to do it.
Which method would you prefer? They both have their merits, but I think more people are going to stick with the pragmatic approach because it's more fun and engaging. The purist approach quickly gets boring, and only the most dedicated (or most externally forced) students are going to get to the point where they can play actual music. Most people would give up on it before they got that far.
Any sufficiently complex field will have this debate between the purists and the pragmatists about what is the best way to teach beginners coming into the field. The analog to the musical purist approach in the software domain is to have students learn all of the proper computer science theory first, before allowing them to write any real software. To a purist, the right computer science education would start with data structures and algorithms, preferably in Haskell or LISP (or both). It would quickly get into compiler theory and lambda calculus so that students would have a firm understanding of the most important parts of computer science.
I have no problem with languages like Haskell or LISP. I think they're great languages, and you'll learn a lot of stuff about programming by learning these languages. But as introductory languages for beginner programmers, they're terribly difficult and a huge barrier to entry.
The purist may claim that difficulty is a desirable trait in a language, and that one of the purposes of introductory programming courses should be to weed out students that don't understand these all important language features and computer science concepts. A computer science education should certify that a professional programmer knows how to write a compiler and an operating system, is proficient in the use of strongly typed languages, and has a firm understanding of closures and first class functions.
Taking this line of thought to its logical conclusion is absurd. Do we really want to require aspiring programmers to learn all the theory first, to certify professional programmers against some arbitrary set of standards, and to only allow professional programmers to write software? What would we test for? How would we differentiate between programmers that are allowed to write compilers from those that are allowed to write payroll software for an HR department? How would we establish which level of the ivory tower each programmer could work in?
Not every programmer needs to know the same stuff to be effective at what they do, and the vast majority of programmers out there don't need to know pure computer science theory to do their work. Besides, thinking that it's even possible to structure a perfect learning path for programming and force every aspiring programmer to follow it is just silly. People don't learn how to program that way.
How did you learn how to program? I bet you didn't start by studying memory safety and resource allocation. I've never heard anyone describe their early programming experiences that way. I started out programming in LOGO, making the turtle draw simple shapes on the computer screen. Then I started writing clones of simple games in QBasic. It was fun and exciting, and that is what hooked me in. I wanted to learn more because these early programming experiences were fascinating. I hear a lot of stories similar to mine, but never have I heard a programmer say they got hooked on programming the first time they learned about closures or type safety.
People become programmers for a wide variety of reasons. Not every programmer loves programming, but the best programmers almost certainly do. The desire to program is necessary for learning the more advanced topics of programming. It's too hard otherwise, and the programmers that don't love it are going to find ways around having to think about the hard topics. In the end, that's okay. They'll still have plenty of work to do without the hard stuff.
For those that do love programming, they will be driven to keep getting better, and as a result they will learn the proper techniques and best methods of programming. The fundamentals are hugely important, and one of the best ways to get better is to practice and develop a deep understanding of the fundamentals. Like the piano student that learns the necessary techniques while playing beautiful music, the best programmers will learn the fundamentals while making great software. If they need to learn a certain programming concept to make their software better or to become more effective as a programmer, they'll do it, but the desire to program originated in those early experiences of writing simple programs to do cool things with a computer.
Creating cool programs in an accessible way should be the focus of any introduction to programming. It may be messy and impure, offending the sensibilities of the proprietors of the ivory tower, but it captures the imagination. And the imagination is a powerful force for learning complex subjects and doing awesome things.
It's a question of a purist approach or a pragmatic approach—the ivory tower or the open market. This dichotomy has a strong presence in software development, but it exists almost everywhere.
Learning an Instrument
One example of this purist-pragmatist divide occurs when learning to play an instrument. Let's say you wanted to play the piano. You wouldn't have to go to lessons with too many different teachers before you would recognize different beliefs about teaching methods.
The purist approach would involve learning all of the proper techniques first. New student would learn scales, chords, finger position, and a bunch of other fundamentals, and then they would drill and drill and drill until they could do these things perfectly without thinking. They would do all kinds of exercises completely outside the context of real music, and then when they were ready, they could begin playing classical pieces. This is the ivory tower of perfection. Mistakes are eliminated quickly, bad habits are never allowed to take root, and the student is always on the straight and narrow path to musical virtuosity.
The pragmatic approach is a bit different. Instead of starting with technique, the students will start playing songs right away. The songs start out simple, to be sure, and proper technique is taught within the context of the songs that use those techniques. New skills are learned as needed to play new songs, and the focus is on playing real music. The belief is that the required skills will be learned in time; it doesn't have to all be done up front. Certain techniques may not be learned until much later and students may spend more time in areas that they enjoy. The students that want to become much better musicians and challenge themselves more will naturally put in the extra effort to master the more difficult techniques because the music is driving them to do it.
Which method would you prefer? They both have their merits, but I think more people are going to stick with the pragmatic approach because it's more fun and engaging. The purist approach quickly gets boring, and only the most dedicated (or most externally forced) students are going to get to the point where they can play actual music. Most people would give up on it before they got that far.
Programming Purity
Any sufficiently complex field will have this debate between the purists and the pragmatists about what is the best way to teach beginners coming into the field. The analog to the musical purist approach in the software domain is to have students learn all of the proper computer science theory first, before allowing them to write any real software. To a purist, the right computer science education would start with data structures and algorithms, preferably in Haskell or LISP (or both). It would quickly get into compiler theory and lambda calculus so that students would have a firm understanding of the most important parts of computer science.
I have no problem with languages like Haskell or LISP. I think they're great languages, and you'll learn a lot of stuff about programming by learning these languages. But as introductory languages for beginner programmers, they're terribly difficult and a huge barrier to entry.
The purist may claim that difficulty is a desirable trait in a language, and that one of the purposes of introductory programming courses should be to weed out students that don't understand these all important language features and computer science concepts. A computer science education should certify that a professional programmer knows how to write a compiler and an operating system, is proficient in the use of strongly typed languages, and has a firm understanding of closures and first class functions.
Taking this line of thought to its logical conclusion is absurd. Do we really want to require aspiring programmers to learn all the theory first, to certify professional programmers against some arbitrary set of standards, and to only allow professional programmers to write software? What would we test for? How would we differentiate between programmers that are allowed to write compilers from those that are allowed to write payroll software for an HR department? How would we establish which level of the ivory tower each programmer could work in?
Not every programmer needs to know the same stuff to be effective at what they do, and the vast majority of programmers out there don't need to know pure computer science theory to do their work. Besides, thinking that it's even possible to structure a perfect learning path for programming and force every aspiring programmer to follow it is just silly. People don't learn how to program that way.
How People Really Learn to Program
How did you learn how to program? I bet you didn't start by studying memory safety and resource allocation. I've never heard anyone describe their early programming experiences that way. I started out programming in LOGO, making the turtle draw simple shapes on the computer screen. Then I started writing clones of simple games in QBasic. It was fun and exciting, and that is what hooked me in. I wanted to learn more because these early programming experiences were fascinating. I hear a lot of stories similar to mine, but never have I heard a programmer say they got hooked on programming the first time they learned about closures or type safety.
People become programmers for a wide variety of reasons. Not every programmer loves programming, but the best programmers almost certainly do. The desire to program is necessary for learning the more advanced topics of programming. It's too hard otherwise, and the programmers that don't love it are going to find ways around having to think about the hard topics. In the end, that's okay. They'll still have plenty of work to do without the hard stuff.
For those that do love programming, they will be driven to keep getting better, and as a result they will learn the proper techniques and best methods of programming. The fundamentals are hugely important, and one of the best ways to get better is to practice and develop a deep understanding of the fundamentals. Like the piano student that learns the necessary techniques while playing beautiful music, the best programmers will learn the fundamentals while making great software. If they need to learn a certain programming concept to make their software better or to become more effective as a programmer, they'll do it, but the desire to program originated in those early experiences of writing simple programs to do cool things with a computer.
Creating cool programs in an accessible way should be the focus of any introduction to programming. It may be messy and impure, offending the sensibilities of the proprietors of the ivory tower, but it captures the imagination. And the imagination is a powerful force for learning complex subjects and doing awesome things.
The Benefits of Working on a Personal Software Project
If you love programming, then you probably already have one or more of your own software projects in the works. Personally, I find that working on a software project of significant size is one of the best ways to learn new tricks as a software engineer, but that is only one of many reasons to write software in your free time.
Before getting into the benefits of personal software projects, it's worth taking a brief look at some of the different kinds of projects you could explore. If you wanted to look at embedded programming, you could get yourself a great embedded system for less than $60 (sometimes much less). The most popular choices are Arduino and Raspberry Pi, but there is also the very powerful BeagleBoard and more specialized embedded boards like the TI LaunchPad series.
If you want to get into mobile development, all you need is a smart phone or an iPod Touch and you can do Android or iOS development. You could write a game, a productivity app made just for you, or satisfy some other nagging itch that's been plaguing you.
If you want to explore web development, you have tons of resources available, both online and off, for any number of web development frameworks: Ruby on Rails, Django, ASP.NET MVC, AngularJS, MeteorJS, or something newer and less well-known. There is no shortage of options, and you're bound to come across something new and exciting.
You could even do classic application development in Visual Studio with C# or XCode with Objective-C. There's also system-level programming in Linux. You could study up on your algorithms, look into machine learning, experiment with a new library, or explore what's new in big data. You can even dabble in parallel computing by programming your graphics card with CUDA.
The possibilities are nearly limitless and readily accessible at close to no cost. All of these types of projects are immensely fascinating, but to make any significant progress, you'll have to focus on only one and resist the urge to jump from one to the next like a raccoon chasing shiny objects. If you manage this tremendous feat and settle down to work on a project for an extended period of time, here are some of the benefits that you may experience.
Big things start from small beginnings. If your personal project is filling a need that extends beyond yourself, it's quite possible that you could change the world. Change doesn't necessarily mean revolution, and the world doesn't necessarily mean all of planet Earth, at least in the beginning. Something that started out as a simple way to connect and share with other people or to optimize your health and lifestyle could grow into The Next Big Thing. Who knows? You'll never find out unless you start that project. Go learn something, have some fun, and change the world.
Choices
Before getting into the benefits of personal software projects, it's worth taking a brief look at some of the different kinds of projects you could explore. If you wanted to look at embedded programming, you could get yourself a great embedded system for less than $60 (sometimes much less). The most popular choices are Arduino and Raspberry Pi, but there is also the very powerful BeagleBoard and more specialized embedded boards like the TI LaunchPad series.
If you want to get into mobile development, all you need is a smart phone or an iPod Touch and you can do Android or iOS development. You could write a game, a productivity app made just for you, or satisfy some other nagging itch that's been plaguing you.
If you want to explore web development, you have tons of resources available, both online and off, for any number of web development frameworks: Ruby on Rails, Django, ASP.NET MVC, AngularJS, MeteorJS, or something newer and less well-known. There is no shortage of options, and you're bound to come across something new and exciting.
You could even do classic application development in Visual Studio with C# or XCode with Objective-C. There's also system-level programming in Linux. You could study up on your algorithms, look into machine learning, experiment with a new library, or explore what's new in big data. You can even dabble in parallel computing by programming your graphics card with CUDA.
The possibilities are nearly limitless and readily accessible at close to no cost. All of these types of projects are immensely fascinating, but to make any significant progress, you'll have to focus on only one and resist the urge to jump from one to the next like a raccoon chasing shiny objects. If you manage this tremendous feat and settle down to work on a project for an extended period of time, here are some of the benefits that you may experience.
Learn Something New
Starting with the obvious, if you pick a project in an area that you have little experience in, you're going to learn a ton of stuff. By working on a project that's not trivial, you will learn much more than if you had only read about it. Don't get me wrong, I love reading, and I read everything I can. But you won't learn a new language or pick up new skills simply by reading about it. Reading is hugely important and a necessary part of learning, but without something more to cement that knowledge in your brain, you'll quickly forget whatever you read. A project is a great learning tool because you'll struggle with things that seemed simple in the literature, and that struggle and the solutions that you discover along the way will stay with you much longer than what you read.Practice, Practice, Practice
Projects don't have to only be about new things. You can do a project in a language and framework that you already know well and get some great practice out of it. You could get practice in other ways, like doing programming problems on various websites, but these problems focus on narrow pieces of a very large puzzle. While solving small, contrived problems can be fun and instructive—I thoroughly enjoy it myself—working on a larger software project will give you more practice in the context of something real. You'll also be developing other skills that small problems don't address, like designing the software architecture, deciding on the features you want to include or not, and defining the problems that need to be solved in the first place. Software development is so much more than merely writing code, and doing a real project will give you good practice at all of those non-coding skills.Take New Tools for a Spin
For most people, using the hot, new programming language or framework at work is strictly forbidden. Unless you're working on a greenfield project in a company that's willing to take a risk on something that's probably not production ready, you'll be working in one of the well-established, mature programming environments at your day job. A personal software project will give you the opportunity to try out an intriguing new language that you've been meaning to learn, but has been off limits for your work projects. Who knows, if you get good enough at it, you could spin up some little utility projects at work to solve immediate problems that you're having and show off the new tool's advantages. One thing could lead to another, and you could end up adding real value to your company's business while getting a chance to play with new tools more often.A Vehicle for Experimentation
When you've learned a new programming technique or a new language feature, it's probably best to not turn around and try your new-found knowledge out in a production code base right away. You're more likely to tick off your coworkers with all of the mistakes you'll make implementing a design pattern for the first time than actually adding value to a company project. Instead, try experimenting with new techniques on a personal project where you can turn your code upside down and inside out until you've gained a true understanding of the issues involved. Personal projects are also great sandboxes for trying out crazy ideas, just to see if they work.Portfolio Gold Plating
A substantial personal project is a great addition to a work portfolio. If you've made some cool projects while tinkering at home, it's going to give you much more interesting stuff to talk about in an interview, and it's likely to help your CV stand out in a crowded field because it shows that you have a passion for programming. What's more interesting, saying that you implemented some of the business logic for yet another enterprise payroll system, or that you implemented an embedded control system for a robot that can navigate a maze? If the only kind of work experience you've been getting is the former, a personal project is your best opportunity to show off what awesome stuff you can do. Of course, doing a project for the sake of your portfolio alone won't work because the motivation won't be there, but if you're already doing it because you can't stop yourself, putting it in your portfolio is a bonus.Glory and Honor
If you manage to build something in your spare time that's truly valuable and noteworthy, and you release it as OSS, you could make a real difference in other programmers' lives. A lot of the frameworks and libraries we use on a daily basis were made in programmers' spare time, and the authors are pretty well-known in the software development community. Fame can be a blessing and a curse, though, and these OSS projects can take on a life of their own. Persevering with a big open source project is a huge commitment, and it will take a lot of energy to see it through. Even more so than portfolio gold plating, you have to have the motivation to succeed with OSS or you won't get very far.For Fun
This is the reason every passionate programmer does their own software projects, because it is great fun. Exploring new corners of the programming world, experimenting with esoteric and novel programming languages, creating useful things with your imagination and a computer—these are the reasons why programmers get excited about programming and play around with pet projects until the wee hours of the morning. Of all the benefits listed here, none is as important as having fun.…And Profit
A personal project could eventually grow into a personal business or something more. Plenty of thriving businesses started out as a simple idea and some after hours effort. The amount of effort required to make a viable business is substantially more than messing around with pet projects, but if you think you have a good product that people would be willing to pay for, don't be afraid to charge for it. The worst that could happen is no one buys it. Then you've still got a great project for your portfolio, you learned a ton of stuff, and you hopefully had fun doing it.It Needs to be Done
Sometimes there's a hole where a software product should be, and if you want it filled, you are the one who's going to have to do it. There's something about your life that keeps bugging you, and you simply can't find the right tool to fix it. You've searched endlessly on the app stores and Google and you've come up empty. Sometimes you don't even need to go to those lengths. Sometimes your problem is so narrow and personal that you know it's only going to get done right if you do it yourself. A custom piece of software can be one of the most satisfying projects because you're solving your own problem with your own skills and ideas. Nothing feels as great as that.For the Children
If you have kids, working on a project at home can be a great learning experience for everyone. You get to teach them something that you love, and they get to spend time with one of their favorite people while doing something fascinating. Depending on their age, you'll have to tailor the project to something they can handle, but all kids love to build things, create things, and control things. Software projects cater to all of those desires, and it's great for them to learn how to solve problems through their own creativity and imagination. They'll learn how to overcome problems by your example. Watching you struggle with the problems that crop up in programming, realizing that they are solvable, and working through them together is invaluable. Watching their reactions when something finally starts working is incredibly fun and rewarding. The sheer force of excitement from seeing their creations come to life will keep both you and and your kids coming back for more.Change the World
Big things start from small beginnings. If your personal project is filling a need that extends beyond yourself, it's quite possible that you could change the world. Change doesn't necessarily mean revolution, and the world doesn't necessarily mean all of planet Earth, at least in the beginning. Something that started out as a simple way to connect and share with other people or to optimize your health and lifestyle could grow into The Next Big Thing. Who knows? You'll never find out unless you start that project. Go learn something, have some fun, and change the world.
There Must Be A Better Way
As developers, we often try to defend our over-engineered and complicated solutions, and indeed, every problem has some amount of irreducible complexity. But there is often a simpler solution if we take the time to look for it. Part of good software development is believing in and spending the time to look for that simpler way. Don't resort to overly complex implementations just because the problem seems to call for it. The simple solution may not seem so shiny, but there is elegance in simplicity. It may not feel like you're practicing good engineering when you come up with a solution that looks too easy, but often times the simple solution is more robust and resistant to buggy edge cases.
In engineering and the sciences, simple is normally equated with elegant. A simple solution is easier to understand, maintain, and extend than a complicated one. The difficulty comes in finding a simple solution. It's hard work to understand a problem so thoroughly that the solution is made to look easy and obvious, but that is the mark of a good solution. All of that hard work may not be apparent in the end result, but it provides much more value than a complicated mess of code that people look at in awe because they can't believe that it even works.
Sometimes it seems like the problem is too complicated to warrant a simple solution. Other times the immense architecture of the complicated solution is a siren call that lures you in and blinds you to simpler alternatives. Finally, there are times when you stare in disbelief at the simplicity of the solution you came up with and convince yourself that such elegance isn't possible. Instead, you discard the easy option and pursue a more complicated path. I experience at least one of these traps with every engineering problem I work on, and it can be difficult to avoid them. Here are some things that I've found helpful in the quest for a better way.
Some programmers seem to revel in the intricate architectures that they create, claiming that the problem is so complex that a simpler solution is not possible. But what they've really created is an unmaintainable mess—a monstrosity that will collapse under its own weight before it ever gets close to shipping. You look at this pile of code and think, "there must be a better way," but the original architect can't see it because he's locked into the path he took and believes that the extra complexity adds necessary flexibility or extensibility.
Finding the simpler solution requires a completely different perspective and a willingness to solve only the problem at hand. A solution should be only as complex as the problem warrants. If the problem actually needs the added flexibility of the more complex solution, then it may be appropriate, but flexibility shouldn't be added for its own sake. Joel Spolsky railed against the unnecessary complexity touted by architecture astronauts when he praised pragmatic programmers that reject complicated frameworks and instead get stuff done. His advice is more pertinent than ever.
The basic idea is that design patterns, threading, templates, and other advanced programming techniques are complicated tools that should only be used when it is absolutely necessary. Don't use them just because you can. Use them when not using them will end up being more complicated. Deciding when to use the Factory pattern is a prime example of this concept. In theory, you could use the Factory pattern everywhere you instantiate an object, but that would clearly be overkill. On the other hand, situations arise where this pattern is genuinely useful. If you are instantiating similar objects from a class hierarchy in many places in your code, the Factory pattern will likely simplify your code instead of complicating it. That's when you want to use it.
I've been playing around on Exercism.io lately, and I've seen plenty of interesting examples of over-engineering in the solutions that programmers have submitted there. I thought about showing a couple examples here, but I don't want to single anyone out. Suffice it to say, some programmers submit verbose solutions with multi-level class hierarchies or delegation for simple problems like RNA transcription or converting decimals to Roman numerals. (Examples like these are plentiful on any coding site. Codewars.com and Project Euler are two more programming problem sites that allow you to compare your solutions with other programmers, and over-engineering is easy to find.)
If these programmers write such complicated code for simple problems, what do they do when the complexity of the problem scales up? I can only imagine. Complicated architectures have mental costs, and we only have so much brainpower to work with when programming. Why waste any of it on the overhead of an over-engineered, over-architected solution that solves more than the problem requires? It doesn't add clarity, and every time you have to deal with that code, it drains your mental reserves.
Even after resisting the urge to over-engineer, the simple solution may still be elusive. Once the code is working, it will still be possible to make it better. I have not written or read a first-cut of a piece of code yet that couldn't be improved. That doesn't mean that every piece of code needs to be optimized, but if a part of the program needs to be extended or has issues that need to be resolved, finding a simpler solution is almost certainly an option.
I'm constantly amazed by the simplifications I miss when programming. While I didn't feel comfortable showcasing other programmers' code, I can certainly use my own, so here's my solution to the RNA Transcription problem on Exercism.
I was happy with the solution, but upon reviewing other submissions, I found that Ruby actually has exactly the method that I created as self.transcribe already included as part of the String class. I could simplify my solution quite a bit by using the String#tr method:
Nearly all of the other problems I've solved have followed this same pattern. I try to come up with as simple of a solution as I can that directly expresses the core of the problem, and then I learn an even better way of doing it by reading other people's code. The same principle holds true for any programming that I do. If I need to find a better way of doing something, I can be sure that it exists; I just have to find it. The combination of Google and reading good programming books helps in the search for the problems I deal with outside of coding practice sites.
It may not feel like doing real work when you come up with a simple solution. You may feel guilty for producing something so obvious after a large investment of time! That's normal. Simple solutions are not something to be avoided because our egos believe that the problem warrants something more complicated. Don't confuse simple with simplistic. Simple solutions are more elegant, and are often more robust to edge cases and changing requirements than the overly-complicated alternatives. Simplistic solutions don't address all of the requirements and fall short of the problem's inherent complexity. We want simple solutions to complex problems.
If the simple solution works, don't worry that it looks small compared to the architectural edifice of a multiple-inheritance hierarchy. Coming back to Joel, he doesn't mince words on this issue:
Besides, complicated solutions always make me suspicious. A simple solution is a sign that the the problem is well-understood and the developers have thought hard about taking the right approach without generating any waste. Great programmers are like great teachers who really understand their subject and can teach it with a clarity of thought that makes complex material look easy. Great programmers will not complicate a design for the sake of over-engineering. They will search for the right solution that is appropriately simple and directly solves the problem. They don't worry about impressing others with complicated monstrosities when working software will prove their worth.
To be a great programmer, start believing that there must be a better way. Then find it.
In engineering and the sciences, simple is normally equated with elegant. A simple solution is easier to understand, maintain, and extend than a complicated one. The difficulty comes in finding a simple solution. It's hard work to understand a problem so thoroughly that the solution is made to look easy and obvious, but that is the mark of a good solution. All of that hard work may not be apparent in the end result, but it provides much more value than a complicated mess of code that people look at in awe because they can't believe that it even works.
Sometimes it seems like the problem is too complicated to warrant a simple solution. Other times the immense architecture of the complicated solution is a siren call that lures you in and blinds you to simpler alternatives. Finally, there are times when you stare in disbelief at the simplicity of the solution you came up with and convince yourself that such elegance isn't possible. Instead, you discard the easy option and pursue a more complicated path. I experience at least one of these traps with every engineering problem I work on, and it can be difficult to avoid them. Here are some things that I've found helpful in the quest for a better way.
Resist Over-engineering
Some programmers seem to revel in the intricate architectures that they create, claiming that the problem is so complex that a simpler solution is not possible. But what they've really created is an unmaintainable mess—a monstrosity that will collapse under its own weight before it ever gets close to shipping. You look at this pile of code and think, "there must be a better way," but the original architect can't see it because he's locked into the path he took and believes that the extra complexity adds necessary flexibility or extensibility.
Finding the simpler solution requires a completely different perspective and a willingness to solve only the problem at hand. A solution should be only as complex as the problem warrants. If the problem actually needs the added flexibility of the more complex solution, then it may be appropriate, but flexibility shouldn't be added for its own sake. Joel Spolsky railed against the unnecessary complexity touted by architecture astronauts when he praised pragmatic programmers that reject complicated frameworks and instead get stuff done. His advice is more pertinent than ever.
The basic idea is that design patterns, threading, templates, and other advanced programming techniques are complicated tools that should only be used when it is absolutely necessary. Don't use them just because you can. Use them when not using them will end up being more complicated. Deciding when to use the Factory pattern is a prime example of this concept. In theory, you could use the Factory pattern everywhere you instantiate an object, but that would clearly be overkill. On the other hand, situations arise where this pattern is genuinely useful. If you are instantiating similar objects from a class hierarchy in many places in your code, the Factory pattern will likely simplify your code instead of complicating it. That's when you want to use it.
I've been playing around on Exercism.io lately, and I've seen plenty of interesting examples of over-engineering in the solutions that programmers have submitted there. I thought about showing a couple examples here, but I don't want to single anyone out. Suffice it to say, some programmers submit verbose solutions with multi-level class hierarchies or delegation for simple problems like RNA transcription or converting decimals to Roman numerals. (Examples like these are plentiful on any coding site. Codewars.com and Project Euler are two more programming problem sites that allow you to compare your solutions with other programmers, and over-engineering is easy to find.)
If these programmers write such complicated code for simple problems, what do they do when the complexity of the problem scales up? I can only imagine. Complicated architectures have mental costs, and we only have so much brainpower to work with when programming. Why waste any of it on the overhead of an over-engineered, over-architected solution that solves more than the problem requires? It doesn't add clarity, and every time you have to deal with that code, it drains your mental reserves.
Find the Simple in the Complicated
Even after resisting the urge to over-engineer, the simple solution may still be elusive. Once the code is working, it will still be possible to make it better. I have not written or read a first-cut of a piece of code yet that couldn't be improved. That doesn't mean that every piece of code needs to be optimized, but if a part of the program needs to be extended or has issues that need to be resolved, finding a simpler solution is almost certainly an option.
I'm constantly amazed by the simplifications I miss when programming. While I didn't feel comfortable showcasing other programmers' code, I can certainly use my own, so here's my solution to the RNA Transcription problem on Exercism.
class Complement
TO_RNA = {C:'G',G:'C',T:'A',A:'U'}
TO_DNA = {C:'G',G:'C',U:'A',A:'T'}
def self.of_dna string
transcribe string, TO_RNA
end
def self.of_rna string
transcribe string, TO_DNA
end
def self.transcribe string, transcription
string.each_char.map { |c| transcription[c.to_sym] }.join
end
endI was happy with the solution, but upon reviewing other submissions, I found that Ruby actually has exactly the method that I created as self.transcribe already included as part of the String class. I could simplify my solution quite a bit by using the String#tr method:
class Complement
RNA = 'CGAU'
DNA = 'GCTA'
def self.of_dna string
string.tr DNA, RNA
end
def self.of_rna string
string.tr RNA, DNA
end
endNearly all of the other problems I've solved have followed this same pattern. I try to come up with as simple of a solution as I can that directly expresses the core of the problem, and then I learn an even better way of doing it by reading other people's code. The same principle holds true for any programming that I do. If I need to find a better way of doing something, I can be sure that it exists; I just have to find it. The combination of Google and reading good programming books helps in the search for the problems I deal with outside of coding practice sites.
Don't Worry That the Simple Isn't Complicated
It may not feel like doing real work when you come up with a simple solution. You may feel guilty for producing something so obvious after a large investment of time! That's normal. Simple solutions are not something to be avoided because our egos believe that the problem warrants something more complicated. Don't confuse simple with simplistic. Simple solutions are more elegant, and are often more robust to edge cases and changing requirements than the overly-complicated alternatives. Simplistic solutions don't address all of the requirements and fall short of the problem's inherent complexity. We want simple solutions to complex problems.
If the simple solution works, don't worry that it looks small compared to the architectural edifice of a multiple-inheritance hierarchy. Coming back to Joel, he doesn't mince words on this issue:
You see, everybody else is too afraid of looking stupid because they just can’t keep enough facts in their head at once to make multiple inheritance, or templates, or COM, or multithreading, or any of that stuff work. So they sheepishly go along with whatever faddish programming craziness has come down from the architecture astronauts who speak at conferences and write books and articles and are so much smarter than us that they don’t realize that the stuff that they’re promoting is too hard for us.Don't go along with the architecture astronauts if a simple solution will suffice. They may appear smarter, but they're also so deep in the complexity that they've created that they can't see that it will risk wasting everyone's time. A simple solution that works now and ships now is infinitely better than a complicated solution that will handle every possible future use case or additional feature but never ships.
Besides, complicated solutions always make me suspicious. A simple solution is a sign that the the problem is well-understood and the developers have thought hard about taking the right approach without generating any waste. Great programmers are like great teachers who really understand their subject and can teach it with a clarity of thought that makes complex material look easy. Great programmers will not complicate a design for the sake of over-engineering. They will search for the right solution that is appropriately simple and directly solves the problem. They don't worry about impressing others with complicated monstrosities when working software will prove their worth.
To be a great programmer, start believing that there must be a better way. Then find it.
Subscribe to:
Posts (Atom)