As I type this, the calm clicks of the keys are punctuated once a minute by a sound I can most accurately ascribe to an evil, mutant cricket. It is the smoke detector at the top of the stairs, warning me that its battery is low. It has, in fact, been warning me quite urgently and persistently since 3:22AM this morning. I do not believe the timing is random, and I suspect it is actually, motive unknown, part of a sinister plan to do me in.

Having invested our full share to inflate the housing bubble, our home meets all of the recent building codes pertaining to fire safety. This includes a full complement of hard-wired smoke detectors — one in each bedroom, at the top and bottom of any stair way and in locations within hallways whose precise specification eludes me. The net result of this is that we have no fewer than nine smoke detectors in the house. I discern at a minimum that the authors of the current building code possess significant stock holdings in smoke detector companies, even if they are not fully complicit in the threats against my life.

While superficially this surfeit of protection may seem well intended, I cannot explain the low battery warnings. These devices are wired into the wall current; could not someone have designed these things with rechargeable backup batteries? Our home is three years old now and the alkalines are all predictably failing. Of the four detectors who have so far called out for assistance, each has chosen the deep, dark of night to start their cry. And this is where the inescapable logic of probability gives credence to the plot against me.

With 24 hours in a day, the probability that any given smoke detector is going to reach its low battery state on a particular hour is 1/24 or roughly 0.042. We might optimistically assume my critical need for sleep occurs between 11:00PM and 5:00AM, a span of six hours. The probability of a smoke detector demanding its battery be replaced during my core sleep cycle is thus 6/24 or 0.25. A 25% chance here makes me sound like a conspiracy theorist, but I assure you this cover of reasonableness masks something much darker.

If the probability that one smoke detector is going to wake me is 0.25, the odds that two consecutive detectors will both do so is 0.25 x 0.25 or 0.0625, or 1 in 16. That scenario is going to happen to a lot of people, maybe even you.

The odds that three detectors will wake me are 0.25 x 0.25 x 0.25 or 0.015625, or 1 in 64. That there is just bad luck. But four detectors, 0.25 x 0.25 x 0.25 x 0.25, is 0. 0.00390625 or 1 in 256. In other words, not impossible but somewhat improbable.

Even still, why do I perceive this as a threat upon my person? The clincher is that this fourth detector, the one that has so thoroughly robbed me of sleep and sanity, is located at the top of the stairway in such a position as to require an extension ladder and nerves of steel to reach. With the stairs involved, the elevation is well over 20 feet and is the highest part of the interior of the house. Any error on my part spells certain death on the unmerciful tile floor below, assuming I’m lucky enough to miss the railing. I cannot quantify this dimension of the problem, but my sleep deprived brain tells me a scenario that both demands and deprives a person of physical dexterity must be at least million-to-one odds. One in 256 odds I can stand, but the likelihood of the detector at this particular location starting its tirade at 3:22AM is simply too much to write off as chance.

I am prepared to say the smoke detectors are out to get me. I’m waiting for the caffeine to fully kick in, then I will do battle with ladder and nine-volt battery in hand. If these are my last words here, you will know who struck me down.

UPDATE: I survived. Here’s the battery to prove it.

Not All Brains Have Equal Processing Capacity

There’s all sorts of interesting supercomputer news out there recently. Los Alamos National Laboratory’s Roadrunner supercomputer narrowly edged out Oak Ridge National Laboratory’s Jaguar to take the title of the world’s most powerful with a score of 1.1 petaflops of computational power. Not to be outdone, Lawrence Livermore National Laboratory announced they were contracting with IBM to deliver a 20 petaflop machine in 2009. That would actually create a machine more powerful than the 500 fastest supercomputers on the planet and represents a significant leap forward.

Naturally, all of us are looking for faster computers these days. Ray Kurzweil’s hope is for computers fast enough to deliver full blown supra-human artificial intelligence. As in brain-in-a-jar, download your head and live forever stuff. You can read all about it in his book The Singularity Is Near. Be forewarned… If you’re the least bit apprehensive about computers inside your skull, Ray’s train of thought may have you pricing cabins in Montana. Ray pegs the arrival of the Singularity as circa 2040.

To hit that goal, Ray suggests we need computers that can perform on the order of 10 quadrillion floating point calculations per second. In supercomputer-speak, that’s 10 petaflops. That’s actually right in the realm of where supercomputers are running today, and IBM’s 20 petaflop supercomputer-beast is scheduled to be turned on later this year. So did the Singularity already deplane and we just missed it?

Probably not.

With post-graduate work in AI, and a day job involving some fairly large distributed transaction processing systems, the computational aspects of the problem here is interesting to me personally. Certainly, the power and capability of computers will continue to rise over the years and more “intelligent” software systems are coming. We’ll leave the ethical dimensions of whether you want to replace your brain with something running Microsoft Windows 9.0 to another blog, but for now the comparison between computers and brains make for an interesting math riff.

In this two-part riff, we’ll look at a brain by-the-numbers to see how big it is, discuss how that relates to computer hardware and then reflect on what that means for near-term computer-based intelligence comparable to human capability.

A Brain by the Numbers

Nominally, we can reduce a human brain down to a very large collection of cells called neurons that are all interconnected. A human brain has roughly 100 billion (10¹¹) neurons. That sounds like a lot, but in the computer world we’re used to dealing with large numbers and 100 billion bytes would be 100GB. Your hard disk is probably bigger than that, and you’d find the equivalent amount of data on roughly two Blu-Ray disks. A neuron modeled in a computer program is going to be bigger than a byte obviously, but even still we’re starting out with something that superficially seems to be of similar scale.

So, why isn’t my computer doing all my work for me? Well, there’s a lot more complexity between your ears.

Neurons need to “talk” to each other to solve problems, store memories or perform all the other tasks your brain ultimately does for you. In the brain these connections are created using biological wires called axons, and where an axon touches another neuron you get a connection called a synapse. To put it another way for modeling purposes, imagine your brain is a map and all of the neurons are cities on the map. The axons are the roads between individual cities. From any given city you can drive directly to some number of cities, but in most cases you may have to go through multiple towns and cities to get from any spot on the map to another.

In the case of neurons and axons, it turns out there are on the order of 10,000 or so connections between any given neuron. In our map analogy, that’s like 10,000 roads leading in or out of a given city. It becomes clear that the complexity of our computer brain model isn’t so much dictated by the number of neurons as it is dictated by the number of axons. If we’re looking at 100 billion neurons, our model suddenly needs to account for 100 billion x 10,000 axons, or one quadrillion (10¹⁶) connections. If we could model an axon as a byte, that’s roughly 1,000 TB of data. That’s 500 of the latest and greatest 2TB hard disks on the market, or a number called a petabyte in the disk storage industry.

Of course, like neurons, we can’t model an axon connection using a single byte. Realistically, we’re probably looking at structures with multiple pointers, threshold potential values or other similar bits of data. A good lower bound on the size of a computer data structure to model an axon and its connection between two neurons is probably around 32 bytes, so our total brain model storage is around 32 petabytes, or about 16,000 of those shiny new 2TB hard disks. I’m starting to feel a little prouder of my brain’s storage capacity.

So how long before we can fit our brain design on a single disk drive? In terms of raw disk storage, 2TB drives are the current state-of-the-art. We need a 16,000x increase, and we know disk storage density is doubling right now roughly ever year. 16,000 is close to a power of two… 2¹⁴ is 16,384. So, if we don’t run into any significant technical obstacles, we should have some storage device capable of holding 32PB of data in around 14 years. Not bad if the rate of progress holds out.

Except storage devices are terribly slow. In fact, disk drives are thousands of times slower than the actual microprocessor that would need to look at our brain model. In fact, given a modern 320MB/s SATA disk interface, just reading the data off a 32PB disk drive would take three years. Disk interfaces will improve over time as well, but clearly we can’t use our brain model efficiently when it’s on a storage device. The key is getting the brain model into faster silicon memory so that we can work on it quickly. And that, unfortunately, starts to tell us where Ray’s dream machine breaks down.

Read the rest of the article in Part 2

As a first step, we might try to figure out how to get 32PB of information into the computer’s main Random Access Memory (RAM). This is different from the disk storage space, which is always a much larger number. The amount of RAM in your computer is significantly less than what the hard disk can store, but it can be manipulated much, much faster. In fact, a computer really can’t do anything with any information unless it’s loaded at least partially into RAM.

RAM is typically installed using a little card called a DIMM that is plugged into the computer’s motherboard. The most cost-effective DIMMs today hold 2GB of memory (you can get bigger ones, but they cost significantly more per unit of storage). It would take 16 million of today’s DIMMs to store our 32PB brain model. If we figured our brain model might run on a high-end computer with space for 16 DIMMs on the motherboard, we would still need a 1,000,000x improvement in memory density.

Unfortunately, Moore’s Law says we only double our density every two years. To get a 1,000,000x increase, that equates roughly to 2²⁰, which means we’re going to need 20 generations of improvement over 40 years before we get out 2PB DIMMs. That’s already got us out to the year 2049, so we’re pushing Ray’s calculations a bit, but we’re still in the park. That is, if RAM is actually fast enough.

Is RAM Fast Enough for Brain Modelling?

Once we’ve got our model in RAM, we can get the CPU to do something with it. The neurons in your head signal each other fairly slowly, but they do it all in parallel. So, even though a neuron might “fire” only 5 or 6 times per second, you’re talking about 100 billion of them working in parallel, so you’ve got quite a bit of processing going on.

Normally the way someone might look at this problem is to start with the number of axons and figure we need to run short bit of computer code across each one a few times a second. If we start with our figure of about 10¹⁶ axons and figure each scan across an neuron or axon might take roughly 10 microprocessor instructions on each pass, and that gets us in the ballpark of 2¹⁷ floating point operations, or about 10 petaflops. No surprise, this is spot-on with Ray’s calculations. But we’re already anticipating super-computers with more than this range of processing power… At that scale, we should be seeing HAL 9000’s pop up around us soon.

We’ve already realized we need to look at axons, which led us to our 32PB data structure. Let’s imagine to mimic a human brain, we need to run our model through the microprocessor five times each second, which means we need to move a little piece of our model out of RAM and into the processor, update it, then move it back to main RAM five times each second. Each pass requires moving 32PB of data back and forth, or 64PB of transfer. To do that five times requires 320PB of memory bandwidth per second.

A modern memory bus moves 128 bits (16 bytes) of data at a rate of 1.3 gigahertz. Multiply those two values and you get a rate of about 20GB/s of bandwidth. How do we figure the needed technological advance here? No problem. We’ll just figure out the relationship between 320PB/s and 20GB/s and Moore’s Law should help us again…

320PB/s = 320,000 TB/s = 320,000,000 GB/s

320,000,000 / 20 = 16,000,000x

So we need a 16 million times increase in performance, or 2²⁴, which gives us 24 generations or (at two years per doubling) 48 more years of Moore’s Law moving forward. Right around the corner, right?

Unfortunately, physics has something to say here. Current technology is building circuits on process technology with a feature size of 45 nanometers. A silicon atom in the silicon crystal structure is actually around 250 picometers (0.25nm wide), which might make you think features on a current silicon die are about 180 atoms across. However, the legs feeding into the transistor are around half the feature size, or about 22nm. That means our current process technology is producing connections that are only 90 atoms wide. Using the same math, the 32nm process that Intel is ramping up for 2010 production will have on-chip connections only 64 atoms wide.

Even starting from current process technology, you can’t cut a number like 90 in half too many times. In fact, after five or six cuts  (or about eight new generations of semiconductor processes on Intel’s road map above) you’re down to a feature size that’s less than an atom across. Halving the feature size yields four times the density, so we’re talking about an potential increase of 4⁶, or 4,000x. That’s exactly 2¹², or half-way up the exponential curve to 16,000,000x on our doubling scale, but in absolute terms only 1/4000th of the way to the memory performance we need. If we project out to 2nm features, or 1nm interconnects, we’re envisioning structures that are only 4 atoms wide. That represents something closer to 2^8 improvement (256x). Realistically, we’re going to be lucky to get 200x-300x more performance out of semiconductor technology before we hit the atomic wall, and we’ll probably see the curve flattening out on the time axis long before we get there.

What About the Super Computer?

But the astute reader says, “Hey! Don’t we already have a 20 petaflop computer coming using today’s technology? Why won’t that work?” The problem with the performance numbers for modern supercomputers is that they’re related to highly distributed problem solving that, so far, doesn’t look well suited to modeling the complex connections between neurons in a brain. In fact, the Roadrunner computer uses 129,000 separate smaller processing cores to reach its peak number.

Modern supercomputers are really collections of very large numbers of independent processors, memory and subsystems that are designed to perform lots of independent calculations. If a single CPU today can perform a billion calculations a second, chaining 1,000 machines together and sending a 1,000 independent problems to each one has the effect of running a billion calculations per second. But, if one of those machines needs to communicate with another for part of its work, it requires sending a message over a similar sort of memory bus or a network, which literally requires thousands of times the overhead of just doing a step locally at the CPU. Once you’re “off the chip” the performance advantage of a large number of processors is quickly eaten up.

Modern CPUs avoid moving off a single chunk of silicon by storing as much information as possible right by the CPU in very, very fast memory called cache. In fact, most of the extra transistors that all the advances enabled by semiconductor process technology go to larger and larger caches because the CPU cores can run much faster if the go less frequently out to RAM. When the CPU needs something that’s in the cache, it gets it darn fast and can keep running full steam ahead. Imagine your CPU is a sports car tearing up the striped line on the highway. As long as the CPU can find what it needs in the cache, it’s like that car is doing 100MPH. When something isn’t in the cache, and if it has to go all the way to main memory, you get something called a “pipeline stall” which is the equivalent of your Ferrari CPU stopping dead at a red light. And if has to go all the way to the hard disk for anything, that’s basically like stopping at a resort for a weekend vacation.

As long as a single problem can be broken down in such a way that there’s very little communication between each part of the problem and you can fit most of this information neatly into the processor’s cache, this works very nicely. Things like 3D rendering, weather modeling or simulating nuclear tests work very nicely on super computers because you know data elements are located in a physical geometry (X/Y/Z coordinates in physical space) that won’t change and their values are a function of their neighbor’s. Some value 1,000 points away isn’t relevant to this point right here. Modeling problems this way eliminates most communication between nodes, and makes quickly running lots of parallel independent calculations possible.

Modeling large scale synaptic networks with 10,000 connections (axons) between nodes (neurons), where the connections aren’t necessarily spatially dependent and where they change frequently, simply doesn’t fit the architecture of current supercomputers. The problem isn’t easily split apart into thousands of independent chunks of work. There’s too much required coordination over too many changing channels of communication.

Even if you think of a neuron only connecting to 100 other potential neurons, you don’t need a large number of “hops” between neurons before you’re basically traversing the whole architecture. One neuron may connect to say 100 others, but those 100 may themselves connect to another 100 each, and so on. If you make eight hops, you’re at 100^8 or 10^16 connections. That means checking our brain model and seeing how a series of eight neurons might connect up could wind up anywhere in the model’s space… You basically need the whole model in a single fast memory (like the CPU cache) to do any realistic work at the CPU’s top speed.

What this implies is that even if you have an extraordinarily powerful microprocessor, or lots of them, or lots of cores, or whatever the current paradigm is for CPU architecture 40 years from now, a reasonable model of the brain is still going to have to access a fairly holistic problem model. Running lots of computers (or independent processors in a super computing cluster) won’t help because those independent processors can’t see enough of an unpredictably changing model without a gigantic cache. Without some radically new architecture or memory model, you really need one “silicon brain” looking at the electronic simulation of our human brain repeatedly (and quickly) to make the magic happen. And the bottom line is we don’t have enough atoms left at the bottom end of Moore’s Law to get enough memory bandwidth there using the current semiconductor technology we employ today.

Ray remains optimistic that we’ll escape the atomic limits of Moore’s Law and move on to sub-atomic scale computing, quantum dot based calculation or some other ground shaking paradigm. Obviously, our brains work well using a different type of technology, so ultimately we’ll find some new approach that reaches and exceeds human capacity. Until then, my advice is to keep your brain in tip-top shape (lots of worksheets!) because computers are a long way from taking over thinking for you.

What Do You Need to Win?

267 Votes versus 271 Votes

39 States Plus D.C. versus 11 States

It’s officially two weeks until Election Day, and we’re already gearing up here for an evening of popcorn in front of the TV. With Dad not being a big sports fan, election night is something like the Super Bowl around here, and this is the first election where the kids realize there’s something like “teams” we root for throughout the night.

That said, if we learned anything in the last two elections, it’s that score keeping is hard. Seriously, if football had scoring rules like the Electoral College I think we’d be looking at a lot more hockey fans out there. Let’s take a closer look!

There are 50 states plus the District of Columbia who are granted electoral votes for purposes of the presidential election.  Each state is given a number of votes equal to the number of representatives it has in Congress. Each state has two Senators, plus some number of members in the House of Representative. The number of House of Representatives members from each state is based generally on population. The District of Columbia is granted three votes even though it has no representation in Congress. For the 2008 election, all of this totals up to 538 Electoral College votes.

Most of the states award their Electoral College votes in all-or-nothing contests. If a candidate wins, the candidate gets all of the votes for that state. The exceptions to this are Maine and Nebraska, who divide their votes in all-or-nothing contests at the congressional district level, and with the two votes corresponding to the Senate seats awarded based on the state-wide popular vote. In theory, this could cause Maine or Nevada to split their votes between the candidates, but because these states have a small number of congressional districts (and a correspondingly small number of Electoral College votes), the math generally works out where the state-wide votes and the votes within individual congressional districts always agree. In fact, a college vote split has never occurred, but polls are close in Nevada this year, so keep a watch there for this potential bit of election trivia to expire.

Half of the 538 vote total is 269, so a candidate must win 270 votes to get a majority and win the game. It doesn’t matter how they get there, but obvious the bigger the haul in one of those all-or-nothing states, the closer to victory. Nearby is a list of the states sorted in descending order by their electoral votes. If you look at this chart, the top 11 states account for 271 votes, or just enough to win the election. In other words, the bottom 39 states and the District of Columbia count as much as those top 11 states. In fact, you have to add up 15 of the lowest ranking contributors just to get close to the 55 Electoral College votes that California racks up.

So why do we persist with this system of all-or-nothing contests? Doesn’t that just perpetuate that whole 2000 election/Florida fiasco? Actually, the Electoral College vote helps encourage candidates to spend time in individual estates, even if it’s only the ones where the contests are close in the polls. If the race was purely decided by the national populate votes, candidates would probably spend virtually all of their time in those top few states since that’s where they’re likely to shift the largest absolute number of votes. The Electoral College encourages candidates to take into account the interests of a wider range of the states, which was really the intent of the system’s designers. Second guessing the Founding Fathers would be a bit like making touch-downs and field goals both count for two points… It might sound reasonable, especially if you’re the kicker, but it’s not football anymore.

When you consider that one of the original proposals was to have Congress elect the President, you’ll probably be just as glad we got the Electoral College system in the bargain.

But back to the numbers. The top 11 states are overwhelming concentrated in the Eastern half of the United States, and the only two big Western states, California and Texas, are respectively and heavily tilted toward the opposing teams. With the outcome of those two states pretty well known, there will be a lot of riding on those remaining top 9 states. For all intents and purposes, our game kicks off 8:00PM EST when polls for the majority of those states will be closed and the national networks will start calling the points.

Several of the big states are tilted one way or the other, but the big open plays in the game are really Florida and Ohio. With the team wearing the blue uniforms solidly polling double digit leads in both California and New York, the red team really needs to tip these wobblers in if they want to stay in the game. If McCain loses these two states, Obama  will likely have eight of the top 11 locked up and the game is basically over before half-time. If McCain picks up both of these states, we could be looking at election-overtime. Either way, we’ll have lots of pop corn ready. Go team!

Here’s a few quick math trivia facts to keep your Electoral College-unaware friends (or your kids) amused during the night of CNN and FoxNews channel surfing…

What is the number of Electoral College votes being decided in the 2008 election? 538

How many votes are needed to win? A clear majority or 270 votes will be necessary to win the 2008 Presidential Election.

Can there be a tie? Absolutely. If both parties end the night with exactly half the total, or 269 votes, the sitting House of Representatives decides the election. If the House vote is a tie, the Senate (which has 101 votes including the Vice President, so it can never have a tie), decides.

How many different ways could the Electoral College votes be cast? In theory, with 51 different contests (ignoring the Nevada/Maine complexity), there are 2^51 or 2,251,799,813,685,248  potential ways the combination of individual elections could be decided. No matter how you slice it, that’s millions of times more than the actual number of voters. How’s that for counter-intuitive?

How many of the largest states are needed to win the Electoral College? Alternatively, what’s the fewest number of states a candidate needs to win the election? The top eleven states total 271 votes, or one more than what is necessary to win.

What’s the range of votes assigned to individual states? California has the most Electoral College votes at 55. Seven states and the District of Columbia have the fewest at 3 votes each. The average state has 10.5 Electoral College votes.

Who has the “home state advantage”? Obama and Biden share 24 votes (21 from Illionois and 3 from Delaware) compared to 13 for McCain and Palin (10 from Arizona and 3 from Alaska). The Vice Presidential picks did little to tilt either candidate’s chances in the Electoral College math.

Some sources:

Breakdown of Electoral Votes by State

State-by-State Poll Numbers Leading Up to Election Day

Check out the Green Papers for Poll Closing Times

A Hundred Billion Pennies Saved, is $1 Billion Earned...

Big numbers starting with seven are in the news this week, and it’s worth a Math Riff to try to get a handle on just how much tax payer contribution our elected leaders are asking us to sign up for. It’s those billion dollar figures again that boggle the mind. Did you know a billion dollars worth of pennies weighs 312,000 tons, and would make a cube 126 feet on a side? Or that there are over 2 billion pennies in circulation today?

The $7 billion dollar amount is the current revenue short fall in California. Governor Schwarzenegger is getting ready to hit up Uncle Sam to cover the gap until the credit markets loosen up and the state can go back to conventional lending sources to cover its cash needs. That should get California through this fiscal year, but what the state will do next time around is anybody’s guess… Unless the California tax payers can find two million pounds of pennies in between their couch cushions or car seats, there could be an even bigger crunch come next year.

For even bigger fun the $700 billion dollar figure represents the potential total amount authorized this week by congress and the president to purchase mortgage backed securities. The hope is this action by the federal government will stabilize some of the organizations holding on to these instruments and put them back in good enough shape that other companies will feel comfortable doing business with them. It’s possible just knowing the government is a potential buyer of last resort may build confidence, and if we’re lucky maybe the full $700 billion won’t be tapped… Only the first $250 billion is approved initially. Initially, that’s 78 million pounds of pennies stacked in a cube almost 800 feet on a side. Maybe it’s time to buy stock in copper mines. Oh wait, pennies aren’t made of copper any more.

Anyway, the government hopes to sell these securities back into the market somehow, or otherwise collect some of the money these securities would generate. In that respect, it’s like the federal government making an enormous bet on the housing market recovering… If we’re actually at the bottom and these securities have some intrinsic value that’s just being buried due to mark-to-market accounting rules, some people think the government could make a profit here… As a taxpayer, I’m not sure anyone should be expecting that.

Other than starting with a lucky digit, is there a comparison to be made between a federal loan for the California budget deficit and the MBS bailout? In theory, the $700 billion MBS bailout is buying something tangible and a good chunk of that should wind up returned to the Treasury as the markets stabilize and markets for those securities re-emerge. The California budget shortfall, however, is pure spending… It’s gone once it’s received and California tax payers will be on the hook for repaying that loan, plus closing the gap with next year’s budget. In my mind, that could be a bit scarier because it’s an ongoing gap. Fortunately, that gap is somewhat tangible.

Per person how much is $7 billion for California residents? According to estimates from the census bureau, California has about 36.5 million people living there. That means California is short this year just a bit under $200 for every person living in the state. That sounds a bit more manageable. You can probably find $200 in loose change in some cars around Beverly Hills.

The MBS bailout may be somewhat secured, but the numbers there aren’t nearly as friendly… The population in the United States is right at 300 million people, so the MBS bailout package at $700 billion amounts to roughly $2,300 per person if the entire amount is spent. In terms of tax payers, the most recent available statistics from the IRS show that in 2006 just under 93 million tax returns were filed, which means that $700 billion amounts to $7,500 per tax payer. If you factor out returns where the net tax was under $100, that reduces the total to 65 million returns or just over $10,000 per actual tax payer. Ouch.

Another interesting statistic is to look at this $700 billion as a fraction of the total real estate assets in the United States. The Federal Reserve reported residential real estate assets in 2006 (the market peak) at a value of $19.8 trillion, and estimates based on the drop in median housing prices are that this value is now about $13 trillion dollars. At $700 billion, the government is taking a roughly 5% stake in the current asset value of all U.S. residential real estate… And probably the riskiest 5% at that.

Here’s to hoping that luck is on the side of the $10,000 investment the government just made on my behalf.

Links:

Tax Payer Statistics, Straight From the Horse’s Mouth
And the Specific Ones from 2006 Cited Above
U.S. Census Data on California
Some Really Fun Penny Facts

Everything You Wanted to Know About Pennies