The New York Times have ran a survey and dismayed to find a full 67% percent say that the mosque should be built on "a less controversial location", a position that betrayed the New Yorker's diverse, tolerant and cosmopolitan identity. Unyielding to public and political opposition, they reassert their belief in the editorial. I find it so courageous that I've to quote them:

"We stand with the poll’s minority: the 27 percent who say the mosque should be built in Lower Manhattan because moving it would compromise American values."

Still I think I am getting better at this work. So I'm sharing some interesting data I'm working with. On one bus route I've collected 20,000 location readings on a single day. The goal is to organize them into individual vehicle trip. After a first pass, the data are reduced to 336 trips. But how good is the quality of data? And how good is my trip segregation algorithm? So I plotted the trips' distance against their duration on a chart. The graph quickly helped me to evaluate the quality of the result.

The first impression is the result looks fairly good. A dense band of points shows that most trip length is about 10 km, and they takes about 40-75 minutes. This seems to match real world experience. At the lower left hand corner is a number of trips that last very short time and cover very little distance. I'll probably treat them as noise and discard them. A curiosity is some trips seem to cover a long distance from 15 to 20 km, much long than the official route. Where did they went?

Closer examination on their track reveal the problem. These are all legitimate eastbound trips. However the bus starts from a depot in the middle of the city. They went all the way to the terminal in the west end. Only from there the real eastbound trip begins. The challenge for me is I'm only interested in the eastbound portion. The trip from the depot to the west end is actually data pollution. I need to find a way to handle these extraneous data.

This still leaves a scatter of points of around 5 km long and last 15 to 35 minutes to inspect. Meanwhile I have difficulty to examine the track because Google Earth is crashing on me all the time. I guess this is a good time to turn away from technical work and to write a blog instead.

I have plotted the adaption of each popular novel on a graph, with different colors to differentiate the country of production. The dense chart shows that each novel has been adapted multiple times, sometimes within only a short span of time. Looking decade by decade, you'll see how the pattern has shifted. Jin Yong stories first appear on Hong Kong TV in the 70s. It was the time when most families started to own a TV. 80s is the period when a lot of early and now classic adaptions are made. Taiwan has also started to made Jin Yong stories in this period. 90s continues the frenzy activities. It is also the beginning of decline of the Hong Kong movie industry. The last Jin Yong movie was made in 1994. In 2000s, the 3 decades of restless Hong Kong entertainment production seems to come an end. The last TV made by actors is in 2001 (an animation came in 2003). Instead it is the era of China, where nearly all new productions are now made.

The filmography is not just a list black and white raw text. It comes to life once we carefully render it into a color graph. And the graph tells a story of the rise and fall of the different players in the industry.

The key scheme works like this. Let's say a customer have $10 in his account. Without a good knowledge the account balance, the customer make multiple debit charges and overdrafted his account. Let says he make a series of transactions in the amount of 5, 1, 2, 3, 11, bringing the final balance to -$12. Instead of denying the charges, Wells Fargo introduced a new feature around 2000 to automatically gives credit to the customer. The catch is each overdraft will cost them a fine of, say, $25. This in itself is not very evil. But what is really enraging is the way the bank manipulate the charges to maximize the overdraft fine. Instead of processing the debit in the order it comes in, the bank rearranged the charges to process them from highest amount to lowest amount order, i.e. 11, 5, 3, 2, 1. In this case all 5 transactions results in overdraft and a $125 fine. If they were processed in chronological order, they fine will be only $50 instead.

The bank insist they have done the right thing for the customers and the customers desire this. Because the larger amount transaction is more likely to be important, therefore it was given priority. Such reasoning is what prompt me to write this blog. It angers me because the bank is offering us a bogus reason. Since all charges are authorized no matter what order they comes in, the customers receive no benefit what so ever from such "prioritization". It would have make a difference if the bank deny the transaction once the account run out of fund. But in the present scheme, the only difference is the amount of fine the customer will incur. It can be mathematically proven that by processing the transaction from lowest amount to the highest, it results in least fine and thus the most beneficial to the customer. Instead the bank scheme to process the transaction from highest amount to lowest, this results in maximum fine and thus results in worst outcome for the customer. There is simply no justification to choose this scheme other than enriching the bank.

Note that while Well Fargo is the first bank to lose an anti-consumer lawsuit, this is an industrial wide practice. I'm looking forward for other major banks to be exposed and fined.

After this ruling and addition regulation against such practice preceding the trial, the banks are grumbling on how much revenue they are going to lose and the possibility of stop offering free checking and other banking service to make up for the difference. For me this should raise an alarm on the corporate board level. When your company is deriving a significant portion of revenue from such unsavory source, it is a clear indicator that it is not doing the right thing. It is like opening a gym with free membership, but in fact the true revenue is derived from the vending machine on the floor selling tobacco. Sooner or later pressure will come to maximize profit. Then the company will be mobilized to sell more tobacco to more people. This will become a corrupting force in no time.

Clearly a company wants to make profit and there is always a pressure to maximize profit. But the question if they earn it because they are providing value to the customer? It is justifiable if they charge for banking services because they are managing my money and dispose them to merchants I authorize to. But the overdraft fee scheme did not provide any value. Instead it is nothing but an exploitation on uninformed customers.

nor am I an Atheist, who has a particular axe to grind with God. I’m not an Atheist in that I have some objection or some dislike of that way of thinking. It’s simply that I see no use or value of entertaining such notions. I feel that I can articulate what I believe to be at the heart of my quest for meaning as a human being without any recourse to such ideas.

I can make perfect sense of the world without making any reference to god. I see no need for me to confirm or deny its existence. I won't call myself atheist though, as this term is usually associated with the opposition to the concept of god. I think a better description for me is "I don't care".

Here is the outline of my solution:

1. Prepare to enumerate all paths.
2. Prefer the local optimal during the search. Remember the best solution encountered so far.
3. Prune the search tree. For example if the current best solution has a score of 50. When you are considering another path, as soon as its score exceed 50, you should abandon it.
4. Improve the pruning by taking the additional score to complete the path into account. For example, if the current score of a partial path is S, the minimum score to complete the remaining walk is Rmin. Then if S+Rmin > best score, prune it.

Actually I have submitted the solution a few months ago. Only today do I find out I have not posted my solution in my blog. So I'm really speaking from my memory. I hope my plan is still accurate. Again my un-tidied code is available.

The other 3 facebook puzzles I have solved was posted below:

• Peak Traffic Solution
• Refrigerator Madness solution
• smallworld solution

The traffic on the highway was free flowing. Yet as you approach city K, local traffic starts to fill up the highways. the It gets to a point when everything comes to a crawl. You curse at the traffic. Then you wonder why must the intercity highway go through city K at all? Why doesn't it just bypass it so that travelers between A and B can have a smoother trip?

I have wondered about this for a long time. Now I have a theory. Roads are built for people. Where there are more people traveling, more investment will be made to build roads. This should make pretty good sense.

Now which kind of traffic have higher volume? Intercity traffic between A and B or local traffic in K? Despite your intuition that everyone seems to travel between A and B, there are actually more local traffic in city K. This is demonstrated by the fact that traffic is free flowing between A and B but clogged within city K. Therefore more funding will be allocated to build roads for residents of city K and not a bypass. Since intercity travelers are the minority, they will not be better off than the local residents of city K.

Although I grew up next door in Hong Kong, somehow I have never visited Shenzhen (besides using it as a jump off point to other parts of China). Yesterday I have an opportunity to take a dip into the city. It was an unplanned trip. I know nothing about the layout of the city. I have not even bother to purchase a map since it is only going to be a few hours. Instead I just trust my instinct. I jumped on the metro and managed to emerge in the heart of the central business district.

Shenzhen Business District

Upscale Shopping Mall

It was more than 10 years since my last visit to China. During the time the economic and urban development of China has make huge stride. I want to witness the extend of development myself. If a cityscape of glass-clad office towers and upscale shopping mall is an indicator, Shenzhen has surely joined the rank of world class cities. The Central City Shopping mall is as good as any great mall in the world. Satisfied with what I saw, I decided to move on.

According to the map in the metro station, there is a large garden outside of the mall. I set off to explore it. The garden is separated from the mall by a wide boulevard. In order to access it, it seems I have to get on an elevated walkway. But how do I access the walkway?? This has become an unexpected challenge for me. I walk in and out and around the mall before I finally found the unobvious access to the rooftop garden.

Central Park

The park is an oasis in the large city. It is well manicured with stylish fixtures that any city will be proud of. It is also a great refuge for birds and a delightful place to hear their chirping. There is only one issue though. The park is eerily devoid of people. I mean this is a prime real estate in the center of a metropolitan. Where is everybody? New York's Central Park it is not, all I can see is just a sea of green. Perhaps no one is as foolish as me to go outdoor under midday's summer heat. But I suspect this is not the only problem. Perhaps the park's access difficulty has challenged not only me but also the locals.

Shenzhen Cultural Center

I went out to see the soaring architecture of the Shenzhen Cultural Center on the other side of the park. I figure the building is too far for me to walk to. So my goal is to find a vintage point to take a good picture. This require a long walk through some confusing pathway in an empty park. But the walk just keep getting longer without going anywhere. Many times I have considered to give up my pursuit. This will be a slap in my face since I pride myself in my wayfinding skill. When I finally reached the edge of the park, I found a boulevard and the cultural center in a distant. Nice architecture. But there seems no way for me to continue the walk. It is not even an inviting landscape for a pedestrian. I just took my picture and then walked back.

By now I have spent a fair amount of time walking under the sun. I need to return to civilization to grab my lunch. But there are more long trek before I have finally reached a shopping center and its food court. I finally got a sense of Shenzhen's cityscape. It is a new city done in grand scale, wide boulevards, monument buildings and huge parks. But it is not a livable city that I enjoy.

Las Vegaseque Sheraton Hotel

You can tell I was getting cranky. The lousy Chinese fast food I have had did not help either. I rushed back to the Hong Kong side to end this short trip.

Hennessy Road is the major thoroughfare in Causeway Bay. The surrounding streets are often narrow and labyrinth like. These streets were once only backstreets behind major department stores. Today they are destinations themselves with many boutiques stores and restaurants. These side streets are lousy for moving traffic but terrific for walking and shopping. It is a superb idea to close them for cars and open them for pedestrians.

Some streets have permanently converted into pedestrian plaza. They are nicely paved and raised to level with the sidewalk. But in other area, it is not obvious what has been done to restrict vehicle access. In the picture above, the only control device seems to be the sign that says "Access Only". In the morning the streets are lined with delivery truck, those access is expressly allowed. Still access restriction is working quite well with relatively few vehicles coming in. The taxi in the picture below is one of those.

With few physical marking, how do I recognize this as a newly created pedestrian area? I think the pedestrians themselves make it a pedestrian street. When sufficient number of people come off the sidewalk and go into the street, it makes driving through difficult. Cars become intruder there. These pictures are taken in the morning when actions have barely started. I'm sure when people are coming out in full force in the evening it will become impassable for cars.

I was shopping in a Causeway Bay supermarket in the basement of a Japanese department store. To see how effective this law is, I've spent a few minutes to observe the shoppers coming out from the cash register. The ratio of people using their own reusable bag vs. disposable plastic bag is roughly half and half. I was hoping even more people would bring their own bag. But compare to the baseline figure a few years ago where reusable bag usage is almost 0, this is certainly a big improvement.

See also my eariler entry about proposed plastic bag ban in California.

It is a quite summer afternoon in Hong Kong. It has been raining since the morning. I sit in my room. My bed is at the corner of the small room. Three years after my parents moved into this new house, we still haven't taken the Styrofoam cover off the bed's headboard. My mom bought the bed for me in case I return for a stay.

Tatchee is sleeping in the couch. I make him sleep on my leg at first. A pleasant surprise for me that he did fall asleep. The temperature is not too high. But he has sweat all over. My dad is laying on his bed reading before his regular afternoon nap.

Last night Tatchee sleep with me in the bed. He fall asleep quickly. At 6:30am he has woke up. I went down to the street to bought him some bread. He didn't really want the bread so much. We played toy robots. He keep asking me to play robot with him all day. We probably played a few hours total in several stretches. It is not such a fun game for me but I like it that we played together. He dropped the ‘Shark’ tank toy on the floor and destroyed it. It was actually a toy I had from my boyhood. It was a gift that my neighbor bought me from a trip in Japan. I find it sweet that he has taken interest in a toy left over from my boyhood. It is sad that it was broken so easily.

It is a rainy day, so I can't take him to other activities outside.

My father cooked. Tatchee ate both meal fairly well.

Yesterday we went to swimming pool. I have promised to take him there. He played water gun and seesaw and water slide. He asked me to go under the shallow water with my goggles. He has fun. It was fairly windy that day and it was surprising cool in pool.

And then we take shower. For the very first time, I shampoo Tatchee under a shower. He used to protest when I shower him over his head. But his time he covered his eye with his hands and manage just fine. An excellent first time.

After we left the pool, we go to a bench so that he ate the sponge cake his mom gave him. And then we went to visit my mom's shrine. We climbed 10 flights of stairs to get to her shrine. I found it without too much effort. Tatchee recognize her pictures on the stone. We say hi to her.

I remember this moment, a summer afternoon at home in Hong Kong.

This time I have as much time as I need to review each episodes on DVDs. The Decalogue One, "I am the Lord thy God; thou shalt have no other gods before me" is a about a tragic accident. Against all the computer calculation and precaution before going on ice skating on the frozen lake, accident has befallen on the young Paweł. Daddy returns home to stare at the green glow of the screen of his trusted computer. Just yesterday it made the assurance on the ice strength. Now it displays a single message - "I am ready _" with a blinking cursor. The picture remains there silently. It is some of the most contemplative moment in the cinema.

I don't remember how I feel when I saw this the first time. The message is simple enough. It questions the primacy of science and structure in modern society and to reflect on the diminishing role of religion and spirituality.

Today I think I have resolved such conflict. It is a false dichotomy between science and religion. To me science is spiritual. The story implies a kind of scientific determinism. I think it is old school. Of course old school probably still describes the majority of people, both religious and non-religious. And also we have to take into account of Kieslowski's background working in the communist Poland in the 80s, where determinism is really the rule of the society and has far more relevance for him.

The flaw of Decalogue One is some may see it as an apology for scientists. Many people may simply conclude scientists are arrogant and ultimately hurt people. If so they are too affected by the accident. Kieslowski is a master to induce us into reflection of a moment. But we must not lose the bigger context to a single moment. Can religion or traditional knowledge make a better guideline regarding safety compare to science overall? I doubt it.

And this is only old school science. The "new" science of complexity sees the world as fundamentally nondeterministic. Many events are unpredictable. Accidents do happens. But this does not mean scientific prediction are not useful. They are useful in plenty of ways. The problem is those people who expect 100% certainty are not realistic of their expectation.

In the middle of the film the father was giving a lecture on language. Unlike other old school point of view, this lecture on the relationship of language and intelligence resonates deeply to me. He laments that computers, a machine that operates on 0 and 1, can somehow displays awareness. He was talking about emergence! It strikes me that human, an organism makes up of simple molecules and a lot of H₂O can display awareness. So a man made machine, despite its seeming simple logic structure, can conceivably display a consciousness on a high level someday by the similar principle.

I don't have a conflict between science and spirituality. Science is my new religion. This is of course the topic of another 100 blog posts.

For me its battery performance is on-par with other smart phones.

The Sprint HTC EVO is the latest Android handset coming into the market. It is a first phone that supports 4G WiMax (though not yet available here in San Francisco). And it boasts a massive 4.3" display and 8MP video camera. But all the report coming in during its first week of debut confirm one thing, its battery life sucks. Some reports complaint the battery won't even last a full day of use.

The verdict aren't universal though. Some people got considerably more battery life than the others. How much people actually do with the phone is of course one major factor. But it also dawn on me that the battery life isn't determine by the phone alone, the network signal strength also play a large role. For those lucky folks who receives a strong signal, the phone exert relative little power to stay connected. Conversely when the network signal is weak, the phone has to increase the power of wireless signal to remain connected, thus draining the battery faster.

The signal in my bedroom has relatively weak signal. It bounces between 2-4 bars out of six. So I turn on the airplane mode when I go to sleep. Nobody want to receive a call while they are sleeping, don't they? I'm still using it as an alarm clock to wake me up in the morning, when I'd turn it back on to receive calls. In 3 nights it saved 24 hours of battery use for unneeded cell connection.

I wasn't testing how long the phone last while idling. I was using it every day. But I use it judiciously with the battery conservation in mind. Here are some of the things I have done:

4G off
WiFi off (except of short use)
GPS off (except of short use)
Bluetooth off

Phone calls a few minutes a day
Some web browsing
As a WiFi hotspot (several minutes a day)
A lot of music playing
Background syncing (e.g. Google Calendar)
10 minutes of GPS tracking

I think EVO's battery is adequate for me. It is worst than my iPhone, which routinely gets 3 days of life even after nearly two years of use. But EVO's battery is sufficient to me.

I want to mention that I've done all these battery conservation trick for my iPhone initially, including putting it into airplane mode during the night. Initially my iPhone 3G sometimes didn't last a day if I didn't use it carefully. After multiple OS update, its battery life has actually gotten considerably better. I hope EVO will also improve as newer version of Android comes out to make more efficient use of the battery.

(Update: the battery has lasted 3 days in my later test)

Three years ago, over the objection of California Grocers Association, San Francisco was the first city in California to ban the use of plastic bags in large stores. Like many progressive ideas sprout from this city, it seems too radical to some people at the time. Business groups opposed and loud mouthed critics spare no effort to ridicule San Francisco. But time and again, what appear as crazy ideas at the beginning get wide acceptance eventually. Remember things like recycling and gay right. Before they get wide acceptance today, they were fiercely opposed and ridiculed at the beginning. After San Francisco, a few cities like Oakland quickly joined to ban plastic bag. Finally it spreads to the entire state. This time the California Grocers Association has turned around to support the registration. It only take them 3 years!

The final opponent to this bill is none other than the plastic industry. I want to take aim at their fallacious claim this will result in higher prices to consumers. The most important point is the cost of disposable bags will be trivial compare to cost of goods. That's why retails are willing to hand out bags freely in the first place. But if the ban of plastic bag has any effect on price at all, it should lower it. 'Free' plastic bags are after all, not free. Their cost are just part of the retailer's business cost and will be reflected in the price.

If I'm plastic bag manufacturers, I would consider switching gear to other high value product like tote bags, which can sell for $10 or more. Why be the bad guy producing millions of plastic bags at low margin and pollute the environment? Stop making product for pennies and start making high margin product! It is good for your bottom line and it is good for your karma.

Below are simply my notes taken from the book.

Ch.1 Mental Accounting, Richard Thaler. Statistical Regression, Sir Francis Galton, 1911 (tendency to fall back into statistical mean).

Ch.2 Prospect Theory, Daniel Kahneman, Amos Tversky, 1979. (Issue framing, e.g. loss aversion, sunk cost fallacy).

Ch.3 The Devil That You Know

Decision Paralysis - tendencies to avoid or delay action, particularly due to the fear of regret and a preference for the familiar.

• Maximizer/Satisfier - Herbert Simon, 1950s.
• Trade off contrast - choices are enhanced or hindered by the tradeoff between options, even for options we wouldn't choose anyway, Tversky, Simonson.
• Extremeness aversion - people are more likely to choose an option if it is an intermediate choice within a group, Tversky, Simonson. (so it can be manipulated by introducing extreme choice).
• Status quo basis, endowment effect - preference for holding on what you have.
• Regret aversion - avoid the pain of regret and the responsibility for negative outcomes.

Counter Measures

• Take opportunity cost into account
• Mark Twain's saying, "Twenty Years from now you will be more disappointed by the things you didn't do than by the ones you did do."

Ch.4 Number Numbness

• Neglecting the base rate - tendency to disregard or discount overall odds, Kahneman, Tversky.

Ch.5 Dropping Anchor

• Anchoring - clinging to a fact or figure or idea that may or may not have any real relevance to your judgment or decision.
• Confirmation bias/Preferential bias - once people developed bias, they tend to view new information in such a way that it supports those preferences (Edward Russo).

Ch.6 The Ego Trap

The tendency to attribute success to your ability but to attribute failure to other causes.

• Heads I win, tails it's chance, Eileen Langer, 1975
• "Expert" predicition, Tetlock
• "Hindsight bias", Baruch Fischhoff

When I was flipping through my picture library to select a desktop background, one of his come out fittingly! I used to use pictures of beautiful landscape or family portrait as a background. What a mistake! The subject, the detail and the contrast are really distracting. Desktop is a place for holding clutters and a place for work. If you want to see beautiful picture, you should run a slide show. Turn out this out of focus picture of some place in my home works wonderfully. It stays in the background as a background image should. It provides good contrast to the icons over it. It has more visual interest than a plain solid color background. And viola, I've a totally unique picture nobody else has!

The first thing that delights me is Droid has a physical keyboard! The onscreen keyboard is one of my biggest issue with iPhone. Despite iPhone fans' claim, the inaccuracy of onscreen keyboard bothers me. Unfortunately, Droid's keyboard quickly disappoints. The keys are flat. Its tactile feeling is so so. The keys are arranged as a rectangular grid rather than staggered. As much as I dislike onscreen keyboards, I can enter keys with a light touch. With this keyboard I have to push hard each time. My thumbs get sore quickly.

There are more serious problems though. I want to kill the designer who line the number keys on the top row. It is a real pain to type any long number like a phone number. Holding down the ALT key with the left thumb and reaching for the number key with the right thumb while trying to hold the handset steady is such a struggle. Yet this can be fixed simply by having a different layout. Arrange the keys in a numeric keypad pattern as in the diagram will do. You can easily use the left thumb to hold down the ALT key and type any numbers with the right thumb all sitting close together.

Another problem is the alignment of the keyboard. Due to used space on the right, the Droid keyboard is significantly off centered toward the left. On a normal keyboard, the G and H keys are at about the center. On Droid, because the H key is so far to the left, it makes it much harder to reach it using the right thumb. As such Droid has a QWERTY keyboard that does not really feel like a QWERTY keyboard. What a shame! To this day my most favorite phone keyboard is still the first generation Sidekick. The large and well positioned keyboard makes it a real note taking machine!

If the keyboard does not work as well as it should, the direction pad is still great. For navigation, the direction pad delivers much better precision compares to touch screen. I like to scroll using it rather than the imprecise, trial and error flicking gesture. Even better is its preciseness for moving the cursor inside a text for editing. With iPhone, it is super frustrating to try to position the cursor in any different location. So much that I simply give up correcting typing error or do much editing. With Droid I am writing this entire post using the phone, editing out every single error as I see it. Still there is a problem with Droid's direction pad. The key is hidden under the screen, making it unavailable when the lid is closed.

The 5MP camera is a big improvement over my iPhone's. The quality is approaching a basic digital camera. But most importantly, it has a macro lens that's pretty good at close up. This allows me to do what I like to do for a long time, to draw sketches on paper or whiteboard, then pull out a camera, take a picture and send it to my computer to be incorporated into a document or email. The keyboard sketches that I used in this post is the way I like to do. The important point here is not to produce the best quality illustration, which takes time and resources. It is that the process should be easy and casual. Just like drawing with pencil and paper. Droid is the first tool that make this feasible.

One of the key component that makes it possible is Gmail. This is how I transfer the pictures from the phone to my computer. Much to Google's credit, integration with Gmail is super smooth. The first time you setup the phone, it ask for your Google account information. Assuming you have a Google account, your are done. Gmail works right away! Google calendar also works! And you don't need to do any manual tethered syncing using iTunes!

Finally I want to comment on Android's UI Aesthetic. After using iPhone for a while, I find Android rather verbose. On iPhone the home screen is populated with big application icons. I can find application by picking from the pictures. With Android it is difficult to identify the application from their icons. They are about 30% smaller and their design seems not very distinguishable. Instead I have to pick the application by reading the text under it. Secondly iPhone UI relies more on icon and direct manipulation. It keeps the use of text to a minimal. On Android I often see long phrases, or even a complete sentences, for describing the action. Contrast their setup screen for the difference in style.

I suspect my son won't like my Android phone as much as the iPhone. Well he is only 4. So why does it matters? The things is he start to play with my iPhone before he turns 3, and has become quite good in applications like the camera or the photo gallery. All these he mostly learn on his own with few instruction from me (indeed I actively discourage him from messing with my phone.) I think this is a testimony that the iPhone UI is a really phenomenal success. It sets a high bar for others to beat.

The data of concern to me is tabular data. In order to do a bake off, I have generated 50,000 records of sample data. The CSV representation is shown below:

seq, name, address, city, age, birthday
1000,John M. Doe,2147 Main St.,Middle Town 14,47,1985-05-15
1001,John N. Doe,2148 Main St.,Middle Town 15,48,1985-05-16
1002,John O. Doe,2149 Main St.,Middle Town 16,49,1985-05-17
1003,John P. Doe,2150 Main St.,Middle Town 17,50,1985-05-18
1004,John Q. Doe,2151 Main St.,Middle Town 18,51,1985-05-19
1005,John R. Doe,2152 Main St.,Middle Town 19,52,1985-05-20
1006,John S. Doe,2153 Main St.,Middle Town 20,53,1985-05-21
1007,John T. Doe,211 Main St.,Middle Town 21,1,1985-05-22
...

Naturally, CSV is a contender for storing tabular data. (Indeed the data source I'm working with is in CSV format.) The two pickle modules produce identical data output. In addition, Python 2.6 also provides a JSON module that do the similar task as pickle but outputs a standard text based format. I included it in the comparison below.

First observation, CSV output the most compact data at 3MB. Pickle output is 40% larger at 4.2MB. JSON is somewhere in between. The speed? CSV is the winner among them all.

Note that CSV reader create data items as string. In the sample data, two out of the six columns are integer fields. In order to do an apple-to-apple comparison I have another test that do integer conversion after loading such that the data loaded is identical to pickle's. This impacted the performance somewhat. But it is still more than twice as fast as the faster cPickle module. The standard library's JSON's performance trailing far behind, making it unsuitable for anything performance intensive. FYI, unlike the other modules, JSON's output is in unicode.

The test is done by Python 2.6 on Windows XP machine with 2.33GHz Core2 CPU (Download source code).

Watching this episode I see a picture emerging. This is the day when machines begin to rule the world. For all the vigilance put in place after the computer induced 1987 market crash, we are still, or perhaps even more vulnerable due to more reliance on computers.

The conflict of man against machine seems like a sci-fi plot. But I think it is more realistic than most people think. The problem is we are looking for an anthropomorphous entity like HAL, an supercomputer that knows it all, have immense analytic power, and the most frightening part, it has a mind of its own. As far as I know I don't see anything like this on the horizon.

Instead there are countless software agents roaming the net, collecting information, building database, distributing information and in some case taking active roles like trading stock. This global computer network has developed immensely in the past 2 decades, impacting every aspect of our society. Mostly it makes our economy more efficient and keeping us better informed. But does the system also enslave us in some way? Are human truly in control of our destiny. Or are we at the mercy of the system?

I am imagining how the next computer glitch would look like. Think Google Calendar. One day it may suddenly 'recommend' a political rally event to all users, those logic that even its creator cannot explain. And our history may have changed as a result.