March 2009 Archives

Sometimes a casual pick can lead to big problems. Early on in developing our new social media product we decided to use JSON for storing our data in HDFS. This idea has worked out really well for us. The performance is good, it's easy to read if you want to take a peak at a file by hand, and there are libraries for every language under the sun, making it easy to write streaming jobs in your language of choice.

What didn't work out so well for us was the library we chose. The net.sf.json library is a big improvement over the code it's based on from www.json.org, but it has minor memory leak, apparently it is using a threadlocal hashmap of objects seen to make cycle detection faster. This fails miserably when we try to send gigabytes of data through the same Mapper object -- it holds a reference to every object deserialized.

Since we were unfamiliar with Hadoop when we started, we didn't identify the library as the cause of the problems. We expected some performance hiccups, and when we ran out of memory, we just gave it more. But still the problems didn't go away. Finally a heap dump pointed us in the right direction, and once we knew what to look for, it was easy to spot the suspect line of code.

At this point we did what we should have done at the outset: ask the mailing list. A quick note to the core-users Hadoop mailing list gave us one person who had previously had trouble with json-lib (and got it fixed quickly after reporting the bug), and two who recommended Jaskson JSON Processor. Jackson is even easier to use than json-lib.

Here's some sample code that shows how to read and write objects.

Santa Cruz County says anyone can sign a ballot argument (from
http://www.votescount.com/books/argument.pdf )

"Signers of arguments for or against a county, school, or special district measure do not have to meet the criteria listed above. The filer of the argument must meet the criteria above; however, anyone may sign the argument."

Shasta county agrees:
"Arguments and Rebuttals must be accompanied by the Verification Statement included in this Guide. (Elections Code §9600) There is a distinction between a “filer” and a “signer or author.” The filer argument or rebuttal must be either the governing board of the district, a bona fide association of citizens or an individual voter who is eligible to vote on the measure. The “signers or authors” of the argument or rebuttal can be any person or organization accompanied by a signature of a principal officer. Filers do not have to be signers."

Sutter county has almost identical language to Santa Cruz. Butte County also.

San Luis Obispo has different "filers do not have to be signers" language.

Orange County allows signers who are not filers, but requires that the filers formally delegate their right to sign. See http://ocvote.com/election/rebuttal_Handbook.pdf

Sacramento County follows a similar system to Orange County.

Where did this all come from? San Mateo County ROV says that our arguments against local tax measures authored and filed by LSPM's secretary can only be signed by people who are allowed to author or file the argument. First, the law doesn't say that. Second, our actual disagreement is about whether we have to use our titles. That is, I'd like to sign ballot arguments as "Kevin Dempsey Peterson, Software Engineer", because that's the designation I'll be using if I run for office again. The ROV wants to put me down as Treasurer, Libertarian Party of San Mateo County. Seems somewhat reasonable, and I've have no problem if the law were actually written that way.

Bonus points: the form they provide for submitting ballot arguments misquotes California election code, combining "authors" and "signers" into the same thing (while the EC makes a distinction).

California's proposition 8, which says that marriage is only between a man and a woman, illustrates fundamental flaws in democracy. The right answer to the "issue" of gay marriage is the same as the answer to the "issue" of what movie I should watch tonight -- no one's business but mine and my wife's. I think that the vast majority of Californian's would agree with this position if asked, but social conservatives were able to frame the debate so that the voters were faced not with the question of whether the government should dictate who can marry whom, but instead forced to codify their gut feeling on marriage into law.

Almost everyone I have spoken to over the age of forty who isn't politically active is uncomfortable with the idea of gay marriage. To them, "marriage" means a man and a woman, and when the question is phrased as whether two people of the same sex can marry, they will say no, just as they would say that you can't be friends with someone you have never met, or you can't sing a song with no words. These aren't moral judgments, these are just simple statements that their definition of marriage means one man, one woman. They don't have any strong opposition to gay marriage, it's just a "how would that work" uneasiness.

On the other hand, against this mild queasiness, you have millions of Californians being told they cannot marry, they cannot gain the same legal recognition for their family as others. These same sex couples aren't just being insulted, they are encountering real, significant consequences of the limitations of the law.

With that background, a small number of social conservatives were able to frame the question as "what is the definition of marriage", rather than the more relevant question of "should same sex couples have the same rights as everyone else". In the eyes of the law, these may be the same thing, but to the voter, these subtle choices of wording and point of view change everything.

This is why I say that proposition 8 was a failure of democracy. From a legal standpoint, I agree with Tom Campbell that the California Supreme Court made a questionable ruling when they legislated from the bench and this should have been legalized through the democratic process. But the initiative process failed miserably. The California legislature, able to weigh the public's general queasiness against the significant legal discrimination against same sex couples, was doing their best to provide those rights, as with domestic partnerships. They were doing a reasonably good job of balancing the conflict between what they knew to be right and good for California, and Prop 22, which banned same sex marriage and they did not have the power to repeal.

But this is not how direct democracy works. The initiative process says one person, one vote, so one person's slight unease at "legitimizing" gay couples who make them a little uncomfortable is given the same importance as another person who desperately wants to marry his long time partner. Similarly, democracy says that the vote of someone living off government assistance is just as valid as the vote of someone who works and pays taxes. We let people with no children vote in school district elections. We let people who do not own houses vote to raise property taxes.

In short, we allow two wolves and a sheep to vote on what to have for dinner. Is there an alternative? Yes, the alternative is freedom. Freedom, coupled with the rule of law, says that we take certain things off the table. Certain things are removed from the calculus of political bargaining. We strictly rule out any bill that restricts freedom of speech, of the press, of religion, of the right to keep and bear arms, of the right to be secure in our homes.

By now you may be snickering, because you know that those rights are no longer strictly protected. They are viewed only as suggestions, or guidelines. What has happened is that the people have forgotten why those rules were adopted. They have willingly reelected politicians who debase themselves and sell our freedom for short term gain, who promise us safety, if only we consent to police searching our bags on BART. Who promise us financial security, if only we hand over half of everything we make.

Remember that prop 8 isn't about marriage, and it isn't about gays. It's about freedom. As long as the voters remember the first question to ask isn't "which side am I on", but rather "is this really a political question at all", we can dig ourselves out. We can go back to the day when you could live your life in peace, and go about your business without having to convince your neighbors, or, even worse, have to hope that some judge would decide on a whim that he liked you, and find some convoluted argument that allows him to rule the way that he wants to.

If I have to choose between freedom and democracy, I'll take freedom any day.