Sunday, August 26, 2012

I am not the best Scramble With Friends player

A friend from high school, JV, is better at Scramble with Friends than I am. There, I said it. After hundreds of hard-fought games, and only an occasional win, her superior skill is evident. I asked her how she could be this good. She said she used to play lots of Boggle as a kid. Hmm. I guess it is too late for me, then. Though I have improved with practice, so has she. I can only conclude that I will never be able to beat her more than once in a while. So, before I continue, let me state, J.V., you win.

So, I did what any decent, but downtrodden word game player who happens to have coding skills would do. (You might say I wrote lots of programs as a kid).  I wrote a program to replace my poorly-skilled self. I had a boring plane ride by myself last week. I brought my netbook along and whipped up a little Python script to solve any given Scramble board.

Since the two inputs, the board configuration and the dictionary, are a fixed size and not too big, I can get away with a less than efficient approach to finding all the solutions. I just put all the possible words in a set, subtract some words, and use the string.startswith method to prune the game tree. After subtracting proper nouns, words that are too long or too short, or that are not comprised of letters on the board, the dictionary only has a few thousand possible words, given a typical board setup. If there were more possible words, for efficiency I might have needed to use a trie instead of a set to store the word list. If the game had an even larger problem space, for instance if the game had a 5x5 or larger grid, I probably would have needed to settle for a heuristic to focus on areas of the board likely to produce high scoring words. As it is, though, it takes more time to key in the board set up than it does to run the solver to brute-force a complete set of solutions, on my little eeepc.

The code is here. It's a pretty straightforward solution. However, if any beginners out there want a code walk, let me know in the comments. It could be a good introduction to recursion. The program should run on any system. But, I rely on the *nix system commands sort, head, and more to present the output in a way that makes it easy to key-in. I suppose this presentation approach could be rolled into the script code itself.

Given this board:
# Set up the board. Use single letters or QU for Qu.
grid = (('QU', 'E', 'W', 'M'),
        ('A', 'L', 'O', 'E'),
        ('A', 'A', 'D', 'O'),
        ('R', 'E', 'T', 'S'))
# Set up double and triple letters.
lmult = ((1,2,1,1),
# Set up double and triple word bonuses.
wmult = ((1,1,1,1),
This command supplies me with the 65 best solutions, one at a time, in a convenient order:
scramble_solver$ python | sort -rn -k2 | head -65 | sort | more -1
Focusing on 2774 word dictionary.
ALATED           26 |....|AL..|.AD.|.ET.
ALDER            21 |....|AL..|..D.|RE..
DALE             18 |.E..|.L..|.AD.|....
DEMO             16 |...M|..OE|..D.|....
DERATS           20 |....|....|.AD.|RETS

I hope you're not too mad, JV. And congratulations. :)

Tuesday, November 9, 2010

Clustering The US Senate

I created a hierarchical clustering graph based on the roll call votes for the 111th session of the US Senate using the same method as my last post. I have a few observations about this chart. The first is that Nelson(D-NE) clusters with the Republican half when cuttree=2. The second observation is that Reid (D-NV) closest associate is Sanders (I-VT). I had assumed that Reid was a middle-of-the-road voter, and that Sanders, a socialist, had more extreme votes. Perhaps I should graph their rhetorical distance rather than roll call voting pattern distance.
It is not a surprise that Bayh(D-IN) McCaskill(D-MO) break off into their own cluster when cuttree=4. I am more surprised to see Feingold(D-WI) and Specter(D-PA) to form their own cluster with cuttree=8.
Again, I have added the source code for auto-downloading and parsing the data, and drawing the chart. Group membership is here. Click image for a larger version.

Sunday, November 7, 2010

Clustering The US House of Representatives

I downloaded the roll call votes for the 111th session of the US House of Representatives. I first cleaned the data, removing columns with a large number of missing votes, and removing representatives without much data. I also removed votes where every congressman voted the same way or did not vote. Other than those filters, all votes were weighted equally. It occurred to me that I should do more filtering based on high correlation, but I did not do so.
Then I used R's hclust function to perform hierarchical clustering based on the distances of the voting patterns of each congressperson. Finally, I used R's cuttree command to cut the tree into 8 colorful parts. Click the image for a huge version.
It is no surprise that Republicans and Democrats cluster exclusively with themselves. The only cluster that has a D and an R is cluster number 8, comprised of just two representatives, Griffith(R-AL) and Bright(D-AL). What did surprise me somewhat is that the Blue Dogs cluster with the larger group of Democrats, and the Progressive Caucus is the one to break off into its own cluster first. Full lists of cluster membership are here.
Source code for downloading and parses the source data and composing the image is here. Scripts require Linux, bash, perl, wget, and R. I plan to generate the graph again once the 112th session begins.

Monday, October 11, 2010

Mortgage Note Fraud Crisis

I'm forwarding several links without comment, as the foreclosure fraud crisis unfolds.

Meet GMAC’s Robo Signer Jeffrey Stephan (naked capitalism)
See this video of the deposition of GMAC’s famed robo signer, Jeffrey Stephan, who estimates that he signs 10,000 documents a month.
Homes in Florida Seized Without Notice of Foreclosure: Suspicously Large Number of “The Dog Ate My Summons” Filings (naked capitalism)
Make no mistake about it: the nature and scale of these frauds cut at the very heart of our judicial process. We didn’t call the Florida courts “kangaroo courts” lightly. A home is most people’s most important asset; shelter is a bedrock of personal security. Both the Fifth and the Fourteenth amendments enshrine the notion of due process, yet we see increasing evidence of it being violated on a routine basis in the Sunshine State.
Foreclosure Fraud Reveals Structural & Legal Crisis (The Big Picture)
We are not discussing economic problems of too many homes for sale and falling prices. What is being discussed here is a full blown crisis underlying home titles, foreclosure procedures, and securitized mortgages. The rampant, epidemic and systemic abuse of legal property protections is now reaching a crisis.
Multi-Billion-Dollar Class Action Suits Filed Against Lender Processing Services for Illegal Fee Sharing, Document Fabrication; Prommis Solutions Also Targeted (naked capitalism)
The common remedy for illegal fee sharing is disgorgement.
Organized Crime: Wall Street’s Foreclosure Fraud Machine (Alternet)
Just when you thought Wall Street couldn’t defraud the economy any further, it went ahead and did it. After pushing millions of borrowers into foreclosure with fraudulent loans, big banks are now being implicated in a massive new fraud scandal involving the foreclosure process itself.
Big Bank Fraud Exposed...Finally! (GoldmanSachs666)
What the attorney from ICE Legal learned in the deposition (information given under oath) was that the person who signed the Affidavit attesting to having had personal knowledge of the account and case - in fact did not. Even more disturbing in his deposition was the admission that he and a team of others literally signed thousands of these Affidavits a month under the direction of the law firm representing GMAC. This admission prompted GMAC to immediately halt all foreclosures in Florida and the other 22 Judicial Foreclosure states.
Fraud Factories, MERS, LPS, Forgeries: Rep. Alan Grayson Explains the Foreclosure Fraud Crisis (The Big Picture)

More Law in the Hands of Banks: Breaking and Entering Homes in Florida (naked capitalism)
Two Canadian tourists returning to their rental home from a day at the beach found evidence burglars had struck — or so it seemed.
Their laptop computer and MP3 player were missing, as were six bottles of wine. A half-empty beer opened by the intruders was still cold and sitting on the kitchen counter.
But why, then, had the locks on the front door been changed?
JP Morgan ‘thug’ breaks into home not in foreclosure: report" (Raw Story)
Panicked, she called 911 and spent 10 nervous minutes on the phone with a dispatcher only to discover that the intruder was an agent of her mortgage company, JPMorganChase, who had come to change the locks on her home.
Foreclosuregate and Obama's "Pocket Veto" (Truthout)
...those errors go far deeper than mere sloppiness; they are concealing a massive fraud. They cannot be corrected with legitimate paperwork, and that was the reason the servicers had to hire "foreclosure mills" to fabricate the documents. These errors involve perjury and forgery - fabricating documents that never existed and swearing to the accuracy of facts not known.
DC Waking Up to Escalating Foreclosure Train Wreck: Grayson Calls for FSOC to Examine Foreclosure Fraud as Systemic Risk (naked capitalism)
Wow, someone in DC has connected the dots: that the banks’ failure to adhere to contractual and legal requirements in the residential mortgage backed securities market are so extensive and widespread as to constitute systemic risk. Alan Grayson, Congressman from Ground Zero of the foreclosure mess, is calling on the Financial Stability Oversight Council to investigate the escalating foreclosure fraud crisis.
Halt! Who Forecloses There? (Truthdig)
Arriving a little late to the party, Bank of America has announced it is halting home foreclosure sales in all 50 states as it proceeds with a widening investigation of flaws in the process.
Too-Big-For-Paperwork: Fixing Wall Street’s Foreclosure Fraud Disaster (Alternet)
Three parties stand to lose big. The most obvious is homeowners—they’re being slapped with enormous, illegal fees invented by fraudulent documents, and frequently being illegally exiled from their homes.
Bank Fraud Edition: Fraud Fraud and More Fraud (GoldmanSachs666)
Even more important here is that by taking this action they are seemingly afraid of the legal repercussions of their illegal actions. While I believe they are betting on the government giving them favored legal status by renewing their "Get Out of Jail Free" cards, there does exist the realities that maybe their cards will be permanently revoked.
Daily Show on Foreclosure-Gate (Calculated Risk)

Debunking Banks’ “Procedural Problems” Defense on the Foreclosure Crisis (naked capitalism)
The affidavit issue is being portrayed in the MSM at a paperwork problem. Lying to the court is not a paperwork problem. Attorneys are prohibited from making a material misrepresentation to the court of fact or law. Further, attorneys in most jurisdictions have an affirmative duty to report known perjury by their clients to the court.
Axelrod signals White House opposition to foreclosure moratorium (Tom Cohen, CNN)
"Our hope is that this moves rapidly and that this gets unwound very, very quickly," Axelrod said.
Cramdown and Foreclosure Fraud (Grasping Reality with Both Hands)
Fifteen months ago, according to the Assistant Treasury Secretary, the Administration had all the tools it needed. Now, as the problem of foreclosure fraud is about to explode, a Treasury official and a senior Administration official claim they didn’t have the right tools, they were helpless.... [A]fter fifteen months of trusting banksters to do the right thing for homeowners hasn’t worked out so well, the Administration is changing its story about whether it needed more tools to motivate those banksters.
Up to 40 states plan joint inquiry into foreclosures (AP)

Cantor Opposes Foreclosure Moratorium: ‘People Have To Take Responsibility For Themselves’ (ThinkProgress)
Congressional Republicans have largely been silent on a foreclosure moratorium, as the Wonk Room has noted. But now, apparently, they are ready to take a stand — in defense of the housing industry.
Bank Disinformation I: PR Machine in Overdrive on Foreclosure Fraud Front (naked capitalism)
And below is the text of a letter making the rounds on the Hill. This is simply dishonest. First, note the complete lack of mention of the foreclosure crisis; this is an effort to divert attention from the real issue, the mess the securitization industry has made of the housing market at pretty much every step of the process, from ginning up bad “spready” loans on purpose to feed demand for CDOs, to deciding to ignore the carefully-devised procedures to make sure the securitization trust complied with all the requirements needed for it to have ownership of the mortgages; to rampant document forgeries and fraud to remedy the procedural failings. Second, it implies that servicers are happy to mod mortgages. Huh?
Financial Engineering: Truth & Consequences (The Big Picture)
The United States has been living a lie.
Bank Disinformation II: Banks Attacking Rule of Law Frontally (naked capitalism)
There is also not a single acknowledgment in the article that affidavits submitted were improper. Look how timid the Journal’s formulation is: “alleged irregularities in foreclosure documents submitted by the banks.” The banks have ADMITTED the affidavits were fraudulent, prepared by people who had no direct knowledge. This isn’t an “allegation”; these are admissions by bank employees in multiple depositions.

Saturday, May 29, 2010

Reasonable Suggestions, Mockery Dominate GOP's New Idea Farm

On Tuesday (WaPo), House Republicans launched an online community, named America Speaking Out, to gather suggestions for the future of the GOP. Semi-anonymous users are allowed to submit suggestions as well as vote Yes or No on suggestions others have made. Suggestions are categorized and can be sorted by popularity.
Immediately, the user content site of the site became dominated by surprisingly constructive, intelligent, and reasonable suggestions, which often conflicted with objectives voiced by the GOP, and brutal mockery, often taking Republican ideals to their logical conclusions with entertaining results. This website serves to highlight the disconnect between the GOP and the people, as well as to highlight the irreconcilable internal inconsistency of the GOP, a curse resulting from the Wall Street, corporate, plutocratic core co-opting the mutually exclusive Religion Right, Anti-Tax, and other easily-manipulated extremist movements, to garner votes throughout history. A great illustration of this is the 3-word submission, "drop corporate taxes", filed under American Prosperity. The No votes are winning. ( I voted no.) When the GOP's core ideas are put on display, the public reaction is crystal clear.

First, the reasonable suggestions.
In the category American Values: Life, all but one of the first full page of suggestions are constructive, and some are progressive. For example:
Government funds should not go towards religious organizations. I don't want big government getting involved in my religion, and I don't want money being wasted on something I don't believe in. End the "christian nation" bull, taxpayer money shouldn't be wasted on religion...

In the category Strengthening Families there are several very nicely worded suggestions.
There are as many different kinds of families as there are Americans. And as in so many areas, this diversity is a strength. We should help respect all of these families, not condemn some to a second-class existence. Two people of the same gender who want to have a family should be allowed to marry and help preserve our great institution of marriage.

I think the Republican party should remove itself from the gay marriage debate entirely. I know a lot of Republicans feel strongly against this overwhelmingly because of religious reasons, but that has no place in government. If the Republican party were to decide to remove itself from this debate as a party, I think that would pave the way for a lot of fiscal and other conservatives to join us. ... I have a hard time coming up with non-religious arguments against gay marriage, and we do have separation of church and state in this country. ... Republicans should drop out of the debate.

Recognize gay marriage. No, seriously. It's a weird consequence of the separation of church and state. A church can marry or refuse to marry anyone; the government, however, should accept the status of its constituents. If they're married in spirit, and living as if married, then the government should accept this. How does this interfere with any church's beliefs? If one church does not think it should marry a gay couple, then another church shouldn't be allowed to? That makes no sense. Allow gay marriage.
I think if the GOP were to adopt these ideas, America would be better off. We need two healthy, reasonable parties. Currently, I don't feel I have a choice in the general election. This is convenient in a way; the choice is always clear. But, with only two parties to chose from, there should be a wider spectrum in both, to prevent marginalization. This may be a moot point, however, since there are already signs of a GOP pushback against these suggestions. "GOP won’t allow ideas to raise taxes on new site, but Hitler references and anti-Hispanic rhetoric are ok," writes ThinkProgress:
Project leader Rep. Kevin McCarthy (R-CA) said the GOP will stick to its “principles” — meaning they won’t incorporate any idea they don’t already agree with. Moreover, the new site comes with some caveats. “Someone who wants to come on and make the suggestion on how to raise taxes, for example,” said Rep. Peter Roskam (R-IL). “They are welcome to do that, [but] that’s not something that we are going to take up.” Also discouraged? “Suggestions on amnesty or a path to legalization for illegal immigrants and anything pro-choice.”
And Rep. Mike Pence (R-IN) Says ‘It’s Not A Listening Tour’. Oh well. Let's let's just move on to the Mockery.

Filed under "Environment Conservation":

Filed under "Infrastructure":

Filed under "Terrorism Abroad":

(Blogger's image upload tool leaves much to be desired)

Filed under "American Competitiveness":

Filed under "Open Mic":

See Also:
Top National Security Suggestions On GOP’s Policy Ideas Website Are Progressive: Banning Guns, Repealing DADT ThinkProgress May 27
GOP won’t allow ideas to raise taxes on new site, but Hitler references and anti-Hispanic rhetoric are ok. ThinkProgress May 26
After Saying House Republicans Will Listen With ‘America Speaking Out,’ Pence Says ‘It’s Not A Listening Tour’ ThinkProgress May 25
GOP Launches Yet Another Hollow Effort To ‘Listen’ To The American People For Policy Ideas ThinkProgress May 25
It's deja vu all over again for the GOP DailyKos Barbara Morrill May 25

Wednesday, March 3, 2010

Not All GOP Obstructionism Is Quantifiable

I had planned on accompanying my last post, Visualization: Cloture Motions Brought To a Vote in US Senate By Term, with some context. Essentially, I intended to say "counting cloture motions filed or even cloture votes taken doesn't really encompass the whole picture," What I show in my graphs is only one kind of visible obstructionism inflicted on our legislature. I hesitated because I was afraid I might not get all the details exactly right, and I moved on to other things.

Luckily, in the meantime, blogger David Waldman at DailyKos wrote, "counting cloture motions filed or even cloture votes taken doesn't really encompass the whole picture," in his post, How filibuster crazy is the GOP?.

The requisite motions might not even get filed if the leadership thinks they can't get close to the magic number of 60 to invoke cloture, yet that bill will have been filibustered as surely as any other on which there's a cloture vote on record. Nor does the official tally count what I've been calling "painless filibusters," that is, roll calls on which a 60-vote threshold is agreed to by unanimous consent as a method of cutting short the time it takes to file for, invoke and finalize cloture. Yes, the higher threshold is agreed to ahead of time by everyone, but only because the threat of a "real" filibuster (whatever that even means anymore) forces everyone to accept the "painless" version as the faster path, while leaving the 60-vote hurdle in place just as surely as it would be otherwise.

Every hold left unchallenged and every objection to unanimous consent endured because of the time involved in breaking it can also be attributed to the filibuster, but will never show up in these charts

So yes, the Republicans are on course to shattering all records for obstructionism. But the costs are much, much higher than the numbers reveal.

This rounds out my thoughts on the matter nicely. Good job, and thanks.

Thursday, February 25, 2010

Visualization: Cloture Motions Brought To a Vote in US Senate By Term

I developed a cloture vote counter aka Tarantino counter.
It automatically updates based on

I traced the source of a graph that appeared on The Rachel Maddow Show, and essentially brought their graph up to date, permanently.

Then I made web-embeddable emblems. These update automatically, too.

View them in place, here, for details about how they are calculated.