Lawrence Hecht's Former Blog and Future Info-sharing Outpost

UPDATE: After talking to Mr. Hammond, I realized the main use case I was envisioning isn’t what he’s focusing on. He is focusing on a final product that is a complete news article, like about a baseball game or an earnings announcement.

ORIGINAL POST: I hope this post doesn’t come across as sacrilegious to data visualization aficionados. Data is useless without analysis. Charts make it easier to analyze information, but don’t suggest what to do with it. Kris Hammond, the CTO of Narrative Science told the Strata Summit about why his company can solve this problem. He has created software that analyzes data and automatically presents a narrative – complete sentences – that helps explain what to do with this information. If this makes BI and other data more accessible, then it will have value.

For example, someone would write software that would process data and instead of creating a pie chart indicating that 75% of people are eating lunch, it would generate a sentence that reads, “since 75% of people are eating lunch, you should consider eating lunch too.”

To me, this seems to be another type of “report”, just like a BI report, but more useful to a decision maker. However, I expect that if you read the conclusion, then you’ll want to actually dig into the actual charts and data.

From my experience, I’ve also noticed that you can’t even get someone’s attention without having a good infographic. Pictures/charts will still be the best way to get someone’s attention, and many people are visual learners. However, considering the difficulty of creating easy to understand graphics that bring in multiple variables, I do think this is a good approach for some use cases.

Overall, this doesn’t actually get rid of the need for analysts. In fact, it can actually generate the demand for more of them. The systemrequires someone to write-up the different angles or conclusions that are presented to a viewer based on certain data. I can imagine the need for significant peer review of the phrase dictionary that would be used by the software. In this regard, the conclusions will have to be reviewed just like the input for recommendations are checked by subject area experts.

One of the benefits of Big Data analytics is that it incorporates previously unmanageable data with existing customer information located in data warehouses. James Kobielus of Forrester didn’t spend time going into the intricacies of the data management, but did talk about how it is being used, especially in conjunction with CRM systems.

This is how it works:

1. The company collects the best historical customer data, and then brings in domain experts that understand issues particular to the industry and customers of focus.
2. Work with “data scientists” to create predictive models. The models should be trying to predict a specific action or target a specific type of audience.
3. Create business rules for automated actions at different points in the customer experience. For example, if people coming from iPhone are more likely to buy red ear muffs, then show them an ad for red ear muffs.

Kobielus talked about two qualifications for implementing this type of system that I think are very important:

• Don’t forget the importance of core business metrics like “customer lifetime value.”
• Try to automate the process of creating predictive models. This may be very hard, but there is room to shorten the timeframe for things like data prep and writing automated reports.

In terms of a “jump start,” this was good. It whetted my thirst to dig deeper into issues.

I shouldn’t have been so excited to hear Jonathan Lethem speak just because he was recently awarded the MacArthur Fellowship, which is the “genius award” my father used to talk to me about. I had never read any of his books, which include Fortress of Solitude and Motherless Brooklyn, but luckily he didn’t refer to them too much during the conversation.

Starting off the fall lecture season, tonight’s public forum was definitely an example of “deliberative democracy”, as the host Brian Lehrer put it. At least four people were removed from the audience for exercising their First Amendment right to scream, mostly about the trampling of the Fourth Amendment right against unreasonable searches.

The forum’s topic was subway security and the underlying theme was the level of civil liberties we are willing to forgo in order to be safe. The forum had two panels, one on random searches, and another on surveillance cameras. The advent of random searches on the subway looking for terrorist bombs was the impetus for the discussion, but hasn’t the subway always been dangerous?

Almost everyone on the first panel agreed with David Harris, author of Good Cops: The Case for Preventive Policing and Profiles in Injustice: Why Racial Profiling Cannot Work , that the best way to catch a terrorist was to look for suspicious behavior rather than ethnicity. Heather MacDonald of the Manhattan Institute played her role as the panel’s conservative to the hilt. She defended racial profiling by saying that the terrorist threat was coming from Islamic terrorists so the police should target South Asian and Arab men. She provocatively asked if it would make sense to search black men if the police were looking for a Klu Klux Klan member. After listening to the other panel members, immigration lawyer Sohail Mohammed and law professor Stephen J Ellmann, I concluded that random searches are so ineffective that they are not worth relinquishing any of our civil liberties.

While random searches might not work, the increased police presence in the subways that comes with the searches probably will decrease crime and terrorism. As everyone panics about terrorism, a side benefit is that anti-terrorism efforts can also reduce random crime. This is what I think might happen as new surveillance cameras are added to subways.

I am reflexively against surveillance cameras but am close to being convinced that they can are a good thing. The NYC Police Department is installing over a thousand cameras into the subways. Almost everyone agrees that cameras don’t prevent crimes or terrorism, so why use them? Because they help solve crimes after the fact. For example, cameras in the London Underground helped capture the terrorists in this summer’s subway and bus bombings. As many convenience store owners can attest, cameras won’t deter most criminals. That said, I have noticed the drop in the number of drug dealers in Washington Square Park since Giuliani installed cameras. Interestingly, no one in the audience represented a high crime area where residents have been asking for subway cameras to protect them from criminals long before terrorists became a concern.

There are many reasons why surveillance cameras are bad. Unfortunately, the second panel’s main opponent of cameras, Donna Lieberman of the New York Civil Liberties Union, came across as a hysterical liberal. She and some audience members overstated the technological ability to be able to monitor, record, and save the data recorded by surveillance cameras. Heck, most tapes are re-recorded over after 24 hours or a week, and most video footage is never viewed by an actual person. Remember, the federal government is already able to monitor our phone conversations and emails, but there are not enough law enforcement officials to actual review the massive amount of data. The Center for Democracy and Technology has an excellent resource for those of you who want to learn about what the federal government is doing to monitor our every electronic move.

The panel was co-sponsored by The Smith Family Foundation and will be broadcast on WNYC.