UPDATE: After talking to Mr. Hammond, I realized the main use case I was envisioning isn’t what he’s focusing on. He is focusing on a final product that is a complete news article, like about a baseball game or an earnings announcement.
ORIGINAL POST: I hope this post doesn’t come across as sacrilegious to data visualization aficionados. Data is useless without analysis. Charts make it easier to analyze information, but don’t suggest what to do with it. Kris Hammond, the CTO of Narrative Science told the Strata Summit about why his company can solve this problem. He has created software that analyzes data and automatically presents a narrative – complete sentences – that helps explain what to do with this information. If this makes BI and other data more accessible, then it will have value.
For example, someone would write software that would process data and instead of creating a pie chart indicating that 75% of people are eating lunch, it would generate a sentence that reads, “since 75% of people are eating lunch, you should consider eating lunch too.”
To me, this seems to be another type of “report”, just like a BI report, but more useful to a decision maker. However, I expect that if you read the conclusion, then you’ll want to actually dig into the actual charts and data.
From my experience, I’ve also noticed that you can’t even get someone’s attention without having a good infographic. Pictures/charts will still be the best way to get someone’s attention, and many people are visual learners. However, considering the difficulty of creating easy to understand graphics that bring in multiple variables, I do think this is a good approach for some use cases.
Overall, this doesn’t actually get rid of the need for analysts. In fact, it can actually generate the demand for more of them. The systemrequires someone to write-up the different angles or conclusions that are presented to a viewer based on certain data. I can imagine the need for significant peer review of the phrase dictionary that would be used by the software. In this regard, the conclusions will have to be reviewed just like the input for recommendations are checked by subject area experts.
I just heard DJ Patel, the guy who created LinkedIn’s data team talk about creating data science teams.
Besides mentioning Python in passing, he didn’t talk about software skills or statistics competency. Instead, he said being a creative problem solver was the most important pre-requisite. Here are a few more things to look for:
- Passion for data
- History of manipulating data to solve problems
- Ability to clean data
- Ability to find and meld multiple data sets
- Skills to visualize data
Interestingly, he didn’t focus on communications skills, which is something I’ve heard people talk about.
One of the benefits of Big Data analytics is that it incorporates previously unmanageable data with existing customer information located in data warehouses. James Kobielus of Forrester didn’t spend time going into the intricacies of the data management, but did talk about how it is being used, especially in conjunction with CRM systems.
This is how it works:
- The company collects the best historical customer data, and then brings in domain experts that understand issues particular to the industry and customers of focus.
- Work with “data scientists” to create predictive models. The models should be trying to predict a specific action or target a specific type of audience.
- Create business rules for automated actions at different points in the customer experience. For example, if people coming from iPhone are more likely to buy red ear muffs, then show them an ad for red ear muffs.
Kobielus talked about two qualifications for implementing this type of system that I think are very important:
- Don’t forget the importance of core business metrics like “customer lifetime value.”
- Try to automate the process of creating predictive models. This may be very hard, but there is room to shorten the timeframe for things like data prep and writing automated reports.
In terms of a “jump start,” this was good. It whetted my thirst to dig deeper into issues.
I’m going to ignore my problems with the term “data scientists” and focus on the actual topic. This is going to be the first of several posts on the subject.
Cathy O’Neil of Intent Media talked about hiring data scientists at today’s Strata Summit. To attract someone good, remember they are looking for interesting projects and good data. She says, and I agree, that you don’t need a PhD, but rather someone with hands-on experience working on independent projects.
When interviewing the person, you need someone who understands stats. If you don’t understand the topic, it would be worthwhile to hire a consultant or borrow a friend to help out. Of course, make sure that the person is a good communicator. If they can’t explain the stats in lay terms, then maybe they aren’t going to work out, especially if they’re going to be a team leader.
In terms of using the new hire, make sure that he/she is solving business problems and are thus deeply involved with company decision-makers. If that actually happens, I’m skeptical about. Most executives say they are data-driven, but in reality
Also, you can use these folks to create reports. This allows the company to not rely on canned BI reports and/or relying on the IT team to create these reports, which are often created using knowledge of SQL.
As someone who works at a company that conducts surveys, I know that collecting more information is pointless if you don’t have the capabilities to analyze it effectively. If people don’t take this conclusion to heart, they will be deeply disappointed by whatever “big data” or predictive analytic solution they purchase for their company.
Luckily, the Corporate Executive Board (CEB) recently published a report that frames this issue effectively: Overcoming the Insight Deficit: Big Judgement in an Era of Big Data. Even though they might not want to admit it, this is a good follow-up for executives who read McKinsey’s Big Data: The next frontier for innovation, competition, and productivity and want to take some specific action. This posting is a summary of their report, with commentary by moi interspersed.
Without a high level of data competency, companies can’t take advantage of the information they already have, let alone the information deluge that is about to come.
CEB creates a compelling case that employees need to improve their ability to find and analyze relevant information to make better decisions. To help executives achieve this goal, they’ve created an “Insight IQ” index that benchmarks the “analytic maturity” of the company. In general, they measure this in terms of 1) information attainability, 2) information usefulness, and 3) employee capability. Unfortunately, CEB doesn’t reveal how they actual calculate the index. For a report about data, it is odd that they don’t actually detail how they came up with numbers. That said, the main value of the report, without the consequent benchmarking service that is being sold is in highlighted actions that can be taken. Here are examples:
- Develop more “informed skeptics” by educating employees on the limitations of data and help them improve their critical thinking. They also note that formal training on analytic tools should focus on techniques rather than the functionality of specific tools. Based on a recent Meet-up I attended, I also agree with their assessment that coaching skills are critically important for consultants or new hires. In fact, interpersonal skills are really important because IT and hardcore data analysts are much less effective if they don’t have the “anthropological” skills to work with business leaders.
- Challenge Biases and Assumptions: Similar to what a good futurist does in strategic planning sessions, the entire company, as a group, should be willing to challenge assumptions about what data is important. From personal experience, I know that executives don’t communicate effectively about the data they want to use to make actual decisions.
- Improve Quality and Sharing of Data: A core problem is maintaining clean data that is accessible to analysts in multiple business units. This is a core issue that requires executive leadership because otherwise IT departments and other fiefs will cause problems.
- Make information usable by providing a greater selection of analytic tools. This recommendation was one of my takeaways after listening to a Focus roundtable on Self-Service BI. I like to say this is the basis of the open data movement: standardize the format of data and make it accessible to people regardless of the tools they use to analyze it. Some people might be Excel whizzes, others might be SAS jockeys, and still others might be writing interactive dashboards with tools like Tableau. The important thing is that the data is sound and the methods are well applied. In that regard, the way the data is visualized, aggregated, and filtered is really important. However, since people have different needs, it is fool’s errand to try to create one über tool to use the data.
Here is an example of why I’m bullish on this projection: TheInfoPro press release. For those who don’t want to click on the link “More than 50% of new servers being installed in 2009 will host virtualization, and future progressive growth indicates 80% by 2012.”
In the process of re-installing this blog, I am using this post, which is real, as a test. In the near future this blog will again be focused on a very narrow range of topics.
Fun, fun, fun, Judaism is cool! What other religion tells you to get drunk, so drunk you can’t tell the difference between Haman (bad guy) and Mordechai (good guy)? In honor of Purim, Nancella, Scott, and I went to American Schmidol at the Bowery Ballroom. This spoof of American Idol was organized by Heeb magazine and the record label JDub and featured karaoke contestants being judged from actors from popular Comedy Central shows.
Nancella was dressed up as Queen Esther. We both got drunk. I flirted with pretty girls, which made me feel confident. Once the show got started I got into it. I felt alive as I screamed encouragement and criticism from the balcony. I got more drinks and went down to the very front of the stage. Eventually, I called my sister and confirmed our plans to meet her at another Purim party. Nancella and I said goodbye to Scott and then Nancella asked me to jump on the stage and give one of the judges a note saying, “You’re cute, let’s hang out, here’s my info.” So, of course I jumped onto the stage in front of several hundred people, handed David Wain the note and supposedly one of the other judges mouthed “sexy” with a nod to the note’s author. Wow, I don’t know if that experience was more exhilarating for me or for Nancella. We continued onward to the Upper West Side where we met my sister Lanna, who looked especially pretty that night. The night ended with Lanna performing a great stunt, but she’s kinda private and won’t let me talk about her exploits that night at a bar called Yogi’s.
I shouldn’t have been so excited to hear Jonathan Lethem speak just because he was recently awarded the MacArthur Fellowship, which is the “genius award” my father used to talk to me about. I had never read any of his books, which include Fortress of Solitude and Motherless Brooklyn, but luckily he didn’t refer to them too much during the conversation.
Do you have childhood memories of a pillow fight? Do you like to engage in public activities others steer away from? If so, then you should have been at the pillow fight in Union Square this Saturday.
Newmindspace organized this event as interactive public art. A Wikidepdia article says this activity fits into the larger social phenomenon of flash mobbing. I heard about this absurd event through the listserv Nonsense NYC. Other bizarre events I participated in include Chengwin’s Homecoming and a reenactment of a Roman Vomitorium. Why do I participate in these types of things? I usually don’t gravitate towards sporting events or organized public activities. I like non-conformity, but this was still a community activity even if most of the participants were weirdos. I took part in the pillow fight not because I had some deeply loved childhood memory, but rather so I could say I did another “only in New York City” thing. The prevalence of people who brought cameras indicated to me that others also attended because of the novelty of it all. That being said, most of us had an amazing time releasing our tension by swinging pillows at each other.
Photos are from brooklynvegan The participants were of all ages, but I bet over 50% were ages 21-28. There were a surprisingly large number of females. At first I felt a bit bad about hitting a girl, but I got over that pretty quickly. I got hit in the head so many times. I smiled a lot even as I repeatedly got hit over the head. After 30 minutes of fun, I was soaked with sweat on the cold winter day. My friends had stood to the side and didn’t participate. I was covered with feathers that I just couldn’t get off my coat, hat, or scarf. At the end I was glad to leave.
Once a month a bunch of intellectual Jews gather in the Lower East Side. Surrounded by Soviet propaganda, the evening could have taken place 50 or 100 years ago, but this is New York City circa 2006. Novel Jews is a series of readings by Jewish authors that is organized by Alyssa Abrahamson of the 14th Street Y and Alana Newhouse of the Forward. KGB Bar provides a kitschy venue with Communist icons adorning the red walls.
When I arrived, I ordered a KGB energy drink that was actually re-labeled Red Bull. Ilana Stanger-Ross read a slightly erotic excerpt from her book Sima’s Undergarments for Women. Narrated from the perspective of an older Jewish saleswoman, she talked about different sizes of breasts and nipples. I heard how women bond by making fun of men for not noticing underwear. Finding a bra that fits is important according to a recent New York Times article I read. I pay attention to lingerie, but it’s not something I’m going to buy for a woman without her being there with me. It was a week before Valentine’s Day and I was seated next to many women in the tightly packed red-themed room. I wonder if my cheeks reflected an image of bashfulness when the Stanger-Ross character admitted to an unwanted glance at a customer’s breasts. I was self-conscious about my own glances and thoughts. The other author that night was Lara Vapnyar, who read from a yet to be published novel, Memoirs of a Muse. She spoke from a Russian immigrant’s perspective. Unfortunately, I got bored trying to listen to the soft-spoken women with a thick accent.