ETech - Better Collaborative Filtering
This presentation made It pretty clear that, in reality, organizational structures look very little like the hierarchical ones laid out on paper.
What is Ethnography?It's the qualitative description of human social phenomena, based on fieldwork. A holistic research method founded in idea that a system's properties cannot necessarily be accurately understood independently of each other.
Charles is an ethnographer.
MotivationIn 1997-98 he worked on electronic communication projects with large orgs (BT, UK Gov, Pfizer, etc.) He was fascinated by the amazing dysfunction of corporate information systems. People were spending far too much time looking for stuff that should have been at their fingertips. He had a hunch something was interfering with instinctive behavior, and had a desire to understand underlying behaviors.
The ProjectHe spent 12 months living in the Isles of Scilly. He adopted a social entrepreneur role in community (an initiative to support IT-related skills), which makes things easier than vs. being there with a clipboard.
He had an emphasis on informal observations, questioning, and learning. There were formal interviews at the end of the project, and he had help from two mentors.
Isles of Scilly
- 50 miles off end of cornish peninsula
- 2km x 1.5km
- 80 people
- sub-tropical climate
- 1 pub, 1 shop, no hotels, no cars
- small-scale tourism, flower farming ceased to be main industry, it's now a crofting economy (bit of this and that)
- just tractors & electric buggies
- unpasteurized milk
- all children aged 5-11 learn together
- 1 teacher, 1 classroom
If people decide to do something, they'll just go out and do it, especially if it's a community-building activity.
Observation & AnalysisLet's say a resident returns from St. Mary's with news: "no boat on Friday evening". This news is relevant to 5 people on the island. In 2-3 hours it reaches everyone who needs to know, and hardly anyone else--five relevant hits, and 2 non-relevant hits. It would happen faster if the news was more important.
How did this happen? Charles had a slide up outlining how persons A,B,C & D share the information with persons E & F, who tells H, etc. I'm not going to go into it here because a) it's a LOT easier to make sense of if you see it, but more importantly b) I didn't get the whole thing. Unfortunately I don't have the slide to share with you, either. But, here are
Five Underlying Behaviors that Make this Happen
- groups have implicit authorization params that govern how information is relayed
- groups pool intelligence on who needs to know something and who is best placed to pass it on
- groups can function as targets for relaying information ("the boat crew" was a target)
- each person and group is identified with certain semantic triggers that activate relaying behaviors. If we hear a piece of information that touches a semantic trigger, we feel a need to pass it on. For example, my parents pass on any interesting article about computers to me. No expertise is required to relay this information.
- the further away two people or groups are in a social network, the higher the threshold for relaying information between them. Two friends would relay all kinds of information, regardless of relevancy. Distant acquaintances would relay information only if semantic triggers were very relevant.
Comparison with a Typical Enterprise
Situation: an employee hears some news that's relevant to six people in the organization.
- distribute to team only (small distribution)
- circulate throughout vertical division
- email across horizontal tiers (noise is high)
- upload onto intranet or wiki (need to search or browse)
What's happening to useful social mechanisms in the enterprise?
- formal structures and electronic information systems are crippling them
- they are alive and kicking is emergent, informal "water cooler" networks
- informal processes are helping to compensate for failures in formal information systems
The costs to the enterprise are very high:
- 34% of sent email is unnecessary
- 49 minutes wasted managing email daily
- 15-35% of a knowledge worker's time is spent looking for info
- Collaborative scoring (digg & slashdot), which are effective inside homogeneous expert communities (not heterogeneous communities, though, which most enterprises are.)
- Semantic scoring of unstructured data (Autonomy), but this is not responsive to individual relevance.
So, how could an information system harness social behavior to make communications better? Using Trampoline.
How Trampoline works:
Person A comes across a document... "this looks interesting, I’ll email it to the team". Sending the email hits a Trampoline module which:
- indexes contents
- finds trigger matches using semantic analysis
- gets authorization
- evaluates the social network
- checks on recipient preferences (do they want to receive certain kinds of information)
- de-duplicates the message (nobody receives multiple copies of the same process)
The result: emails are sent appropriately to unspecified recipients, documents are sent appropriately to related people, and the message is also sent to trampoline modules of other groups, which cascade.
However: you hit insuperable scaling problems if you try to evaluate every item for every person. It works OK up to 50 people, but a technical limitation is hit after that.
So they invented a completely new technique for distributing items through a social network (and patented it). Their largest client has 3.5k users. By the end of 2006, Trampoline will be comfortable with everything up to 50k users.
For it to work, you need activity, content and user preference data from across corporate ecosystem. So they built an integrated collaboration platform. The larger organizations are not interested in collaborative platforms, but are interested in intelligent filtering and alerting functions will be introduced to client systems in summer 20006.
Charles then shows the Tramposcope, which is an insanely cool visualization that shows all the email communication between all people in a company. You can see who the important people are in an org, and the strength of the communication ties between people. It looks like a massive web, with thicker lines between people who send a lot of email to each other...nothing like an org chart.