A Modern Proposal
Last month, I gave a presentation about paid crowdsourcing in the humanities at SDH-SEMI. Below are my notes.
I
The rhetorical model in the humanities is appreciation: we believe that by paying attention to an object of interest, we can explore it, find new dimensions within it, notice things about it that have never been noticed before, and increase its value.
In a 2004 talk, John Unsworth characterized the dominant model of the humanities as one of appreciation– rigorous and qualitative. By examining a work from multiple angles and multiple contexts, our belief is that we can learn “notice things about it that have never been noticed before, and increase its value.“ Such research does not easily lend itself to large-scales like quantitative work does: qualitative undertakings, ones of concentrated appreciation, are restrained by the amount of human involvement available.
However, as we explore new ways to utilize our digital environment for humanities research, so-called ‘big data’ approaches are not only becoming possible but inevitable. The archival efficiency of computers coupled with the digitization efforts of historians, librarians, and digital humanists has resulted in endless bytes of data to understand and call our own, while the offline limitations of scale have left a large area of questions thus far unturned.
In these democratic days, any investigation in the trustworthiness and peculiarities of popular judgments is of interest.
There are numerous approaches for scaled up humanities research. Today, I’ll speak of one in particular: crowdsourcing. In doing so, I’ll describe how crowdsourcing is currently being undertaken and share a project of my own — one where semi-anonymous online users rewrote Jonothon Swift’s A Modest Proposal — as one approach to crowdsourcing workflow.
So-called big data is an important area of growth in our field. This year, Unsworth himself has spoken a few times about the need for librarians and other scholars to start thinking about approaches to large-scale research. The reason is that, even though computing allows us to make arguments from data, “data still requires interpretation, and you can still make better and worse interpretations, and more or less compelling arguments” (John Unsworth, 2012). In field questions at another talk, Unsworth argued that regardless of whether we welcome big data, somebody is going to do it, so we should start thinking about how it should be done.
In his own forward-looking blog post earlier this year, Dan Cohen made a similar observation. He writes, “all of us are facing either present-day or historical archives of almost unimaginable abundance, and we need sophisticated methods for finding trends, anomalies, and specific documents that could use additional attention.”
Finally, Ted Underwood has written that,
DH doesn’t have to be identified with scale. But the fact remains that problems of scale constitute a huge blind spot for individual researchers
-Ted Underwood, 2012
Underwood added that problems of scale “also define a problem that we know computers can help us explore.” However, I will argue for a broader, more humanistic understanding of the computer’s role in tackling problems of scale. Yes, on one end of our tradition we quantify our data for computers to make sense of — as seen in text analysis, text and data mining, and visualization. With crowdsourcing, we can make use of another feature of computing: its ability to connect large groups of people and efficiently distribute works (and work) to them.
Crowdsourcing is an approach for achieving large conceptual tasks through large-scale distributed collaboration. The efficiency of digital technologies in connected people has allowed coherent organization of many contributors, resulting in large-scale or rapidly realized projects that use human reasoning and cognition.
Contributors to crowdsourcing projects are often interested volunteers. For example, in DH, we have the Suda Online and Transcribe Bentham projects.
Suda Online, or SOL, is a translation of the Suda, a byzantine Greek encyclopedia. The Suda is a particularly gnarly work, written in a transitional language and with many grammatical and factual errors. This makes translation difficult and annotation necessary. Thus, in 1998, a number of scholars on a classics mailing list resolved to translate it together, opening the doors to anybody (with the necessary skills, of course, politely vetted) and deciding on a continuous online publication of the work-in-progress rather than waiting for a final edition. The day the precursor to Wikipedia was announced in 1999, Ross Scaife joked on the discussion list that “they must have gotten this idea from the Suda on Line…” . Fourteen years later, SOL is nearing completion and is already an often-referenced work among students.
Another volunteer crowdsourcing project is the Transcribe Bentham Project. Transcribe Bentham is a manuscript transcription project at University College London (UCL) testing the feasibility of outsourcing transcription work to members of the public. The project was seeking to digitize 12,500 folios of manuscripts by philosopher and early UCL champion Jeremy Bentham, a subset of 60,000 held by UCL Library Services (Moyle et al. 2010). The Transcribe Bentham project proved to be popular among retirees and academics-on-leave.
Volunteer labor is an effective approach to large abstract projects, if there is a reason to participate. With SOL, for example, students sometimes contribute translations as part of their studies. Others, as senior editor David Whitehead explained to me, volunteer “for the fun of it.”
However, for volunteer labor, you need a motivating factor. Crowdsourcing is not simply a case of “if you build it they will come” — a point I explore and hammer out in my 2010 thesis. Furthermore, it takes time to attract participants. Finally, the complexity of balancing user motivations, publicity, and managing community-development means that there is a possibility of failure. For projects that need a guarantee of completion, volunteer contributions are not the most desirable approach (especially to your funder).
For more tedious tasks, where there is little or no intrinsic motivation for volunteers, crowdsourcing projects can also compensate online workers. Platforms like Amazon’s Mechanical Turk exist for this purpose, emphasizing micro-payments for piecemeal tasks. The goal is to split up human labor tasks into small parts and treat their completion in the same manner as one treats computational cycles. Turk is human intelligence with the convenience of machine processes.
Turk is great for projects with tedious tasks, where many people conducting a few tedious tasks is often more humane that asking a few people to perform many. It is also useful when qualitative labour is needed quickly. Finally, for participation-critical tasks, where you need a guarantee of concluding the task, paid crowdsourcing is much more reliable than volunteer crowdsourcing.
However, there’s another side of paid crowdsourcing.
First, there are potential detriments to the work at the heart of the crowdsourcing. It can grow costly, and there is an additional, financial motivation for users to game the system.
If [Tom Sawyer] had been a great and wise philosopher, like the writer of this book, he would now have comprehended that Work consists of whatever a body is obliged to do, and that Play consists of whatever a body is not obliged to do.
Mark Twain
There is also a notable difference between extrinsically and intrinsically motivated work.
For example, (Mason and Watts 2009) found that users given a word search on mechanical Turk would find more words when they had less financial incentives to do so. Panos Ipeirotis has blogged a concise, capable literature review on this topic.
While the balance of benefits and detriments to your project will tilt in one way or another depending on your needs, there are also possible detriments to users. In treating workers as computing cycles, as Amazon encourages, there is a risk of dehumanizing workers: forgetting that you are in fact working with people. This may result in unrealistic expectations of quality, low wages, or unethical work. It’s a possibility that Jonathan Zittrain calls digital sweatshops.
On Turk forums, one can see countless posts of workers complaining about shady or unfair requesters, as well as posts of requestors overly standoffish. In just one of many examples, a requester put up badly designed tasks and then withheld all the payment when workers did not perform to their expectations, showing surprise when they started hearing from dissatisfied workers. Their surprise was stunning, as their withholding of payment for work was not in malicious intent but rooted in a belief that payment simply hadn’t been deserved.
Examples like this are a risk, but not a necessity. When volunteers crowdsource, they have to be treated with decency because their very participation hinges on the requesting party’s sincerity and tact. When paying people however, and doing so in a system that keeps them at arm’s length, it is the duty of the requester to remember such qualities.
II
Last year, I set out to utilize Mechanical Turk for a literary crowdsourcing undertaking. My purposes were pedagogical: to demonstrate paid crowdsourcing in the humanities while foregrounding the issues that requestors must be aware of when doing so. To this purpose, I crowdsourced a modern colloquial update to Jonathan Swift’s A Modest Proposal (1729), hoping to instill the satire with some of its original shock though a naive rewrite by semi-anonymous paid workers.
Swift is not a new object of study in DH. In 1967, Louis T. Milic published a computational text analysis of his work in A Quantitative Approach to the Style of Jonathan Swift (Worldcat; see also, review in Computers and the Humanities). Patricia Köster later emulated this study in a 1971 attempt to identify unattributed works suspected to be by Swift based on their style.
In his famous pamphlet, Swift describes the plight of the impoverished in Ireland before offering a solution: for the poor to sell their children to the wealthy for food. Swift was parodying the language of social engineering essays of the time: ones that look at the big picture of society’s woes while disregarding the humanity of the people involved in these “solutions”. Swift began in a familiar style, casually progressing into its unspeakable proposition.
Yet a modern read of A Modest Proposal as a familiar satire is different from a completely naive read. Clark and Gerrig (1984) argue that the strength of the pamphlet lies in readers first taking Swift seriously and gradually realizing the pretense, comforted that others would not realize it at all. However, it is difficult to mistake such a famous work as a serious essay today, dulling some of the edge. Furthermore, the language, while arguably effective in its cold formality, might be inaccessible to some today.
In creating a colloquial crowdsourced rewrite, I tried to reinstate some of the shock from the original, to see if the satirical effectiveness survives. At the same time a satire needs a purpose, not simply to shock. The choice of Swift was intentional, paralleling paid crowdsourcing’s worst-case scenario with Swift’s jab at cold, dehumanizing social engineering solutions.
In this way, part of the audience for the rewrite was the workers themselves. Workers were removed from the context, rewriting the pamphlet sentence-by-sentence. What would they think, looking at this sentence written in such unassuming prose and deciphering it, only to realize the terrible proposition that they’ve written? Would they realize it?
The task was done in two steps: rewriting and voting, adopting Bernstein et al’s Find-Fix-Verify crowdsourcing pattern. “Find-Fix-Verify splits complex crowd intelligence tasks into a series of generation and review stages that utilize independent agreement and voting to produce reliable results. … This process prevents errant crowd workers from contributing too much, too little, or introducing errors into the document.” (2010) Bernstein and his colleagues applied this pattern to a series of crowdsourced editing tools, which allow you to outsource editing from Microsoft Word to Mechanical Turk workers.
In the Swift project, the Fix and Verify steps were adapted. In the first stage, rewriting, the original essay was divided into individual sentences. Each worker was given a single sentence and asked to rewrite it in simple English. In order to deter cheating, character-limits were included as artificial restraints. The limit provided was 140 characters: the length of a Tweet or text message. This resulted in an additional layer reflecting our modern colloquial language, and was inspired by the co-founder of text messaging, Friedhelm Hillebrand, who came up with the character limit after a series of informal experiments convinced him than any single thought can be conveyed in 160 characters.
For each sentence in a Modest Proposal, three different users were tasked with rewriting it. Once three rewritten sentences were collected, that sentence was sent to the next step: the voting stage. In this stage, workers were presented with the original sentence and the rewrite candidates. Here, they voted on the rewrite that best embodied the original line. Once again, multiple workers voted, and the candidate sentence with the majority of votes was chosen.
So, how exactly did it turn out? Below are two rewritten lines. The first line does a fairly good job of creating a simplified sentence. The second example, however, is likely too bare. Nonetheless, there’s something about the plainness of the rewritten sentence that I find particularly compelling about example 2.
Example sentence 1
Original:
For this kind of commodity will not bear exportation, and flesh being of too tender a consistence, to admit a long continuance in salt, although perhaps I could name a country, which would be glad to eat up our whole nation without it.
Rewritten:
This commodity can’t be exported as the flesh is too tender to remain salted for long, but I know a country that would eat ours without it.
Original:
There is likewise another great advantage in my scheme, that it will prevent those voluntary abortions, and that horrid practice of women murdering their bastard children, alas! too frequent among us! sacrificing the poor innocent babes I doubt more to avoid the expense than the shame, which would move tears and pity in the most savage and inhuman breast.
Rewritten:
My idea is good because it stops the sad practice of abortion, which is murder and should break your heart.
Generally the workflow was effective. Tasks were completed quickly and effectively. Once everything was set up, most of the tasks were completed within 3 hours, though some tougher sentences straggled for longer. The main issue was that voting sometimes favored sentences that had more words in common with the original rather than those which were more drastically simplified. Still, the rewritten essay kept it’s cold feel: of a writer making a argument that by all means appears persuasive, except for the fact that it forgets the humanity of the pawns involved.
Since the format matched, I briefly began publishing the essay on Twitter. However, while it was interesting to consider the effect of seeing the isolated context of the sentences in a Twitter stream (see my original post), ultimately the Twitter account got to some lines that I could not stomach pushing out to a public feed.
Crowdsourcing with volunteers forces researchers to consider the crowds and offer them a satisfying intrinsic reward, but once compensation is introduced users simply become workers. While not inherently bad, it introduces a slippery slope to an exploitative relationship. Through irony, A Modest Proposal criticizes the dehumanization of the poor. It is only appropriate that nameless workers would give it a fresh layer of discomfort.
With thanks to Stan Ruecker for suggesting that this project be shared.
