Excellent advice about succeeding in a PhD program

Sociologist Fabio Rojas from the OrgTheory blog has put together a wonderful guide to being a successful social science graduate student, called the Grad Skool Rulz. This is available on demand for a worthwhile $2: http://orgtheory.wordpress.com/grad-skool-rulz/

Would that I had known all of this information up front - it would have saved considerable time and tears.

Status Identity Issues in Crowd Sourcing: Transaction Costs or Reputation Effects? (mTurk vs. CrowdFlower)

Another interesting point about crowd sourcing - what does identity have to do with it? Now, due to the strong presence of scammers, many of the researchers in this area think that reputation is the best way to control quality with micro tasks. (E.g. http://behind-the-enemy-lines.blogspot.com/2010/08/reputation-integration-and-future-of.html) But I'm a contrarian sociologist, so I am not so sure that I would necessarily agree. To see why, I first need to point out the issues of social standing and transaction costs that factor into this labor market.

First, I started to think about transaction costs as a reason why micro tasking was popular for over qualified people like college students to make very little bits of money. (Leaving out the 'game-like quality' arguments here.) For example, as a PhD student, I can always go work part-time at something, but getting the job, training for it, and then settling into a routine of work that won't interfere too much with my research takes time and effort -- indeed, it would probably require a lot of searching with the current economy. On the other hand, I can work in little bits and pieces with Mechanical Turk, earning little money, but in a way that doesn't require a lot of startup costs. I can tell you from experience that motivation is there, and low barriers to entry (transaction costs) make it easy to sign up with mTurk and start working right away. Who doesn't already have an Amazon account?

However, should we also think about this in terms of a status identity-driven distinction? It might take a lot of effort to go and get a reasonable part-time job, but maybe the issue is also that it just feels wrong or demeaning to take a low skilled position. However, with mTurk, who of my friends are going to know that I'm doing this for extra money? Nobody, unless I tell them. Thus, this is work that the overqualified could do, easily, but without the negative reputation effects that could occur if, say, your friends knew you were working in fast food management instead of doing corporate law. Working on mTurk -- say if I were unemployed or staying home with kids or between jobs or just part time -- wouldn't necessarily have the same status identity hit. I could rationalize this to myself as "just earning some extra money" and it would be unlikely to generate negative reputation effects with my peers, because it is unlikely that they would find out I was doing this unless I told them. 

And that, my friends, is why I think we can definitely find overqualified workers lurking on Mechanical Turk. (So long as they don't get burned too quickly by scammy requesters.) In other words, maybe we should be asking "to what extent are 'too good' workers using mTurk as a way to generate some extra money in ways that are a) easy and/or b) not identity-threatening/demeaning?" (This is also a nice explanation for why you might see overqualified people doing temp work, together with the ability to move into a good FT position...)

Testing Status Identity vs. Transaction Costs
A nice experiment to test this would be to see how well or quickly the same people completed a boring task using CloudCrowd vs. mTurk -- experiments anyone? The way that CloudCrowd works is to use a Facebook login to start working, which is what gave me the idea for this post in the first place, because I immediately went "eww.... I don't want to let my friends know I'm doing this low-status work."

CloudCrowd uses facebook and existing accounts to guarantee that people are 'real' by looking at facebook. But not only will this create the status identity issues I mentioned above, with one's friends finding out you were doing menial labor, but a corollary is that we might get in fact worse quality work from some people who would be just fine in the semi-anonymous mTurk scenario.

Here's why that might happen: for some people, requiring login with your facebook ID is just fine, and through social sanctions and reputation effects, there will be an incentive to provide high quality work. I imagine this works just fine for some workers. Using the issue of status identity, however, I can also see a quality-destroying incentive here: if I don't want to show my friends I'm doing this, I might make up a new facebook ID, and then with that "worker login" I would probably feel less pressure to do quality work. Heck, if I'm going to start creating fake IDs, I may as well make a couple of them, and maybe have a throwaway bank account as well, to make it easier to abandon if one was flagged.

So, to be academic about it, I would expect to see this the two-tiered quality effect, and it would become my testable hypothesis if I were going to do this kind of research. (But I'm not, so it's fair game, people!) Another alternative might be that overqualified workers would just not do this type of work if they had to go through facebook or another social media source that was linked to their visible personal profile.

Well, now I really need to stop messing around with mTurk and work on my real research...

Mechanical Turk and Artificial Reviews

So I've been very interested in the whole mTurk phenomenon, with crowdsourcing small tasks. I decided to start doing tasks myself in order to see what things were best for this type of work, but frankly, I was appalled by some of the uses. I've seen solicitations for newsletter signups, which are bound to get someone spammed. I've seen solicitations for "please post 3 comments to the following blogs" to artificially get a conversation going. I've seen requests for Facebook follows/likes/etc., requests to break CAPTCHA's, requests to "download college material from this site," and "fill out a free credit report online." (If that last one isn't a serious identity scam, I don't know what is.)

Now I am able to see that this is part of the engine behind artificial website rankings and reviews, to pump up garbage, and game the ranking metrics. Here's a terrible one for academics: just today, I saw someone use it to artificially bump up his SSRN downloads, which I happened to report to both SSRN and to mTurk. I also 1-starred the book who's PR company was soliciting 5 star reviews on Amazon.

This brings me to an interesting point, though, because a whole community of people has evolved around mTurk, and some of them have moral stances, flagging questionable tasks like this. (See http://turkers.proboards.com/) The sad thing is though, not everyone has this kind of moral compass or perhaps they need the money/just don't care. I can understand those motives even if I don't condone it. So I have to say that this experience has really changed my view of what is "real" on various ratings and review sites. We might think those things are easy to moderate out, but some of the requests are really so specific as to require "at least 200 facebook friends" or other qualifications, and of course the sanctioning power of reputation effects doesn't matter if someone has already created an artificial online presence or several. (See what I mean by googling up "black hat SEO" where people will proudly talk about creating multiple artificial identities and "even using them to talk with one another.")

Ok, that last one is a little involved for the mTurkers, and could still happen even before the internet. But if there are people who will post bogus reviews for $0.05 each, at least there are a few people who will probably feel the moral outrage at this kind of thing, like myself, and post a negative comment on some of those same sites. At least that's what I'm hoping.

But I can easily see how this is going to become an increasingly difficult business to check and to moderate. It is going to require new and perhaps more sophisticated methods of doing marketing research, for example, because it is just so easy to post a "help me vote for funding for X medical disease online" task on mTurk--I actually saw this one--and if one just trusts the survey results, you might not be getting anything near a real sample. I'm just waiting until I see one along the lines of "buy one share of this company stock."

That said, mTurk does have some great potential, and I'll have more later on the useful and interesting tasks you can do with mTurk. However, I have come to the conclusion that if Amazon wants to police this better, they need to moderate requestors better (ala Amazon stores), and/or create an "approve this mTurk task" as an ongoing HIT to clear each and every request, which would work nicely with new or otherwise questionable requestors. I am sure that if they police things, new blackhat sites will spring up, but since Mechanical Turk is associated with the Amazon brand, they do need to be careful not to let it be the wild west, or they could suffer their own negative reputation effects. 

 Read More
A nice academic blog on Mechanical Turk: http://behind-the-enemy-lines.blogspot.com/ 
Interesting labor regulation issues from the crowdflower blog: http://blog.crowdflower.com/2010/06/regulating-distributed-work-part-three-why-its-a-good-idea/

Preparing for the academic job market

Useful online powerpoint presentation from UPenn.  Kind of general, but good to get your feet wet.
http://www.vpul.upenn.edu/careerservices/gradstud/resources/AcademicJobSearch/player.html

More Suggestions for Your First Articles as a Graduate Student

Writing articles. This can be a pretty stressful task if you are in a research oriented program and not working on a project where you are expected to publish with your advisor or other faculty. I envy other graduate students who have had that kind of excellent apprenticeship. On the other hand, you already know how to write, so it is possible to learn how to write scholarly articles. 

One of the key things you should know is that publication is usually a very long process, so you should push yourself to start on it early in your graduate career. Unlike blogs or newspaper articles, going through one or more rounds of peer review will basically ensure that your work will be properly vetted. In fact, another advantage that of this longer process is that I have realized that it allows me to let research papers go out before I actually "feel ready" or feel that the writing is perfect. With long time horizons, there will still be plenty of time to make sure that I'm going to be happy with the final product. (Besides, almost all articles will require revisions to respond to the peer review comments.)

Another key problem is having data to write about. As a social scientist, it is usually not sufficient to just write a theory paper or a literature review, meaning that some kind of data must be processed and analyzed. You might be gearing up to do a large survey or international research for the dissertation project, but that doesn't mean you should wait to find publishable pieces. It would make sense to use a smaller project or class assignment for this purpose instead, and while that is easier for quantitative researchers to collect some useful bits of data, qualitative researchers can do this as well. One of the easiest ways to get some reasonable empirical data is to just do a quick IRB and a local participant observation project on some phenomenon; even without interviews, you could get something interesting with a reasonably low hassle or time factor. (In my own case, I went to Chicago auctions in the beginning and watched bidding, which was helpful for later observation on my larger project.) Just make sure to apply for IRB clearance beforehand -- although this is not required for a class project, it is required for publishing with human subjects research, cannot be applied for retrospectively, and it is another good thing to practice.

Once you have written a draft paper, based on data, for a class or otherwise, it might be a good idea to submit it to a conference, and/or present it at a scholarly workshop. This will help you to get some more input on the theory and methods of your study, and be sure to write up comments soon after so that you don't forget them.

Now comes decision time, and you should critically assess the level of quality of the work. Be honest with yourself about quality and think about the current paper length and most interesting findings to focus on. I don't advocate putting out sloppy research, but even if you realized that you had some major flaws, most studies should have some interesting parts that you could focus on, and feel free to round file the other parts. (Or save them for later, because you never know -- however, remember that you will need to keep up your IRB clearance if you later want to reanalyze human subjects data with identifiable information, otherwise you would need to strip out any names and identifiers, deleting/shredding any files that were used to keep a real names list of subjects.) Thinking about the parts of the research to use can be stressful, which is also why taking the work to a conference or workshop for peer feedback is helpful - helpful to help you realize what parts of your findings were interesting to other people and the types of related literature that you will need to draw from. 

Once you have a reasonable paper, with some interesting parts, and not too long, then you need to find a potential publishing venue and revise the work for submission. I think revision is some of the hardest work, but it does get easier, and it certainly improves the quality of the findings. This is also the stage where you should start on the focused literature review. One of the typical problems I see with student papers is when someone has done the literature review for a class, but then the actual findings of the data point to a completely different theoretical area. Don't fall into the trap of being lazy about this problem, because it can cause rejections, and besides, you might be relying too much on the theoretical literature and not focusing enough on your actual data. Fixing this problem by cutting down and refocusing the literature review is part of the process.

A typical timeline for a paper with a small dataset would be 1) gathering data over a few months, analyzing and shaping the paper, 2) revising this for a workshop or conference, and then presenting, which can add a few months to a year, 3) Respond to peer comments from the presentation, pick a potential venue and revise for submission over 2-3 months. Depending on the journal, you might get a rejection, in which case you return to step 3, incorporating comments and picking a new journal target. When you do get a revise and resubmit, 4) you should respond to reviewer comments systematically, reshape the article to make sure that theory and data match, and tighten it up to meet the word count. Make sure that you really pay attention to your writing at this point, because it may be the last stage where you can make major revisions. Then 5) you will get a page proof, where you can review to check for any glaring errors and make a few minor fixes. Hopefully this gets you to 6) scheduled to publish, although that could be an unknown time in the future depending on how long the article queue is for the journal. 

Clearly, we all want major articles in major journals, but I think that it is less paralyzing to start out by focusing on subfield journals with shorter articles. Although I had a... disappointing experience with one such journal, and ended up pulling the manuscript, in other cases, I found that working with these smaller journals was helpful in terms of time, getting great feedback from the editors, and I also had helpful comments from reviewers. These focused journals won't have the larger readership of a generalist journal, but it is a nice way to start building your scholarly reputation. (Another target would be a graduate student journal, but subfield journals are probably going to be higher impact.) 

In order to focus on learning the mechanics of writing up a research article, an easy filter would be to look for journals that have a 5,000 to 8,000 word limit. That type of limit is also an easy way to figure out whether you have "enough" for an article - more than likely, you'll find out that you have way too much for just one of these. Throughout this process, you should expect to do many revisions, and they can sometimes be painful, such as figuring out which parts to cut to reduce the word count, but the advantage is that you will be (or at least should be) increasing the clarity and focus of your work. 

As part of the process, one of the best tools that I've found is to use a reverse outline. This has a lot of potential uses. 

1. This is helpful for figuring out how to revise the work for the initial journal submission - diagram article structure from sample papers of the journal where you want to submit, and use that as a template. This will be a nice mechanical exercise, but will help you to become acquainted with the stylistic conventions of the journal and how other people write about their research. Depending on the length of the article, I start with a first pass of the different sections, then eventually do a few on a paragraph by paragraph basis, asking "what does this particular paragraph do in terms of the argument?"

2. In order to focus your writing, do a reverse outline on your own paper. This will be helpful to figure out which pieces can be cut, and if you need to rearrange for clarity. 

3. When you finally get to an R&R, many months later, you can diagram your own writing as a way to figure out what the heck you were talking about, and show you where you need to make revisions.

4. When you feel stuck -- do this, either diagramming a different article for the journal or doing this with your own paper. I find that it almost always helps me to move out of that terrible place of not knowing quite what to do next with the article, or feeling lost in a sea of details. 

A note about reviewer comments: regardless of whether you were accepted or not, a good practice is to make those into a to-do list -- you don't have to do them all, but this helps to depersonalize them and make them into action items. It is also good form to to do this for a revise and resubmit, because as you do the edits, you are making a list of changes. Then you can include this list and your comments about which changes you did or didn't do, to include with your revised manuscript. It introduces a nice element of accountability into the process.  

My EZ solution to get people to pay for online news

Sigh. I've been very disappointed with garbage SEO content from web searches lately, which only reaffirms my feeling that there will have to be a new type of shakedown with web providers, rankings, and so forth. (Lame SEO as the new spam? Maybe.)

The issue comes back to my previously mentioned problem of needing online curation methods. Which websites do we trust? Which provide the best* information?
*Yes, that's subjective, and I know plenty of people would take TMZ over NYT, but still.

 So how does this relate to the issue of newspapers and online content? Well, if you break the problem down to essentials, the reason you'd pay for a paper newspaper vs. just hearing your neighbor tell you about the warehouse fire down on the docks is an issue of quality and reputation. You pay for reputation. (Let's exclude the entertainment-purposed newz lite for simplicity.)

Online news provision changes the game though, now information can spread, it can travel, and it doesn't take an economist to figure out that people don't like to pay for things that they can get for free, or where they can get acceptable substitutes for free. Question is, how do you then monetize high quality news online then?

Well, I've got a solution. In an online age, where information travels quickly, monetizing the news is difficult, because there are so many alternatives. But you can certainly monetize community participation. Let's think about this a bit more...

What's the difference between reading the news at the New York Times and paying for some content, vs. finding an AP story elsewhere for free? Well, NYT is reputable, and hey, they don't have a bunch of flashing graphics ads that are distracting. (Yes, I would pay for no ads versions, but I think I'm in the minority on that one.)

Actually, let me change examples to a site where this will work a lot better, and use an example close to my heart (as an academic) to talk about The Chronicle of Higher Education. What the Chronicle offers, and you wouldn't find on other websites that might repost the same or similar content, is the comments section. Take the following randomly selected story about professors being required to be present in their offices for a minimum number of hours per week here: http://chronicle.com/article/Professors-at-U-of-North/124308/. As you might find from a quick skim through the rest of the website, the Chronicle requires subscriptions to view some articles, but not all of them. However, almost all articles now allow comments, and with many things, you get quite a healthy argument going under the fold.

Which brings me to my point - the Chronicle may be charging for some content, but I bet they would convert a lot more people by switching to charging for the right to participate in the conversation and making comments. Yes, moderation issues and TOS issues will probably have to be worked out to disallow spam, and of course with an audience of PhDs and university administrators, you usually get very high quality comments, which won't be the situation for many other news providers. But these caveats aside, I think it would be pretty effective to charge readers for the right to comment, rather than the right to read material. Basically, we've got it wrong - forums used to be where you could get free information, but maybe the future of news is that you only get to participate in forums if you pay, whereas the news itself is free.

Of course, this isn't a perfect solution, because it could have negative effects like shifting news providers toward comment-worthy or controversy heavy stories, it would make active management of forums essential, you would miss out on intelligent comments by people who wouldn't pay, it might be undemocratic by not allowing poorer people to participate,* the commentary might drive away some potential readers, and who knows, I could be wrong about the majority of people being willing to pay to comment. So it's probably not the panacea, but it would be an interesting experiment to see how well this would work to monetize high quality news and reporting, vs. the paywall schemes that are being rolled out lately.
* This could be mitigated by allowing editorial selection of some comments for free, the same way letters to the editor are sorted and chosen. 
** Also, we do still need a viable micropayments system to capture the nonsubscriber / infrequent visitor issue, which decreases the 'barriers to purchasing' in technical terms. Hello, Google? Paypal? Why isn't one of the big boys working on this?

Research organization using OneNote

I think I can safely say that for the past 3 years I've been looking for the ideal way to organize my research materials. By the time I went to do my fieldwork in New York, I had already started on a big paper collection, and since that time it has only expanded. (Yes, how many people do you know that fly off with a suitcase of papers and journal articles. I am ashamed to say that has been me.)

In starting on the ethnographic work in earnest back in 2008, I did a survey of qualitative analysis software, and after testing Atlas.ti, MaxQDA and nVivo, I picked nvivo to work with. (I later ran across Qualrus, which some people like, but I had already bought nvivo so I didn't review it particularly.) Now nvivo is a very powerful tool for qualitative researchers and it definitely helps us to be better organized. But it wasn't quite the holy grail that I had been seeking for research organization, although I definitely will keep using this type of software.

I found that I wasn't using it quite as often as I would have imagined for a key reason: it takes a long time to load in XP, and having said no to Vista, I'm currently waiting to have the time to move over to Windows 7. So it wasn't the easy interface that I could just pull up to make notes with the same way as writing in a physical journal like I do when I'm out interviewing.

Now, to take on that function, I tried a few different things in the last few months - evernote, Mendeley, OneNote, back to Mendeley, and now back to OneNote. Now that I have been working with OneNote for a while, I have to say that I absolutely love it. It apparently was started as an initial technology for tablet PCs, and has great integration there (so I hear) but it is so much more. I suppose this is the kind of epiphany that mac users had when they stared using Devonthink, but for those of us who are on PC, there wasn't this great kind of option.

You can do an online search yourself and see all about onenote and why people like it, but let me tell you why I like it for research.
1. It's intuitive and easy to use
2. There are many more useful features once you get going with it
3. Easy searching in it
4. Easy to draw
5. Stays pretty organized by default
6. Easy way to hold all those notes and scribblings

Now, I won't say that there are no problems. It isn't great with PDFs, and does this silly 'print to OneNote' thing for those. If you make a table, you can't do an autosum, as one might expect. If you draw, there's no way to group the drawing objects like in word or powerpoint (sigh). There is no OneNote mobile for blackberry.

But regardless, it is making me so happy to use this and I feel that I'm now able to put together my research notes and quotes in a way that really makes sense, and is searchable. This is definitely worth the $80 or so that I paid for it, and to be quite honest, it should probably be included with all versions of Office, because it really is just that useful. But really, don't just take my word for it, download the trial version and give it a whirl.