A new way of ranking journals*

Journal rankings can be controversial. At the same time, the quality of a journal in which one’s research is published is generally thought to be very important for a researcher’s career, and many researchers are thus rightly concerned about it.

Here at Academic Sequitur, we came up with a new way (to the best of our knowledge) to think about how a journal is perceived by the profession. This blog post focuses on the field of economics. We start with the premise that the top 5 economic journals (AER, Econometrica, JPE, QJE, and ReStud) are really the best in the profession, on average. Then we calculate what percent of authors who published in another journal have at least one “top 5” publication. The higher than number is, the more likely it is to be a high-quality journal. In non-top-5 journals, we considered all articles published since 2018 (results are similar if we start in 2009, when the AEJs were started). For calculating whether an author has a top-5 publication, we used top 5 articles since the year 2000.

Before I show you the results, it’s important to note that one thing which can affect this ranking is the topic of the journal under consideration. If a journal’s focus is not “sexy” enough for a top 5 journal, that’s likely going to lower its ranking. Whether this is a feature or a bug I will let you decide.

So with that, here are journals that come out on top, based on the overall proportion of authors publishing at least one article in any top 5 (the rest of the columns show the journal-specific proportions).

Four interesting things about these rankings: First, the shares are really high for eight out of ten of these journals, with about half the authors having at least one top 5. To me, this pattern suggests that we definitely shouldn’t overlook the non-top-5-journals when looking for quality articles. Second, these eight journals are pretty close to each other according to this metric, suggesting that quality differences between them are not large. Third, this metric largely aligns with what I think are the general perceptions of applied microeconomists in North America, with one exception: we seem to be giving the Journal of the European Economic Association less credit than it deserves. Fourth, the rankings would definitely change if we used specific journals for comparison rather than the overall top-5 metric.

Here’s the next set of journals. Keep in mind that we didn’t perform these calculations for all journals in our database (this is just a blog post, after all). So if you don’t see your favorite journal, that doesn’t mean it’s ranked lower than these. It just means we didn’t calculate a ranking for it. But if you’d like, leave us a comment and we’ll tell you how your favorite journal ranks!

While this metric is unlikely to be perfect, it is also unlikely to be worse than citation impact measures. And its benefit for economics specifically is that it isn’t as affected by publication lags as a citation-based measure. What do you think?

When to give up on a paper

Following the publication of the post on where to submit your paper, someone asked, “How do you know when it’s time to give up on a paper?”

This is a really hard question. We put a lot of work into our papers (I’m assuming in this post that it is a completed paper) and, despite the theoretical wisdom of “Ignore sunk costs”, it’s difficult to let go of months or years of hard work no matter how bleak things look. But there’s also no magic number of rejections beyond which it’s clear that you should just give up. Here are my two cents on how to make the decision.

First, here’s a clever trick I use to make “giving up” on a paper easier psychologically – I have never permanently given up on a paper. But I do have four papers and a lot of never-made-it-to-paper-stage-projects “on the back burner”. I haven’t worked on them for years and don’t plan on doing so unless I have nothing better to do. In other words, instead of asking the hard question of “Should I never try to publish this paper again?”, ask the easier question of “Should I prioritize other projects over this paper for now?” I always have the option to pull papers out of the “back burner” folder, but lo and behold, I keep having better projects to work on and don’t think much about the archived ones.

Of course, that still leaves the question of “Should I prioritize other projects over this paper?” open. I’ll discuss three related cases where this question becomes relevant and offer some general guidance for how to decide.

#1 Your paper has gotten rejected multiple (let’s say at least five) times for roughly the same reason, you don’t think you can do anything to address that shortcoming, and you have other, more promising, projects/ideas. If that reason is “this paper isn’t making enough of a contribution” AND you’ve revised your introduction substantially in between submissions to make the best possible case for your contribution, this may be a sign that it’s time to drop down a tier (though see some discussion below on when this is a good idea). At the same time, the contribution of a paper is hugely subjective. If the only thing reviewers find wrong with your paper is the contribution, then trying another journal within the same tier is fairly low-cost, assuming your contribution is actually within the realm of what gets published by the tier of journals you’ve been submitting to. Here, talking senior colleagues is especially helpful.

If the reason your paper keeps getting rejected is something related to the paper’s data/methodology – for example, no one believes your instrument, no matter how many robustness or placebo tests you’ve added – then dropping down a tier is also an option, but is less likely to be a successful strategy. I came close to giving up on a paper because no one seemed to like the IV. I ultimately decided to keep trying though because (a) a lot of the rejections were desk rejections, allowing me to re-submit without revising (since there was no real feedback given) and (b) I believed in the instrument myself and thought we made a good case for it. After six rejections, the paper was published.

By contrast, if your paper is getting rejected for diverse reasons, it is probably good to keep trying (though in that case I would recommend taking a close look at the writing to make sure your exposition is clear).

#2 You feel that your paper would only be publishable if you dropped to a tier of journals where your current colleagues generally don’t publish, you have other, more promising, projects/ideas. (Presumably, you think you need to drop down a tier because of numerous rejections. Otherwise, perhaps you are underestimating your paper!) For better or worse, publishing in a journal that your department really looks down on is sometimes viewed as a negative. So, if you otherwise have a good chance of getting tenure at your department (and want to get tenure at your department), you may want to put the project down and move on to something else. Two of my archived papers were archived for this reason.

#3 It looks like the path to publication in an acceptable-tier journal would be painful and you have other, more promising, projects/ideas. Maybe your case is not as extreme as the two cases above: you’ve had 3-4 rejections, you feel like you may have a shot at an acceptable but not stellar journal tier but, given the feedback you’ve gotten so far, you have a gut feeling that it would be painful for various reasons. Maybe a ref said the paper is not well-written and after taking a close look, you realize that the ref is right and that the whole paper needs an overhaul (I speak from experience). Maybe you have your own misgivings about the methodology/data and feel like an overhaul there is warranted. If you have other great projects in the pipeline with a lower cost-benefit ratio, by all means feel free to prioritize them. No one said you have to publish every paper you write.

Yes, I put “you have other, more promising, projects/ideas” in every entry on purpose. If you don’t have any other projects or ideas that have a reasonable shot at publishing at the same tier or higher than what you’ve been submitting to, then keep working on publishing the paper, even if it means a major overhaul. Use the suggestions I wrote about in a previous post on what to do after a rejection. While you wait for reviews, work on new projects and ideas and if a better one comes along and your submission gets rejected, by all means abandon the project.

A final word of caution is in order. According to my scientifically constructed chart below, our level of excitement about a project is always highest at the idea stage, when the promise seems unlimited and the pitfalls and barriers to getting there are not salient. So, if you find yourself constantly putting completed papers on the back burner and picking up new shiny ideas, stop! Go back to your best completed paper and publish it (and work on the shiny new ideas while you wait for reviews). Then repeat until you have a few publications.

How to pick which journal to submit your manuscript to

There are many high-quality journals out there and choosing which one to submit your paper to can be a daunting task. Below, I offer some suggestions.

A great starting point is your manuscript’s reference section. Identify the papers most closely related to yours and tabulate the journals that published them. Think of a few more distantly related papers that may not have made it to your references and add the journals they are published in to your list. If your list is short or, by contrast, there are too many options, look at your advisors’ CVs for guidance. If you are still not happy, look at CVs of colleagues working in related fields.

Once you have identified at least five potential journals, go to each journal’s website and ask yourself: how often does this journal publish work similar to mine (in terms of subject, research methods, etc) relative to other journals on my list? The less frequently a journal publishes papers in your research area, the lower your chances of acceptance. If the journal rarely publishes related papers, you may want to try somewhere else first.

Are there exceptions to this rule? Yes. If the journal has a new editor who works in your area, your chances are probably higher than historical publication information may indicate. Browsing the editorial board of a journal can thus help you assess your publication chances as well. The best sign is if your paper cites the work of at least one of the editors in a positive light. Not only does this mean they are more likely to handle your paper, but they are also likely to view it more favorably than someone outside the field. (Unless you’ve tried to disguise a bad paper as a good paper. But you wouldn’t do that, of course.)

Next, you want to ask yourself, how quickly does each journal process the average submission and how much time do you have? If you’re approaching a milestone like tenure or if there are many other people working on the same topic and you’re worried about being scooped, you may want to prioritize journals that turn papers around quickly. Sometimes journals publish these statistics (e.g., number of days to first decision). Other times, you may have to ask colleagues about their experiences with particular journals.

Another obvious consideration is the ranking/visibility of the journals on your list and your own goals for the paper. If you are trying to get tenure, prioritize the journals that are more valued by departments where you could plausibly get tenure (my suggestion is to not put all your eggs in one basket and ignore the idiosyncratic preferences of your current department unless you have a really really good reason for doing so). If you’re trying to maximize the impact of your work, consider which journals are most respected by people in your field. These are often correlated with general journal rank, but there may be some divergence.

Finally, think about how much rejection you want to take. On average, the more competitive the journal you have chosen, the longer it will take to publish the paper and the more likely you are to receive negative feedback. I’m personally of the opinion that the rejection process can be used productively and would thus recommend toughing it out if you have time, but this approach may not be right for everyone. For better or worse, rejections are inevitable for all of us, and learning how to deal with them is part of the academic career. More on that in another post!