Today, I again used data from the literature tracking tool Academic Sequitur, this time to examine some gender patterns in publishing across fields. I took article data from 2018-2020 and estimated the share of female authorships for 38 different research fields, as determined by the field of each journal.* I excluded names that could not be classified as female or male; thus, the share female and share male add up to 1 in each case.
What are the most male-dominated fields? Mathematics barely clears 20 percent female authors, with computer science and finance close behind (or ahead?). Economics just makes it over the 25 percent hurdle and has fewer female authors than engineering. Business does slightly better, with 32 percent female authors. Archeology rounds out this group with just under 40 percent women.
The bottom half of the male-dominated scale has many fields with that are right around 40 percent female, including urban studies, neuroscience, epidemiology, health policy and pharmacology. Finally, three fields have a greater than 50-50 female representation: demography (60.0 percent female), social work (65.7 percent women), and gender studies (66.0 percent female).
Although a few research fields were excluded from this analysis for conciseness, it’s pretty clear that gender parity has a long way to go in academia in the vast majority of fields, even if we look at the most recent data.
* A journal may belong to more than one field. Highly multidisciplinary journals, such as Nature, Science, and PNAS, were excluded from the sample.
Over a year ago, I wrote a post tabulating the share of AER: Insights authors who have also published in a top-5 journal*. (The answer was 67%, significantly higher than most other journals, except those that generally solicit papers, like the Journal of Economic Literature.)
Now that AER: Insights is in its second year of publishing and has 60 forthcoming/published articles, I decided to revisit this question, again using Academic Sequitur data. The graph below shows the percent of authors that (a) have published/are forthcoming in a given journal in 2018-2020 and (b) have had at least one top-5 article published since 2000. The journals below are the top ten journals based on that metric.
With a score of 66%, AER: Insights still has the highest share of top-5 authors among journals where submissions are not generally solicited.** The next-highest journal, Theoretical Economics, is five percentage points behind. (There is some indication that the share for AER: Insights is coming down: for articles accepted in 2020, the top-5 share was “only” 60%.)
What if we condition on having two or more top-5 publications? That actually causes AER: Insights to move up in the ranking, overtaking Brookings Papers on Economic Activity.
Whether this pattern exists because AER: Insights is extremely selective or because less-established scholars are reluctant to submit their work to a new-ish journal or for some other reason is impossible to know without submission data. But no matter how you look at it, the group currently publishing in AER: Insights is quite elite.
*Top 5 is defined as American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies.
**AER: Insights would be even higher-ranked by this metric (#3) if we ignored top-5 publications in American Economic Review. Therefore, this pattern is not driven by the fact that both journals are published by the AEA.
We all know that economics is a male-dominated discipline on average. But how does the representation of women look across different journals? Armed with Academic Sequitur* article metadata (going back to around 2000), I determined the genders of 82% of all authors in the data and calculated the prevalence of male authors by journal for 50 top-ranked journals in economics.** To see how things have changed over time, I also repeated this exercise with articles that were published in 2018-2019.
Just to set some expectations: in the gender-matched dataset, 82% of author-article observations are male (80% when restricted to 2018-2019). So if a journal has, say, 75% male authors, it’s doing better than average. With that, here are the top 10 male-dominated journals, ranked by share of male authors over the entire data period.*** To be super-duper scientific, 95 percent confidence intervals are also shown, and I added a vertical line at 82.1% for easy benchmarking to the average.
So three of the top five journals (Econometrica, QJE, and ReStud) have also been the three most male-dominated journals, at least historically, with 90%, 89%, and 88% male authors, respectively. A fourth (Journal of Political Economy) also barely made the top ten, with 87% male authors. These numbers also illustrate that there’s not much difference between the #1 and #10 male-dominated journal.
Encouragingly, there are some improvements as well. The share of male authors in QJE was almost 9 percentage points lower in 2018-2019 compared to the whole sample period. JPE‘s share decreased by 7 percentage points, putting these journals in the top 5 most improved. If ranked based on 2018-2019 shares, Econometrica would be #6, ReStud would be #11, QJE would be #24, and JPE would be #28, just barely in the bottom half.
TheJournal of Finance, by contrast, has taken a small but statistically significant step backwards, with a 3 percentage point increase in the share of male authors. If ranked by the 2018-2019 male ratio, it would be number 1.
Here are the least male-dominated journals (rank 41-50). Economics of EducationReview and JHR are both about 66% male. Surprisingly, both applied AEJs are in the least male-dominated group (AEJ: Applied is 71% male; AEJ: Policy is 74%). This may be because they are newer, though it is worth noting that their overall average is below the 2018-2019 average of 80%.
Here’s the rest of the pack. First, here are journals ranked 31-40 on the male-dominated scale (i.e., next 10 least male-dominated), ordered by share male in the overall sample. AER and ReStat are in this group, with 80% and 81% male, respectively. Thus, AER has historically been an outlier among the top five on this dimension (using 2018-2019 shares, it would rank #19, right in the middle of the other top five journals).
Here’s rank 21-30, all in the low-to-mid 80’s.
And here’s rank 11-20. AER: Insights is 84% male. The other two AEJs are in this group, with males representing about 85% of all author-article observations.
These patterns do not necessarily reflect discrimination: the representation of women in a particular field will obviously make a difference here (as evidenced by the positions of macro and theory journals). I leave it up to you, the reader, to interpret the numbers.****
* Academic Sequitur is a tool I developed to help researchers keep up with new literature. You tell us what you want to follow, we send you weekly (or daily!) emails with article abstracts matching your criteria.
** Close to 1.5 percent of the initial observations are dropped because only the initials of the author are available. About 16.5 percent of the observations cannot be mapped to a name for which the gender is known. This includes a lot of Chinese names, for which it is very difficult to determine gender, according to my brief internet research. Names which can be both male and female are assigned a gender based on the relative probability of the name being male.
*** Each observation in the sample is an article-author, so those who publish in a journal multiple times will contribute relatively more to its average. Each coefficient is from a journal-specific regression. Confidence intervals are based on heteroskedasticity-robust standard errors.
**** If you want the numbers underlying these graphs, you can download the csv file here.
Part I: agricultural economics, lab experiments, field experiments & economics of education
Most of us have a sense that it is more difficult to get certain topics published in the top 5 economics journals (American Economic Review, Econometrica, Journal of Political Economy, Quarterly Journal of Economics, and Review of Economic Studies), but there is not much hard data on this. And if a particular topic appears infrequently in top journals, it may simply be because it’s a relatively rare topic overall.
To get more evidence on this issue, I used Academic Sequitur data, which covers the majority of widely-read journals in economics. The dataset I used contains articles from 139 economics journals and spans the years 2000-2019. On average, 6 percent of the papers in the dataset were published in a top 5 journal.
I classified papers into topics based on the presence of certain keywords in the abstract and title.* I chose the keywords carefully, aiming to both minimize the share of irrelevant articles and to minimize the omission of relevant ones. While there is certainly some measurement error, it should not bias the results. (Though readers should think of this as a “fun-level” analysis rather than a “rigorously peer-reviewed” analysis.)
I chose topics based on suggestions in response to an earlier Tweet of mine. To keep things manageable, I’m going to focus on a few topics at a time. To start off, I looked at agricultural economics (5.3% of articles in the dataset), field experiments (1.0% of articles), lab experiments (1.9% of articles), and education (1.8% of articles). I chose these to have some topic diversity and also because these topics were relatively easy to identify.** I then ran a simple OLS regression of a “top 5” indicator on each topic indicator (separately).***
The results are plotted in a graph below. Field experiments are much more likely to publish in a top 5 journal than in the other 134 journals (about 5 percentage points more likely!), while lab experiments are much less likely. Education doesn’t seem to be favored one way or the other, while agriculture is penalized about as much as field experiments are rewarded. Moral of the story: if you want to publish an ag paper in a top 5, make it a field experiment!
Now you might be saying, “I can’t even name 139 economics journals, so maybe this isn’t the relevant sample on which to run this regression.” Fair point (though see here for a way way longer list of econ journals). To address this, I restricted the set of journals to the 20 best-known general-interest journals—including the top 5—and re-generated the results.**** With the exception of lab experiments, the picture now looks quite different: both field experiments and education research are penalized by the top 5 journals, but agriculture is not.
Combining the two sets of results together, we can conclude that the top 5 penalize agricultural economics research but so do the other good general-interest journals. The top 5 journals also penalize field experiments relative to other good general-interest journals, but top general-interest journals as a whole rewards field experiments relative to other journals. Finally, top 5 journals penalize education relative to other good general-interest journals, but not relative to the field as a whole.
The second set of results is obviously sensitive to the set of journals considered. If I were to add field journals like the American Journal of Agricultural Economics, things would again look much worse for ag. And how much worse they look for a particular topic depends on how many articles the field journal publishes. So I prefer the most inclusive set of journals, but I welcome suggestions about which set of journals to use in future analyses! Would also love to hear everyone’s thoughts on this exercise in general, so please leave a comment.
*I did not use JEL codes because many journals do not require or publish these and we therefore do not collect them. JEL codes are also easier to select strategically than the words in the title and abstract.
** An article falls into the category of agricultural economics if it contains any of the following words/phrases in the abstract or title (not case-sensitive, partial word matches count): “farm”, “crop insurance”, “crop yield”, “cash crop”, “crop production”, “crops production”, “meat processing”, “dairy processing”, “grain market”, “crop management”, “agribusiness”, “beef”, “poultry”, “hog price”, “cattle industry”, “rice cultivation”, “wheat cultivation”, “grain cultivation”, “grain yield”, “crop diversity”, “soil conditions”, “dairy sector”, “hectare”, “sugar mill”, “corn seed”, “soybean seed”, “maize production”, “soil quality” “agricultural chemical use”, “forest”. Field experiment: “field experiment”, “experiment in the field”. Lab experiment: “lab experiment”, “laboratory experiment”, “experimental data”, “randomized subject”, “online experiment”. Education: “return to education”, “returns to education”, “college graduate”, “schooling complet”, “teacher”, “kindergarten”, “preschool”, “community college”, “academic achievement”, “academic performance”, “postsecondary”, “educational spending”, “student performance”, “student achievement”, “student outcome”, “student learning”, “higher education” “educational choice”, “student academic progress”, “public education”, “school facilit”, “education system”, “school voucher” “private school”, “school district”, “education intervention”. Articles may fall into multiple categories.
*** Standard errors are heteroskedasticity-robust
**** The 15 additional journals are (in alphabetical order): American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Journal: Microeconomics, American Economic Review: Insights, Economic Journal, Economic Policy, Economica, European Economic Review, Journal of the European Economic Association, Oxford Economic Papers, Quantitative Economics, RAND Journal of Economics, Review of Economics and Statistics, Scandinavian Journal of Economics.
Given the centrality of peer review in academic publishing, it might astonish some to learn that peer review training is not a formal component of any PhD program. Academics largely learn how to do peer review by osmosis: through seeing reports written by their advisors and colleagues, through being on the receiving end of them, and through experience. The result is perhaps predictable: lots of disgruntled researchers and the formation of such groups as “Reviewer 2 must be stopped” on Facebook.
This post is my attempt to make the world a better place by giving some advice on peer review. I have written over 100 reports, and I would like to think I do a good and efficient job (then again, I also mostly learned through osmosis, so you be the judge). Some of my advice is based on a great paper by Berk, Harvey, and Hirshleifer: “How to Write an Effective Referee Report and Improve the Scientific Review Process” (Journal of Economic Perspectives, 2017).
As a reviewer, your job is to decide whether the paper is publishable in its current form and what would make it publishable if it is not. This is a distinct role from that of a copyeditor, whose job is to scrutinize every word and sentence, or a coauthor, whose job is to improve the contribution and substance of the paper. A reviewer’s goal is not to improve the paper, but to evaluate it, even though in the process of evaluating it, he may make suggestions that improve it. Of course, it is difficult for people to completely separate their own opinions from objective facts, but the harder we strive to play the right role, the fairer and smoother the review process will be.
Your explanation of the paper’s strengths and weaknesses is more important than your recommendation. Many of us agonize over whether to recommend rejection or revise-and-resubmit. But reviewers do not know how many other submissions the journal receives or what their quality is. Even if you think the paper is great, it may be rejected because there are many papers that are even better. And a mediocre paper may make the cut if the other submissions are inferior to it. So the biggest service you can do for the editor is to help her rank the paper against the other submissions she is handling. Thus, you should aim to explain to the editor of what’s most impressive about the paper and what is lacking. The recommendation itself is secondary. When I recommend a rejection, I use the letter to the editor to outline the issues that make the paper unpublishable (there are usually 1-3), and why I don’t think they can be fixed by the authors.
In case of rejection, make it clear to the authors what the deal-breakers are.The most frustrating and confusing reports to get are ones that raise seemingly addressable issues but are accompanied by a rejection recommendation. It may seem easier to save the “worst” for the letter to the editor, but it will leave the authors trying to guess why exactly the paper was rejected. Anecdotally, the most likely conclusion they will come to is “The reviewer just didn’t like the paper and then looked for reasons to reject it”, which is how Reviewer 2 groups get formed. Of course, you should use professional and courteous language in your reports. But don’t hide your ultimate opinion about the paper from the authors.
In case of a revise-and-resubmit, make it clear to the authors what the must-dos and nice-to-dos are. Point 1 does not mean you should avoid suggestions that wouldn’t make or break publication. Many of my papers were improved by suggestions that weren’t central to the revision (for example, a reviewer suggested a great title change once). So if you have a good idea for improving the paper, by all means share it with the authors. But keep in mind that they will have at least one or maybe two-three other reviewers to satisfy, and the “to do” list can quickly spiral out of control. Sometimes the editor will tell the authors which reviewer comments to address and which to ignore. But sometimes the editor will pass on the comments to the authors as is. By separating your comments into those you think are indispensable and those that are optional, you’ll be doing the authors a big favor.
Don’t spend a lot of time on a paper that you’re sure you’re going to reject. This is perhaps the most controversial piece of advice (see this Tweet & subsequent discussions) because some authors view the review process as a “peer feedback” system. But it is not (see point 1). And, at least in economics, many of us are overwhelmed with review requests and editors sometimes have a hard time finding available reviewers. Treating the review process as “peer feedback” exacerbates this problem. If you think the authors’ basic premise is fundamentally flawed or the data are so problematic that no answer obtained from them would be credible, you should not feel obligated to give comments on other parts of the paper.This does not mean that you should not be thorough – there are few things more frustrating than a reviewer complaining about something that was explicitly addressed by the authors. But in such cases you do not need to give feedback on parts of the paper that did not affect your decision.
Finally, I’d like to wrap up with an outline of how I actually do the review. First, I print out a physical copy of the paper and read it, highlighting/underlining and making notes in the margins or on a piece of paper. Second, I write a summary of the paper in my own words (it is useful for the editor to get an objective summary of the paper, and the authors can make sure I was on the same page as them). Third, I go through my handwritten comments and type the most relevant ones up, elaborating as needed. Fourth, I number my comments (helpful for referencing them in later stages, if applicable), order them from most to least important, and separate the deal-breakers or must-dos from the nice-to-dos. Fifth, I highlight the deal breakers (if rejecting) or must-dos (if suggesting revisions) in the letter to the editor. Finally, regardless of my recommendation, I try to say something nice about the paper both in the editor letter and in the report. Regardless of its quality, most papers have something good about them, and authors might be just a tad happier if their hard work was acknowledged more often.
When I asked my undergraduate advisor for a recommendation letter to PhD programs, he replied, with genuine surprise, “Why do you want to get a PhD?” I was too stunned by his question to ask him to elaborate. In my mind, why wouldn’t you get a PhD? You get to learn more about what you’re interested in, you usually get enough money to support yourself, and aren’t more educated people more employable in general?
Since then, I have come to appreciate my advisor’s reluctance to unequivocally endorse PhDs. (For the record, I don’t think his reaction had anything to do with his opinion of me – he did write me a letter. Also for the record, I do not regret getting a PhD!). “A PhD is an expensive degree” is something I cannot say often enough. Yes, usually you do not pay for the degree directly, but the earnings and quality of life you give up for 4-6+ years are similarly important costs to consider. With that, here are important questions to ask yourself before enrolling in a PhD program.
1) Do you like doing research in the field you are considering? Most PhD programs are geared toward training researchers. And academic research is very different from research you may have done in a class. Class research projects are doable and have an easily identifiable end (aka the due date). Academic research can be incredibly fulfilling because you get to be at the frontier of knowledge. But it is also unpredictable, uncertain, and (usually) involves unforeseen and frustrating setbacks. In the course of a research project, you may discover something incredible or you may end up discarding months of hard work and starting over.
An important emphasis here is on “doing research”. I love reading about new discoveries in genetics and wanted to be a researcher in genetics when I was in high school. But then I learned more about how research in genetics is conducted and realized that being a consumer of research and a producer of research are two very different things. That example also highlights that just because you do not enjoy research in one discipline does not mean that there isn’t another discipline out there for you.
The best way to figure out if you like academic research is to work as a research/lab assistant for a professor or scientist. You will probably end up doing bottom-of-the-barrel work, but you will observe and experience how the process works, which will give you a pretty good idea of whether research is for you.
How do you find research opportunities? Your undergraduate institution may have formal programs. But it’s also perfectly fine to email professors directly and ask if they have research opportunities. You may not get a ton of responses, but you only need one. Looking for a full-time research assistant position is another good option. Finally, you can try research yourself by writing a senior thesis or independent study under the supervision of a professor.
2)How much will your career depend on successfully getting grants? I was happily oblivious to the fact that many academics’ careers live and die by whether they successfully raise money to support their research and their students. Luckily for me, in economics fundraising is optional. But in many other disciplines, applying for grants is an incredibly important part of a researcher’s career. Constantly trying to get new grants to keep your research agenda and students funded can be very stressful. Knowing what kind of funding pressures you might face is important information to incorporate into your decision.
3) What would you do if you didn’t get a PhD and where would that get you in 4-6 years? I always ask students who want to apply to a PhD program why they want a PhD (not with the surprised tone my undergrad advisor used though). About half the time, the answer makes it clear that the biggest reason is that they aren’t sure what to do next, so getting more education seems like a safe fallback option. If that describes you, spend some time researching other career options. And don’t just consider what entry-level jobs you could get instead of a PhD. Remember, a PhD is a big time commitment, so you should be comparing getting a PhD to spending 4-6 years working. Often, those years of work experience can get you far, both financially and in terms of doing interesting work.
4) How difficult is it to get a faculty/researcher position after the PhD program? If you have tried research and loved it and are satisfied with the grant funding situation in your chosen field, there’s still the harsh reality that, in many disciplines, only a small fraction of PhDs end up getting academic/researcher positions. Quite a few end up working in positions that are only tangentially related (Data Science and Finance are popular destination for math and physics PhDs).
The answer to this question obviously varies by institution, and there are no programs that can guarantee a research-based job after. If whether or not you get a PhD hinges on the answer to this question, I suggest applying, seeing where you get in, and then asking those programs about their placement records. If you don’t get into a program with a placement record that satisfies you, working for a year or two, beefing up your credentials, and trying again may be a good option.
I don’t mean to sound too negative about PhD programs. For many, including myself, the intellectual satisfaction of research and the ability to set your own course more than offsets the costs of a PhD program. But I think the world would be a better place if most prospective PhDs knew what exactly they were getting themselves into!
How do we judge how good a journal is? Ideally by the quality of articles it publishes. But the best systematic way of quantifying quality we’ve come up with so far are citation-based rankings. And these are far from perfect, as a simple Google Search will reveal (here’s one such article).
I’ve been using Academic Sequitur data to experiment with an alternative way of ranking journals. The basic idea is to calculate what percent of authors who published in journal X have also published in a top journal for that discipline (journals can also be ranked relative to every other journal, but the result is more difficult to understand). As you might imagine, this ranking is also not perfect, but it has yielded very reasonable results in economics (see here).
Now it’s time to try this ranking out in a field outside my own: Political Science. As a reference point, I took 3 top political science journals: American Political Science Review (APSR), American Journal of Political Science (AJPS), and Journal of Politics (JOP). I then calculated what percent of authors who published in each of 20 other journals since 2018 have also published a top-3 article at any point since 2000.
Here are the top 10 journals, according to this ranking (the above-mentioned stat is in the first column).
Quarterly Journal of Political Science and International Organization come out as the top 2. This is noteworthy because alternative lists of top political science journals suggested to me included these two journals! Political Analysis is a close second, followed by a group of 5 journals with very similar percentages overall (suggesting similar quality).
Below is the next set of ten. Since this is not my research area, I’m hoping you can tell me in the comments whether these rankings are reasonable or not! Happy publishing.
Finally, here’s an excel version of the full table, in case you want to re-sort by another column. Note that if a journal is not listed, that means I did not rank it. Feel free to ask about other journals in the comments.
Machine learning seems to be everywhere in economics these days. I wondered – has these been a gradual trend or is this a sudden explosion? So I again turned to Academic Sequitur data. This time, I decided to stick to NBER working papers as my data source, largely because they lead journal publications by a few years. I looked for the following terms in the abstract or title: “machine learning”, “lasso”, “neural net”, “deep learning”, and “random forest”. The graph below shows the percent and number of NBER working papers that meet these criteria over time (on the left and right y-axis, respectively).
An explosion indeed! Virtually no paper abstract/titles mention anything machine-learning related in the abstract in 2000-2014. Then we have a respectable five papers in 2015, one paper in 2016, followed by 15 papers in 2017, 22 papers in 2018, and five papers so far this year. As a percentage of total papers, the machine learning papers are small, however, making up at most 1.5% of total papers. Whether the numbers stagnate or keep skyrocketing remains to be seen!
Update: here’s how the graph would look if you also counted “big data” as indicating machine learning. Prize for first NBER paper to mention “big data” goes to “The Data Revolution and Economic Analysis” by Liran Einav and Jonathan Levin, issued in May 2013.
Forthcoming (if this post is popular): published papers utilizing machine learning!
A few weeks ago, I proposed that one could rank journals based on what percent of a journal’s authors have also published in a top journal. I calculated this statistic for economics and for finance, using the top 5/top 3 journals as a reference point.
Of course, one does not have to give top journals such an out-sized influence. One beauty of this statistic is that it can be calculated for any pair of journals. That is, we can ask, what percent of authors that publish in journal X have also published in journal Y? This “journal connectedness” measure can also be used to infer quality. If you think journal X is good and you want to know whether Y or Z is better, you can see which of these two journals has a higher percentage of authors from X publishing there. Of course, with the additional flexibility of this ranking come more caveats. First, this metric is most relevant for comparing journals from the same field or general-interest journals. If X and Y are development journals and Z is a theory journal, then this metric will not be very informative. Additionally, it’s helpful to be sure that both Y and Z are worse than X. Otherwise, a low percentage in Z may just reflect more competition.
With those caveats out of the way, I again used Academic Sequitur‘s database and calculated this connectedness measure for 52 economics journals, using all articles since 2010. Posting the full matrix as data would be overkill (here’s a csv if you’re interested though), so I made a heat map. The square colors reflect what percent of authors that published in journal X have also published in journal Y. I omitted observations where X=Y to maximize the relevance of the scale.
A few interesting patterns emerge. First, the overall percentages are generally low, mostly under 10 percent. The median value in the plot above is 3 percent and the average is 4.3 percent, but only 361 out of 2,652 squares are <0.5 percent. That means that a typical journal’s authors’ articles are dispersed across other journals rather than concentrated in some other journal. This makes sense if the typical journal is very disciplinary or if there are many equal-quality journals (eyeballing the raw matrix, it seems like a bit of both is going on, but I’ll let you explore that for yourself).
There are some notable exceptions. For example, 41% of those who have published in JAERE have published in JEEM, 54% of those who published in Theoretical Economics have published in JET, and 35% of those who have published in Quantitative Economics have published in the Journal of Econometrics. These relationships are highly asymmetric: only 13% of those who have published in JEEM have published in JAERE, only 16% of those who have published in JET have published in Theoretical Economics, and only 4% of those who have published in the Journal of Econometrics have published in Quantitative Economics.
There is also another important statistic contained in this map: horizontal lines with many green and light blue squares indicate journals that people seem to be systematically attracted to across the board. And then there’s that green cluster at the bottom left, with some yellows thrown in. Which journals are these?
I had the benefit of knowing what the data looked like before I made these heat maps, so I deliberately assigned ids 1-5 to the top 5 journals (the rest are in alphabetical order). So one pattern this exercise reveals is that authors from across the board are flocking to the top 5s (an alternative interpretation is that people with top 5s are dominating other journals’ publications). And people who publish in a top 5 tend to publish in other top 5s – that’s the bottom left corner. In fact, if you omitted the top 5s, as the next graph does, the picture would look a lot less colorful.
But even without the top 5, we see some prominent light blue/green horizontal lines, indicating “attractive” journals. The most line-like of these are: Journal of Public Economics, Journal of the European Economics Association, Review of Economics and Statistics, Economics Letters, and JEBO. Although JEBO was a bit surprising to me, overall it looks like this giant correlation matrix can be used to identify good general-interest journals. By contrast, the AEJs don’t show the same general attractiveness.
Finally, this matrix illustrates why Academic Sequitur is so useful. Most authors’ articles are published in more than just a few journals. Thus, to really follow someone’s work, one needs to either constantly check their webpage/Google Scholar profile, go to lots of conferences, or subscribe to many journals’ ToCs and filter them for relevant articles. Some of these strategies are perfectly feasible if one wants to follow just a few people. But most of us can think of way more people than that whose work we’re interested in. Personally, I follow 132 authors (here’s a list if you’re interested), and I’m sure I’ll be continuing to add to this list. Without an information aggregator, this would be a daunting task, but Academic Sequitur makes it easy. Self-promotion over!
If you think of anything else that can be gleaned from this matrix, please comment.
My recent post on a new way of ranking journals using data from Academic Sequitur (which you should check out, by the way!) was more popular than I expected. People pointed out important theory and macro journals I had missed (I’m clearly an applied micro person). So I added more journals. They also pointed out that making top 5 the reference journals may mean that the ranking reflects who is in “the club” with these journals more than anything else. One thing I will do in the future is make a giant matrix of pairwise journal relationships, so if you don’t like using the top 5 as a reference, you can use a different journal. But for now what I did is calculate what % of authors in a journal have only one top 5. This could plausibly make the rating noisier (maybe these people just got lucky), but it should reduce the influence of those who live in the top 5 club (as opposed to guests!).
Finally, someone pointed out that because AER and AEJs are linked, using publication in AER as a metric for the quality of AEJs may be misleading. So I calculated the percent publishing in top 4, excluding the AER. This metric is what the data below are sorted by.
So without any further ado, I give you the expanded and revised rankings! First, the “top 10”.
One thing worth pointing out here is that Quantitative Economics is linked to Econometrica, as is also evident from the high proportion of its authors who have published there. Theoretical Economics and Journal of Economic Theory were not originally in the set of journals I ranked, but they score high both with and without counting the AER. Overall, the rankings get re-shuffled a bit, but given how numerically close the original percentages were, I would call this broadly similar.
Next ten journals:
And here’s the final set:
How do the rankings with and without AER compare? Four journals rise by 5+ spots when AER is excluded: Quantitative Economics, Journal of Mathematical Economics, Review of Economic Dynamics, and Quantitative Marketing and Economics. And four journals fall by 5+ spots: AEJ: Micro, Journal of Human Resources, Journal of International Economics, and Journal of the Association of Environmental and Resource Economics (abbreviated as JAERE above). AEJ: Policy falls by four spots, AEJ: Macro falls by one spot, and AEJ: Applied stays in the same rank.
What if we only count authors who have just one top 5? That changes the rankings much more, actually, with 13 journals rising 5+ spots, including ReStat, JHR, JIE, JUE, and JPubEc. Nine journals fall by 5+ spots, including AEJ: Applied, JEEA, RAND, JEL, and IER. To me, that suggests that who we count matters much more for the ranking than which journals we count.
Bottom line is: stay tuned (you can subscribe to be notified when new posts appear on the bottom right). I plan to play around with these rankings a lot more in the next few months to figure out if/how they can be useful! If you want to play around with the data yourself, the full spreadsheet is here (let me know what you find).