Themes & Future Prospects - Hannah Rothstein, Baruch College, CUNY

DR. ROTHSTEIN: For those of you who are still awake, you might be wondering why is it that somebody from a management department is here in this role, since I don't have a background in education, except for all the years I spent becoming educated, and then all the years I'm still spending trying to educate my children. Due to the age range of my children, I am now in the 22nd year of parenthood, dealing with educational institutions. I think maybe that doesn't make me an expert, but it makes me a good critic.

My PhD is industrial/organizational psychology, and at this point, 24 or 25 years later, I really think of myself as a social scientist, and not wedded to a particular discipline. That being said, most of my searching experience is with APA products. And I probably wouldn't be sitting here today if it wasn't for some of the vulnerabilities that I encountered in the APA system. So, I hope I get to just chat with Susan a little bit too.

But before I proceed, I want to thank today's sponsors, because they are very important. The whole day wouldn't exist without them, and if I don't mention them now, I will probably forget, and then have to kick myself later. So, I would like to thank the National Science Foundation, I would like to thank the Spencer Foundation, and I'd like to thank the Hewlett Foundation, as well as Lisa and the rest of her incomparably talented, indefatigable staff for doing all of this.

And I would also like to thank all of the other speakers to allow me to listen to them, because it's been a real treat to hear people with so many good ideas, and so much passion for their work. And it's going to be something that I'm going to remember for a long time.

My particular interest at this point is publication bias. And what publication bias is concerned with is the worry that what appears in the literature, or what research is available to the public in a particular research area is somehow systematically unrepresentative of all the research that has ever been done on that topic.

And I wanted to lay that out, so you understand my background and interest in abstracts. So, I'm concerned that first of all, everything that is out there may not be a systematic representation of what has been done, conversation in part to have with authors, in part to have with editors.

Then I'm concerned that what actually is reported, and is out there in published articles, that is accessible to us is not an accurate representation of what's been done in those studies. And I think that I don't know if it was Catherine or maybe Teresa who alluded to it when they said well, we know we've run 50 multiple regressions and then we up with that one table that we pick.

Well, one of my worries is that table is really selectively chosen, and maybe if we saw the other 49, things would look different. So, I think as many of the speakers suggested, even when you get to look at whole articles, they are not really accurate or faithful representations of the research that has been done, and that concerns me.

And so, obviously, the next step here is well, what is out there isn't everything that has been done. What we see isn't everything that has been done about what is out there. And then I'm concerned that the abstract is not an accurate representation of even what's being said in the article. So, if you follow that path.

The reason that I'm particularly concerned with abstracts, and why I would like to see some changes in abstracts is because I view them as the front door to a lot of research. An analogy I have been using lately is that if the abstract were the singles ad, and you had to have a blind date with the article, would you really trust the abstract as your guide to accept that date or not?

And I think in most cases the answers are you know, I don't really think I'd trust that. So, we should have a mechanism that we can trust to at least be a faithful representation of what is being said in the article.

Now, one of the truest things that I have learned from my jogging partner, a clinical psychologist, is that the world is divided into lumpers and splitters. And I think we have heard a lot of lumpers, and a lot of splitters today, and you can kind of know already who's who.

Since I'm a lumper, first I'm going to state what I think we all agree on. I think we all agree on the fact that current abstracts have some serious deficiencies that need to remedied. And I think we all agree on the fact that one size fits all structured abstracts are not going to work. Beyond that, I don't think we agree on much else. But okay, two things are good.

A little bit of my concern is that I feel bad at this point, for the abstract, because at the same time we have both given it too much and too little credit. So, I think that it does, for better or worse, cue people as to what the article is saying. And I think you're right, a lot of times they are not accurate.

I'll just my own APA journal background. I don't search just APA journals. I review for three of them. I have two friends on the council of editors, both of whom I lobby a lot. And I do a lot of reviewing, and I think that abstracts -- nobody really checks to see whether they conform with those wonderful list of things they are supposed to.

And often they really have a barely passing resemblance, even in the best APA journals. And I think APA produces the best journals. I haven't read yours as deeply, although I have looked at them to prepare for this a little bit. So, I think that something needs to be fixed. Whether it's this particular form or not, I guess we should discuss that. But I think if the comparator of interest is the current abstract structure, that's just not working.

So, I think that people give abstracts too little credit in terms of what they signal about the article, but I also think that we give them too much credit if we think that somehow they are going to have this huge political effect on the kind of knowledge that is allowed, and the knowledge that is created. I don't see that happening.

I think that the editors have so many other places where they can control things, from who the pick on the editorial board, to who they get to review the articles, to the decisions they make. So, I don't see just the abstract itself as being so constraining that it's going to fundamentally change the nature of the way research is done, but maybe there is something that I'm missing there.

I'm more concerned about how popular topics or policymakers' priorities affect what gets done. Sort of the political hot thing of the day, and whether that influences research, and what we think we know. So, I think maybe we're dumping a little too much on the head of the abstract, at the same time as we are devaluing it.

I also wanted to mention I work closely with both social scientists and medical researchers. And hold onto your fantasies about medicine, because the reality is a lot messier. First of all, the number one complaint I hear from people in the Cochrane Collaboration, which is the organization that does meta-analyses of health care interventions is that practitioners don't listen. They ignore them.

They either do what they were trained to do, what the guy in the office next door does, or God forbid, and this is probably the worst, what the drug salesman tells them is the best thing to do. So, even though they do have maybe better abstracts, it doesn't seem to affect practice.

The other thing is that in our own country we talked about autonomy of physicians. Well, what about managed care? So, there are a lot of things going on that affect our fantasy about how research and evidence-based practice happens in medicine that I really think is a little less than true. That doesn't mean we should get discouraged though. I think we should plug on to make sure that we have the best and most accurate depiction of the research for whatever audience.

Now, I wanted to first thank the first panel for making my job a lot easier. I think that it is helpful that Bill and Ed admit that this is not a one size fits all. It's not going to work for everybody, but I do think it got the discussion going. If it hadn't been this, I would have been something else that comes out of the specific research area that reflects these idiosyncracies of that kind of research. It just happened to be this one, but it didn't have to be this one.

As was pointed out, it really should start us thinking about the general process of how we can improve the quality of what abstracts look like.

And I think Felice did a great job in pointing out the ways in which we can expand from this model, to a model that would fit behavior and social sciences more generally. It's interesting to hear all of this referred to as an intellectual movement and a call to arms. I rather like that.

I think that it's good that we realize that these things can be written in basic science. My own favorite journal, which is Journal of Applied Psychology has "applied" in the title, but it does publish a lot of theoretical research. And when I sort of tried to play around in my head with what a structured abstract would look like say on an article comparing theories of leadership, it seemed to work.

Now, I wonder, are we interested only, just looking from the perspective of a researcher, are we interested only in our own narrow field, or do we want to stay up-to-date in a broad area as well? Again, I here is the lumper versus splitter thing.

I want to be an expert in my particular field, but I don't want to become so narrow that I don't know anything else that is going on sort of in the broader field of industrial psychology, or even more broadly in social science research. I think structured abstracts really would help us be broader and expand beyond what Bob calls the historical accident of discipline.

So, I think one of the constituencies that we haven't talked about was all the people who live in the space between the people who are interested in that particular research question. Then we talked about the brokers. Then we talked about the users.

But what about us in the middle who just want to be informed social scientists beyond the specific questions we ask? Do we really want to know more and more about less and less? Or do we want to be able to know something about more and more? Which means we need to get back to things like commonalities of language, and ability to cross boundaries and still have some understanding about what's going on.

I think people are concerned if we have some kind of structured abstract that we'll have to reduce the length of the actual articles. Well, they're too short anyhow to contain all the information that we want. We have heard that, right? And I really think that there is a limited life left to paper journals.

So, once we have everything online, and with I think the cost and availability of memory, we should have infinite archival capabilities, and there should be something in articles that enables us to click on what's in that part of the article or that part of the text, and drill down for more information.

That puts a big burden on authors, but I think more people write too much about too little, and are probably better off writing more in-depth about what they did. Someone mentioned tacit knowledge. I think there is a lot of tacit knowledge that doesn't get communicated in the journal article format. And we should be thinking about ways that we can include that.

So, I'm definitely not thinking that people are going to use the structured abstract to replace articles in areas where they really want to go in-depth. If anything, I'm proposing that this has to be one of the set of changes to make scientific reporting a more accurate reflection of the science that has been done. And I only see that as one piece of it.

Another thing, let's try to think about this Medical Library Association, where it does have models for lots of different types of articles, ethnographies, case studies. But rather than trying to react just to this, I would suggest that people in the audience take a look at the breadth of structured abstract types that are available.

Someone mentioned that we should have study type. And then study type would then branch us into the specific headers or categories for what's in the articles. I think that's a very good idea. I think this is a lot more expansive than most people were imagining.

To me, as a research synthesis, it was only when I started reading abstracts and articles really closely that I realized how much was missing. And I think that those of you who don't like the idea of imposing more structure and more guidelines are not going to like this at all, but I would really like to see more reporting standards for the articles as well, not to constrain people into having to follow specific instructions, but more as a checklist to make sure that there are not glaring omissions.

And I think again, Susan very honestly alluded to this. But I think that editors and reviewers are really so overburdened, because of the incentive system for editors and reviewers, that even though they do a very careful job, and spend a lot of time, there is still a lot of things that slip through the cracks. And it would be nice if somehow the incentive system were changed so they could actually do a better job, and there would be less missing stuff.

But I think that one of the things about having a structured abstract that works for me is that it really helps you notice things that might have slipped by, that are missing from the article. And even if it just became sort of a checklist against which you can measure or look for these glaring omissions, that itself would be I think a worthwhile thing.

I think that Norman in this last session asked about what we might have as alternatives to structured abstracts. I like to think of them as supplements to structured abstracts instead. I think that there really do need to be indexing changes. I think again, as Lisa mentioned with Susan, that things aren't retagged when something changes. We don't have the sore eye.

And I think this happens in education. The same theory or the same intervention's name gets changed over a decade or two, so when you search you really only find part of what is going on unless you are very creative, and have a great historical sense. It would be very helpful if there were thesauri, if we knew how these related terms were derived.

People are concerned about control issues. I'm a little concerned about who decides for me what related terms are, who decides for me what related articles are. I would want to at least know how that was mapped out before going down one of those paths, because I feel that it's much more dangerous in terms of someone else making decisions for me than having a structured abstract.

I really don't understand the idea that no matter what format it takes, or no matter how much control within a discipline have over these things, it invariably has to be a straightjacket. I think if anything, it would open things up to more public inspection by more people, because people could find these things more easily. And if you can't find it, then it could be beautiful, brilliant research, but people are not going to know that it's there.

I think for those of us who are concerned with equity issues, I think we don't understand how important abstracts are to people in developing countries, who have bad libraries, who have limited resources, who really rely on these abstracts, because due to resource scarcity, that's how they decide where to put their limited resources in terms of hunting down studies. If we want to really open things up to people with limited resources, the abstracts are I think a doubly important purpose, because they really do rely on databases a lot more than we do.

To me, cumulative knowledge across social science implies certain commonalities. If the commonalities are there, then I think if we work hard, we can discover them and build these into the abstracts so they can be used to some degree across disciplines. I don't see that common abstract structure does any better job in reflecting the complexity or the problem or the article than the alternative that is being proposed. And I like to compare things to the available alternatives, not to some ideal abstract or some ideal system that doesn't exist.

So, I think that whoever talked about the healer and the grandchild, to me that was absolutely fascinating. And if I had a student who was doing research on something like occupational apprenticeships or transmission of knowledge in informal settings, even though I'm in management, I would want a student to somehow be able to find that literature, because I think that literature would enrich what they are doing.

And I don't think that under the current either way abstracts are written, or the way search engines are built, they would ever find that. And if they ever find it, I would never see them again, because they would be in the library forever. If you want to do such an exhaustive search that you are really going to track down everything under the current system, you are never going to get past that library search thing.

I'm sorry that I have never heard of Prof. Ogbu's research before. And so, I guess my question is you said that he was the most influential person in his discipline, but apparently it doesn't have a much greater audience beyond that discipline. Maybe it's just my lack of knowledge, but if he's that influential, he should have made it into the popular press. He should have made it into lay science articles. Steve Olson should have been writing about him, and he has.

And I think actually it would be really great if things like sociocultural context, background, theoretical perspective could be built in this. If I got to add to my wish list of what was going to go in an abstract, I would pick something about variables. Like I would like to know if there are predictors, what the predictors were.

I would like to know especially if they are outcome variables, what the outcomes were. That doesn't appear here, and I think that's a failing. But again, I think it's a failing of one particular suggestion. I don't think it's a failing of the whole model.

If you are concerned about again, the complexity issue and how to communicate complexity, telling people that what they should do is write op-ed pieces or press releases about their research is definitely going to make things way simpler, and there should be something intermediate. And I think that we want to try to get more complexity into the hands of people who are going to make those decisions, without making it so complex that they can no longer understand what's going on.

I don't know whether it's structured abstracts that could do that, but if we write structured abstracts that help the brokers, and they said that it did, then maybe we can actually communicate some of that eventually to the people who do policies, and policy and practitioner people.

I think Pamela mentioned that she thought that structured abstracts might privilege certain types of research, but just listening to her talk, she mentioned five kinds of things that she looks for as an editor, and I don't see why those can't serve as the five headings for the structured abstracts in her field. They were: theory, the research question, the evidence and how it was evaluated, the findings, and then the conclusions and the implications.

Just asking every author to lay things out straight like that, I think would be really helpful to people. I wish you were an editor in a journal that I submitted to, because it sounds like you put in more work to abstracts.

In any case, I think once you take the things that you expect the researcher to tell the audience, you can put those into something more of a structured abstract, and we just have to be a little more imaginative about it. I think that as someone said, evaluation has to be built into this. You mentioned that somebody in Rehabilitation Psychology did away with structured abstracts. Was that on the basis of evidence? Was that on the basis of any kind of experiment?

MS. HARRIS: I don't know the answer to that.

DR. ROTHSTEIN: Okay, because I was curious.

Professional judgment. Well, evidence shouldn't have to compete with that.

I think editors should work with authors on their abstracts. I did some informal surveys, talking to all the editors that I knew, and that's certainly not typical.

Teresa mentioned focus, methodological approach, sociocultural context, and significance. You didn't mention them as possible headings, but I'm suggesting that they could serve that way. I think that having to pay more attention to the abstract would make authors make their assumptions, context, and sort of implicit belief systems more explicit, which I think would be good for everybody.

I think the panel with the brokers were pretty positive about their use of abstracts. I'm certainly concerned about misleading abstracts, but again that's probably not a function of the abstract, as much as the effort that is put into the abstract. And I think that whatever else we decide, one of the conclusions of this workshop should be that abstracts really deserve a lot more attention and effort and finetuning than they have received presently.

They really do serve mostly as an afterthought, which is kind of sad, since they are the only thing that most people see of any article.

Steve mentioned that he kind of was excited by the fact that abstracts are often laudatory or exhortatory. I'm horrified when that happens, because I don't really think that's what should be in a scientific article.

People think that an RCT is so easy to summarize into a structured abstract. Try it. It isn't. But again, I don't think that theoretical articles, articles in my field for example, leadership, motivation, productivity could all be related.

I see abstracting as a part of a larger picture that would involve as I said, just more effort, indexing keywords, thesauri, and more historical perspective. I think that's something that editors and editorial boards could clearly have a lot of impact on, again given the fact that the incentive system would help them do that.

I think although I am part of the ERIC steering committee, so this is not a wholly neutral statement, I think that Luna's group shows a lot of courage in at least tackling this problem, even though it may not turn out the way they expected or it may not work on its first shot. I think that hopefully that as in biomedical science, once people see its advantages, they are going to all jump on the bandwagon, and you can say that you were the first one.

I think that if we moved any of these systems, there would have to be retagging, there would have to be revisiting the old stuff, or there is going to be a privileged position given to new research that is in the new forms, and that would be a tremendous historical loss.

I think that rather than looking at the medical model, which may or may not, for reasons I have mentioned, be relevant to interventions and people in sociology or anthropology might want to look at what epidemiologists have been doing, because epidemiologists on the health care side do deal with social contexts, do deal with larger issues, don't deal with interventions at the same level of physicians and surgeons do, and yet they have been able to make a lot of I think progress in this area of improving the nature of both reporting standards and abstract standards.

People who are interested in improving reporting standards and see abstracts as just a piece of the problem, as I do, might want to look at two documents. One is the Consort Statement, and the other one is the Trend Statement. The Trend Statement was developed in part by people at CDC together with some social science editors, mostly who have some health care interests, because it was developed for interventions with people with HIV. But based on the fact that when they went back to look at articles, a lot of information that everyone was sure would just be there, wasn't.

And again, my final point. I've been rambling, and it's after five o'clock and you all want to go home, but I really see the problem with abstracts is just a piece of the problem with journal reporting standards in general. But I think that if we tackle the smaller piece, which is the abstract piece, it will lead us to be able to have the tools and the strength to tackle the larger ones, so that we can accomplish the objective I think that we would all agree on, which is to enhance the accuracy and the validity of research reports that appear in the published literature.

Thank you.

DR. BRADBURN: Bob or Lisa, a final blessing? Thank you all for coming.