National Center for the Dissemination of Disability Research (NCDDR)
Guest: Dr. Margaret Campbell
Moderator: Joann Starks
May 12, 2005
11:00 a.m. EST
TRANSCRIPT OF TELECONFERENCE WITH DR. MARGARET CAMPBELL
[ Download 88KB MS Word document: transcript_51205.doc ]
[ 53 Minute Audio / 23.9MB WAV file: call_51205.wav ]
[NOTE: There is a recording gap at the start due to a technical error, but following are the written notes used to initiate the teleconference with Dr. Margaret Campbell of NIDRR. The recording started in the middle of her first sentence.]
JOANN STARKS: Welcome to everyone, sorry for the late start, but Margaret is here now. The reason we are having this call is that Margaret is leaving NIDRR. She has a wealth of knowledge about such things as outcomes planning, logic models, and performance reporting, and she is willing to give us a chance to "pick her brain" before she leaves. She is going to an institution that has a NIDRR grant, so she will still be a close colleague and we are glad for that.
This call is typical of an activity that can be scheduled as part of a "Community of Practice" that the NCDDR is sponsoring for NIDRR grantees. A Community of Practice is a group of people who share a passion-for example-disability and rehabilitation research, and by sharing knowledge and experiences, can help each other to increase their knowledge and thus advance the field….the "practice." We will be following this call with more information about the NCDDR's Community of Practice and will be inviting you to join with a core group of grantees that has been working to organize the Community.
Margaret, do you want to say a few words first, or just start taking questions?
[ START TRANSCRIPT ]
MARGARET CAMPBELL:…whether you want to characterize it in terms of, some people say use as an umbrella term "performance assessment," other people use "outcomes planning," some people use "accountability for results" as sort of the umbrella concept.
Whatever umbrella term you use, it really is a fundamental experiment we're all part of. And it has a lot of implications, it's changing a lot of aspects about the way we work, both in terms of research administration as well - and research administration evaluation - and the conduct of research, conduct of research and development activities.
So we are really all part of this and we are really all partners in it, and we are learning as we go. The reason I was a few minutes late is, as some of you know and as we've talked about, we are getting ready this summer to launch the pilot phase of our - what will become our annual portfolio assessment expert review process.
And this process is replacing the old summative review, and it's different. It's different than summative review, too, which was focused on individual awards. This is really focused more on the assessing the overall quality of outputs as well as accomplishment nuggets or outcome nuggets at the portfolio level, but it is based on the contribution of individual awards to that overall portfolio.
And this is a huge undertaking and something that is cutting edge in terms of how the government, federal government, R&D agencies, are going about trying to implement performance measurements and implement performance assessments.
So, I just think, you know, that we're all part of something very new here, and I think the idea of a community practice is perfect because there's so much that we need - we need higher levels of exchange in order to keep up with this experiment in this revolution.
So, I think I'll just stop there and say that when - I'll just explain that I'm going to answer any question. This would be any question related to - let's just have some boundaries here, any question related to outcomes planning, or performance reporting, performance assessment.
How about that? Any questions related to those topics? And of course that would include logic modeling also.
KRISTA KUTASH: Margaret, this is Krista in Tampa. Can I ask a pretty big philosophical question to kind of start us off here? I guess I, too, am very amazed at how the performance measurement system is pervasive across a lot of federal agencies. In fact, I was just doing some performance measurement for some other federal agencies in looking at the consistency of the message that we really do need performance measurement to make sure we're doing the best quality work as possible.
But let me ask about the other side. I'm very interested in your perception as we all do our performance measurements and get our nuggets together for accomplishments and impacts, can you share with us how an agency such as NIDRR goes about collecting those nuggets, those impacts, in a whole to help advocate for the necessary growth in disability research.
How does an agency such as NIDRR strategically plan to use those nuggets to advocate for its mission?
MARGARET CAMPBELL: Well, the first half of your question - is this Krista?
KRISTA KUTASH: Yes.
MARGARET CAMPBELL: OK. Hi, Krista. First half of your question I can answer. Second half probably goes beyond my pay grade, or maybe beyond what we've figured out yet, to be more honest.
How we collect them? It's exactly what we're engaged in now. We have designed a supplemental information form that we will have to use this first year of the pilot phase of the portfolio assessment.
We have to use the supplemental information form, unfortunately, because the current Web-based performance reporting system that you all use does not contain this kind of data element. So we have designed - in fact, that's just what I was working on finalizing on the telephone, an instrument to collect accomplishment nuggets.
And it is - we are experimenting with how to collect these. We've pulled together, you know, the best that I can find from other federal agencies and our own experience from formative and summative review, briefing books on how to do this, and we have designed this supplemental information form.
It's a pilot test of it. We will use the results of that to shape our redesign of the Web-based annual project performance reporting form so that next year, 2006, when you do your continuation report, you will encounter a whole different kind of form that will ask you for your outcome plans and it will guide you through all the data elements necessary for that.
And it will also ask you to report on your accomplishments, and it will take you through a whole series of data elements related to those accomplishments.
So, I think we have figured out at least how we're going to go forward now on a pilot basis, an experimental basis, with how to collect information on accomplishment nuggets, be they outputs or be they outcomes.
How - your second part of the question, as I understood it, was about how agencies are going to strategically use this information. Well, I think we're going to spend a lot - we are going to - our job - this is my response to you. This is not official NIDRR policy, but we are going to have more and better data than we've ever had before, consistent data.
So, we're going to be able to do all kinds of analyses. For example, how many of these accomplishment nuggets relate to what impairment groups? Relate to what age groups? Relate to, you know, what part of the disability population? We'll be able to sort them that way.
We'll be able to sort them in terms of how they correspond with the domains of the long-range plan. How many bona fide accomplishment nuggets do we have in employment versus in participation and community integration, versus in technology, versus in demographics?
How many do we have by program funding mechanism? What program funding mechanisms are producing the most high quality and relevant accomplishment nuggets?
So, we'll be able to do tremendous - we'll be able to advance our analytic capability tremendously to describe A, not only what we're investing in better but what the accomplishments are better and how they distribute across all these kinds of variables.
Once we do that, you're right. Then the real tough strategic management tasks come into play. Performance management tasks come into play. How do we use that information?
Well, one thing that's clear already, and we talked about this yesterday afternoon, is that accomplish - I mean, outcomes are incremental, and I've been saying that for a year now but it is absolutely a very, very important thing and that the kinds of accomplishments - or outcome nuggets we can capture within a funding cycle are limited by the nature of the life cycle of R&D activities, right?
You first have to plan the activities, you have to conduct them, you have to produce your output, and then you - it takes a while for the outcomes to materialize, even the preliminary outcomes to materialize.
So, we are going to allow grantees to go back into a previous cycle and report on outcomes that have materialized or come to fruition from work conducted in a previous cycle. But what if you didn't have a previous cycle?
So, we have to do a much better job of showing the stages of knowledge development. That's one way we need to use this information, this greater amount of information we're going to have is we have to do a better job of showing the stages of knowledge development and how these accomplishment nuggets contribute or reflect stages of knowledge development towards what for us is really - the gold standard for us which is that we - our research that we invest in, our R&D that we invest in, and the products that result from that are contributing to changes in improvement in policy, practice, behavior and system capacity.
So it's a long answer to a very big question, but I think we have a much clearer handle on the first part of your question than we do the second part.
KRISTA KUTASH: Thank you, Margaret.
MARGARET CAMPBELL: OK. Thank you.
PEG NOSEK: I have a question, please.
JOANN STARKS: Yes. And who is this?
PEG NOSEK: This is Peg Nosek with the Center for Research on Women with Disabilities.
And my question is how will you determine the quality of these accomplishments?
MARGARET CAMPBELL: Good question, Peg. We are doing that through expert review. The way we have designed our performance measures, of which we now have 14, is that - let me get the numbers right here, we have how many?
OK, we have - how many are long term? How many are - I'm forgetting how many are long term and how many - I think it's six that are long term and five that are annual. The five annual performance measures we will assess every year, and they will be quantitative.
And they will be based directly on the information you put into your redesigned, Web-based, annual project performance reporting form. So there will be no extra assessment of those. They will be reported directly because they're quantitative.
But then every three years, every portfolio - we've divided NIDRR's entire agenda into six portfolios. Those six portfolios include employment, one, participation and community living is two, technology is three, health and function is four, model systems is five, and then we have combined capacity building and knowledge translation into six.
So we've divided the whole agenda into six portfolios. Every three years, each portfolio will be assessed via expert review. This year we're starting off with employment and technology, two portfolios, in our pilot phase of this portfolio assessment expert review process.
Next year we'll do some more, and the year after that, two more. So, it's a three-year cycle. In that three-year cycle, we will aggregate the quantitative data that we've collected every year over three years, and we will present that at the aggregated level to the expert reviewers, as well as we will report a sample of the accomplishment nuggets we have captured from your reports.
Those accomplishment nuggets and the aggregated quantitative data will be written up in what we call a portfolio report to these expert reviewers, and they will make the judgments on the quality and relevance of the accomplishment nuggets.
VATHSALA STONE: Margaret?
MARGARET CAMPBELL: Yes?
VATHSALA STONE: This is Vathsala speaking. What is the connection between the APPR, the three-year portfolio assessment, and the formative review? You said summative review is gone now, right?
MARGARET CAMPBELL: Correct.
VATHSALA STONE: So I'm a little bit confused.
MARGARET CAMPBELL: Yes, it is confusing. The APPR is the cornerstone, the redesigned - not the existing APPR but the redesigned APPR is the cornerstone. The way it's going to unfold is as follows.
You get your award, you get your award. You congratulate yourself, you're congratulated, you get your award. You will be - there will be training provided, technical assistance and training on not only how to fill out your new APPR but there will be Web-based training available through various kinds of technical assistance materials on how to do outcomes planning because your first APPR, your new redesigned APPR that you fill out, will require you to put your outcome plan in there.
Then ideally, at least on an experimental basis, you will go through a formative review. That may not - we may eliminate formative review. That hasn't been decided because these - the new APPR is going to be such a comprehensive data warehouse that we may not need the formative review process.
And the problem with formative review is that we can't do it for all awards. It would still only be done for the largest awards, which are the RERCs, the RRTCs, and the Model System.
VATHSALA STONE: OK.
MARGARET CAMPBELL: So, we are evaluating whether we're going to continue with formative review. But even if we don't continue with formative review, project officers are being trained as we speak in how to interact with this kind of data and information from grantees to give them feedback on their outcomes plan.
We will then start collecting the APPR is the basis from which we will collect the data we use for all the performance measures, both the annual and the long term. The annual remember are quantitative. The long term are all qualitative.
The APPR - so you will put in, your APPR will contain not only your plan for your outcome, your outcomes, your outcomes plan, and all the data elements that are necessary to characterize that plan and describe that plan, but the APPR will also contain a large section on performance reporting where you will tell us what's been accomplished, what milestones you've accomplished and give us information on that.
So the APPR is the cornerstone for these new portfolio assessment expert reviews. It is also the cornerstone from which we will report annually to GPRA on our annual GPRA measures.
So it is really - it is really key.
VATHSALA STONE: Thanks, Margaret. A related question then about this portfolio assessment every three years...
MARGARET CAMPBELL: Every - yes.
VATHSALA STONE: That's right . The first one is going to be Employment and Technology.
MARGARET CAMPBELL: Correct.
VATHSALA STONE: Well, how does that work - does that mean that only a project relevant to those areas will be included in the portfolio?
MARGARET CAMPBELL: Will be included this year.
That is correct. And ideally we would have loved to have put this off until we had the new redesigned APPR in place.
But we couldn't get away with that. We couldn't get away with that because it is a requirement of our PART review that we had done back in 2003. One of the so-called recommendations/requirements of that review was that we launch this process, even on a pilot basis, in 2005.
VATHSALA STONE: OK.
MARGARET CAMPBELL: So, that's why we are starting. We're calling it a pilot because it's a huge undertaking and a very complex process. And we - there are many things we need to learn from it, such as, specifically, how do you collect? What are the key data elements you need to ask for performance reports on to get adequate information on accomplishment nuggets for expert reviewers to assess?
So, unfortunately, this first year we are going to be asking who are part - grantees who are part of the employment and grantees who are part of the technology portfolios are going to have to fill out this supplemental information form, which is going to be Web-based, as well as their regular continuation report.
VATHSALA STONE: Thanks, Margaret.
MARGARET CAMPBELL: Yes.
TOM SEEKINS: Hey, Margaret. This is Tom Seekins.
MARGARET CAMPBELL: Yes. Hi, Tom.
TOM SEEKINS: How are you doing? I want to really congratulate you guys for developing the notion of reaching back for the accomplishments. I think that's really a key.
And I wonder, just to push it a little further, is there any thought of trying to collect those accomplishments or outcomes across Centers historically in any way, or is that a step maybe that NARRTC or this community of practice might do if you guys are too stretched to do that or can't?
MARGARET CAMPBELL: That's a very interesting question. Originally, Tom, the plan was that in addition - it makes me breathless to even say this, but originally the plan was we were going to have like four or five different levels of performance assessment/outcomes evaluation.
And they were going to start with formative review, which is now questionable whether we will continue that. Then it was going to go to these annual, which we are doing, the annual portfolio assessment reviews.
Then we were going to have something else called comprehensive study evaluations. And then we were finally going to have full impact analyses or outcome evaluations such as an IOM-type study.
We have - so, taking us back one step to these comprehensive study evaluations - We have already conducted one for the model systems, and it is in that context that we did go back to collect historic information on historic accomplishments.
What we are realizing, Tom, is that - and the idea was that we would do one of those for each portfolio, so to speak, every so many years, you know, every seven to 10 years, something like that.
And I think Joel Myklebust is still very interested in that idea. The problem is funding all this. You know, we're not getting an increase in our budget to pay for all these evaluations.
These come out of not salaries and administration costs. This comes out of programmatic costs. And so we're uncertain, at this point, whether we are going to be able to sustain those types of comprehensive retrospective studies.
So, I think that's maybe the best answer I can give. The one for the Model Systems, we thought it would take a year. It's taken two years. And that's reasonable. That's about what it takes to do a good job of it.
I think it's been very informative. We started with the Model Systems first because the Model Systems are coming up - the bulk of them are coming up for continuation this year and next year, and we wanted some major input into how to redesign that program.
So right now I think that explanation is the best I can give you, what are thinking was. Whether it's going to be sustainable is another issue.
DOUG WATSON: Margaret, Doug Watson, Arkansas.
MARGARET CAMPBELL: Hi, Doug.
DOUG WATSON: Hi. Can you talk to us about when you anticipate that NIDRR will be able to put out requests for proposals in a format that would fit right into the logic model? In other words, applicants would be given instructions on how to set up the proposal that will dovetail into the logic system.
MARGARET CAMPBELL: Well, you guys have got all the key questions. You're hitting all the right hot spots.
That is very much front and center in the plan. As to when we're going to have that, it's - you know, as you all know from operating Centers it's how many different sets of requirements can you respond to at the same time, simultaneously.
It's like - and so my understanding is that Dick Melia has done some work on that already but that the big push right now has been responding to our PART requirements, getting this portfolio assessment expert review launched, and getting the redesign of the APPR.
Once those things are at least in place, even on a pilot basis, I very much foresee that NIDRR is going to, next in line, turn its attention to revising the selection criteria for the initial application. But I can't give you a date.
I can tell you sort of the sequence of tasks and projects here but not a date.
DOUG WATSON: OK. Thank you.
MARGARET CAMPBELL: Yes.
MOLLY STORY: Margaret, this is Molly Story, the RERC on Accessible Medical Instrumentation.
MARGARET CAMPBELL: Hi, Molly.
MOLLY STORY: Can you talk to us a little bit about who is on the expert panel, whether that's a permanent panel or whether it's made up for each individual review?
MARGARET CAMPBELL: Good question. We are, right now, working on composing these panels. Our outside contractor for this is what used to be Conwal, what used to be CESSI, and is now New Editions.
We're having a meeting on that exact topic next week, next Thursday, and they're going to present to us a preliminary roster. We've established some criteria for those panels, those expert review panels.
We're looking for senior people. We're looking for, you know, fairly eminent senior people. I can tell you that we're - the regular types of categories, you know, in terms of constituencies, stakeholders, we're trying to fill, you know, researchers, in your case, technologists, developers, industry, consumer advocates…
In the case of more RRTCs, researchers, methodologists, obviously subject matter experts, but more senior people, the kinds of people who would not necessarily be able to give up three plus days to participate in the field-initiated peer reviews but would be available to do this.
We're designing these right now. The agendas for these portfolio assessment expert review sessions are two-day agendas. What we're trying to - what we're working towards, I can tell you, is since technology is a huge field, obviously the technology portfolio, is working on identifying three to four clusters, trying to group all the awards all into - all under that portfolio into clusters.
And we're actually going to recruit the expert reviewers to be on a cluster so that each cluster would have a set of awards that they're reviewing. And we're looking at those clusters having somewhere from three to five people each.
And in terms of whether these would be standing, we'd very much like them to be standing, but since they're only going to meet every three years, that is hard. But we're going to work towards that to try to get that kind of commitment.
What else can I tell you? Maybe I'll stop there and see if you have a follow-up question.
VATHSALA STONE: Yes, I have a follow-up question, Margaret. This is Vathsala. You know, in one of the workshops I think I heard NIDRR saying that the review panel would also be trained in logic models. Is that still on?
MARGARET CAMPBELL: Oh, yes. Oh, yes. Oh, yes, but this - you're talking about peer review maybe now?
VATHSALA STONE: OK.
MARGARET CAMPBELL: For this performance assessment or portfolio assessment, you know, certainly knowing something about logic models is essential. They have to know something.
But they're not going to be evaluating outcome plans. They're going to be evaluating the quality and relevance of accomplishments. So, when - in your case you will receive one of these supplemental information forms.
We are not asking you at this point to lay out your outcomes plan. We are asking you to report on certain kinds of accomplishments, and not the full range of your accomplishments, just certain kinds of accomplishments, without telling us your outcomes plan. Because, you know - because this is - the focus of this is performance reporting, performance assessment, not outcomes planning.
Ideally - you know, again, if we could have done this all sequentially, ideally we would have rolled out in 2006 the new APPR. We would have started first with assessing just the quality and relevance of outcomes plans and then sort of phased in performance reporting and assessment, but we didn't have that luxury.
So, it won't be as crucial for these expert reviewers to be completely up to speed on all the aspects of outcomes planning.
JOANN STARKS: Thank you.
RADHA HOLAVANAHALLI: Margaret, this is Radha from Dallas.
MARGARET CAMPBELL: Hi.
RADHA HOLAVANAHALLI: Hi. How are you doing? I had a question about the annual performance report. And I was looking at the Form 524B, which has been modified considerably. It was running a page before and now it's running five pages.
And there is a lot of quantitative and qualitative information that's required. Would you be able to throw some light on that?
MARGARET CAMPBELL: I'll try, I'll try. Yes, you're absolutely right, it's been modified. It's a new Department requirement, and the problem is it doesn't fit very well with our kind of work and with NIDRR grantees because it assumes, sort of, that everybody is working towards contributing to the same performance measures in the same sort of quantitative way.
And so it's great for state monitoring projects, nationwide projects that involve, you know, state monitoring, et cetera, but not very good for individual discretionary R&D work. Nevertheless, we were obliged to do our best this year with it.
So, we had to pass that on to grantees, do your best. And what we tried to do is, without coming right out and saying it in so many words, is to try to say, you know, "Only really focus on the publications aspect of it."
RADHA HOLAVANAHALLI: OK.
MARGARET CAMPBELL: But having said that, what we are doing is we are folding it into the redesign of the APPR. And in folding it in, we are translating it into our terms, terms that we think will make much more sense to you.
So you probably won't even know when you fill out certain data elements that those are data elements that are going to be, that we're going to analyze and aggregate and use to report for the 524B Form because it's going to be integrated into the redesigned APPR come next year.
Does that make sense?
RADHA HOLAVANAHALLI: It does. And I was just wondering if we don't have any information to present on some of those measures, would it still be OK for us this year?
MARGARET CAMPBELL: Well, I think - my colleague who can hear me who's sitting next to me could probably have a better answer there. My understanding is you do the absolute best you can.
What we want is the publications. Beyond that, you do the best you can and if you don't have anything to fill in, you don't have anything to fill in.
RADHA HOLAVANAHALLI: Thank you. That makes it easier.
MARGARET CAMPBELL: OK.
RADHA HOLAVANAHALLI: Thank you so much.
CATHY BODINE: Margeret, this is Cathy out in Denver.
MARGARET CAMPBELL: Hi, Cathy.
CATHY BODINE: My question is what do you see that NIDRR is doing, facilitating or fitting in with this whole notion of communities of practice?
MARGARET CAMPBELL: I don't know that I'm the best person to ask that question of. I think the best person would probably be Ellen Blasiotti or Dick Melia. So I don't think I'm the best person.
I think NIDRR's certainly not seen itself as sponsoring it, but I think NIDRR's, you know - I think Dick Melia's very keen on the idea of communities of practice as is my colleague Bill Schutz, as am I, as is Ellen Blasiotti, but I don't think that they see themselves - I don't think they see NIDRR as taking this on and sponsoring them.
That's probably as far as I can go on that.
JOANN STARKS: But we are hoping, Margaret, that you're going to join our community of practice once it gets established.
MARGARET CAMPBELL: Oh, I look forward...
JOANN STARKS: In your new position.
MARGARET CAMPBELL: I look forward to that. I look forward to that. I sort of see it as changing sides of the equation.
BRENDA CAVANAUGH: This is Brenda from Mississippi State University RRTC on Blindness, Margaret.
MARGARET CAMPBELL: Hi, Brenda.
BRENDA CAVANAUGH: Hi. I was wondering if the peer reviewers will have orientation to this new logic model in outcomes planning this next fiscal year, or is there a time on that?
MARGARET CAMPBELL: Again, I'm probably not the best person to answer that question. We have certainly advocated for it. When I say we, I mean, our Division. The Planning Budget and Evaluation Division of NIDRR have advocated very, very strongly for it.
We have - and the research plans division that still runs peer review has heard us. It's not that they haven't heard us and that they are trying to make some materials available. It has not been - at this point, it's not been as focused and, you know, systematic in orientation or training.
I think it's fair to say, though, that everybody is very aware of that gap and the risks of that gap, that grantees are trying to incorporate something into their proposals, into their applications, that is not yet either directly in the selection criteria so that reviewers don't know how to assign points to it and/or that the reviewers are not themselves up to speed in it.
So I think that the awareness is there and there's certainly the advocacy for it. It's finding a systematic way of doing it. We have had discussions, and these are just discussions, about the idea of contracting with, you know, an outside contractor to develop a Web-based course that could be sent to reviewers and that could be used by reviewers.
And I think that will continue to move forward. I can't give you a timeframe for it and I can't even give you a go-to person for it yet, but we've - certainly the awareness is there. We know the need. I mean, I feel it acutely.
And we just, to be honest, haven't gotten there yet.
BRENDA CAVANAUGH: OK. Thank you.
JACKIE SMITH: Dr. Campbell?
MARGARET CAMPBELL: Yes.
JACKIE SMITH: This is Jackie Smith from the Wireless RERC in Atlanta.
MARGARET CAMPBELL: Hi, Jackie.
JACKIE SMITH: Hi. I have a question. Normally we plan about two to three months out on doing our APPR. Have you identified the technology centers that will be receiving the supplemental information report?
MARGARET CAMPBELL: All of them.
JACKIE SMITH: All of them?
MARGARET CAMPBELL: All of them.
JACKIE SMITH: And when will they be due? Go ahead.
MARGARET CAMPBELL: Yes. You will get, certainly, adequate time.
JACKIE SMITH: OK.
MARGARET CAMPBELL: We're trying to get this out to you. I have a big echo. I don't know if you can hear me.
JACKIE SMITH: No, you sound fine.
MARGARET CAMPBELL: OK. You will be receiving this within the next four weeks.
And it could really be as soon as two to three weeks. That's how close we are. And you will definitely be given adequate time to fill it out. It is not, however - it is - we have a lot of - we have very few open-ended narrative fields, very few.
JACKIE SMITH: OK.
MARGARET CAMPBELL: It's a lot of - as much as we can possibly give you drop-down menus, et cetera, to choose from.
We have done that. And I can demystify it a little bit. The focus is on the three purposes for doing this.
One, it focuses on just those performance measures that we have to get information on this year. OK? That's one purpose.
Two, we are experimenting, as I've said, with how to collect information on accomplishment nuggets. We are asking for three different kinds of accomplishment nuggets.
One is your best scientific and/or technical publication. That's one accomplishment nugget.
JACKIE SMITH: Oh, OK.
MARGARET CAMPBELL: And we're going to allow up to two. You can nominate up to two. And for each one, we ask you to give us roughly 10 different pieces of information. A couple of those pieces of information require paragraphs. The rest are close-ended.
So that's the first accomplishment nugget. The second accomplishment nugget are short-term outcomes, not surprising. Again, we ask for no more than two. We ask for you to nominate no more than two. And for each short-term outcome nugget, we again ask for about 10 pieces of information.
The third one is, if applicable, intermediate outcome nuggets. No more than two again.
And the other three questions on the supplemental information form…The first one asks you to give us the discipline background orientation of your investigators, investigators and co-investigators.
And that's directly for a performance measure. We have to assess the multi-disciplinary balance. OK?
JACKIE SMITH: OK.
MARGARET CAMPBELL: Now the other two ask you about your studies. One is we get a profile of all your studies with close-ended information, just so we know what all the studies are, and then we ask if you are conducting any site-specific effectiveness studies using experimental or quasi-experimental designs.
And we ask you for, again, about seven pieces of information related to that. Or if you are conducting any multi-site controlled studies, we ask you for about seven pieces of information on that.
And then there's a final question on any major difficulties you've encountered that have seriously affected your ability to complete your planned activities, produce your outputs or generate outcomes.
What are those difficulties? Why did they rise, et cetera? And that's the end. So it's not as extensive and doesn't require as much written response.
JACKIE SMITH: OK. Very good. Thank you.
MARGARET CAMPBELL: Yes.
JOANN STARKS: Well, Margaret, you've been doing a great job. I really appreciate the time and thought you've put in. Do you have time to go with us a little bit longer?
MARGARET CAMPBELL: Sure. Sure.
JOANN STARKS: OK….Anyone else have a question for Margaret?
DEBBIE GILDEN: This is Debbie Gilden in San Francisco.
MARGARET CAMPBELL: Hi, Debbie.
DEBBIE GILDEN: Hi, Margaret. Any chance of getting a copy of the transcript of this hour?
JOANN STARKS: Yes, there is. I am taping it, and we will get it transcribed.
DEBBIE GILDEN: And then how do we get it?
JOANN STARKS: That's a good point. I think I need to have everyone email me that was on this call because even though I tried to write everyone's name down, I'm sure I didn't get everyone. So if you could all email me then I will make sure that whoever does email me will get a copy of the transcript.
And we'll also make it available on our Web site, so you could go back and download it if you wanted to.
DEBBIE GILDEN: Excellent. What is your email address?
JOANN STARKS: It's jstarks, J-S-T-A-R-K-S, @sedl.org.
DEBBIE GILDEN: Thank you.
KRISTA KUTASH: Joann, how soon will this be available?
JOANN STARKS: That I'm not sure. I have to get with the telecon people to see how long it takes them to get the transcription.
KRISTA KUTASH: OK.
JOANN STARKS: But we'll do it posthaste.
KRISTA KUTASH: All right. Thanks.
JOANN STARKS: Yes. OK, another question for Margaret?
ANDREW HOUTENVILLE: When are you getting - this is Andrew from Cornell. So, could you describe to us what you're going to be doing at your new position?
MARGARET CAMPBELL: Andrew, I'm going to be scrambling, aren't I?
ANDREW HOUTENVILLE: I think so.
MARGARET CAMPBELL: Oh, I'm going to be scrambling to switch gears, first and foremost. Well, I'm going to be, I think I told you this, Scientific Director of the Center on Health and Disability Research at the National Rehabilitation Hospital.
And Thilo Kroll is a part of that team, I think many of you know him, as is Gwen Jones and Sue Palsbo used to be and Pei-Shu Ho and a couple of other folks…
I'm going to be trying to, first of all, orient myself and learn more extensively about what everybody's doing, what research activities they're involved with, what their interests are, et cetera, and then trying to build a research program, of course, using very much the logic model.
I will try to develop two or three programmatic themes that we can develop into really sort of research programs, lines of research. I'm, of course, very interested in one of those being within the realm of aging with disability.
We're also engaged in some conversations with Georgetown University in trying to develop or exploring the ideas of developing a program where we could have a collaborative arrangement with them, they are folks from what's called the Health Systems Administration Department, and they're very interested in things like health literacy and health competency.
And so, you know, I'm not going there with ready-made proposals and a quiver that I can just kind of pull out of my quiver and fire off, and fire off, because I haven't been doing that kind of work.
So I'm really going to have to go through a period of switching gears. And this has been very much discussed in our negotiations and my deliberations about this job that I really needed time to be able to do that and to mesh my research interests with needs in the field and with the research interests of the folks that are there to try to build something, you know, to fashion something coherent out of all of that.
So I'm excited about that. So that - does that answer your question or give you a little flavor?
ANDREW HOUTENVILLE: Sounds good.
MARGARET CAMPBELL: I think I wanted to say, if there are no other questions, I could say one more thing about this upcoming pilot phase of the portfolio assessment expert review process, to tell you how this is going to work in terms of these accomplishment nuggets.
And I'm pretty excited about this part. I'm excited about a lot of it. I think that NIDRR is not necessarily leading the pack but we're up with the big boys in terms of - big boys and girls - in terms of how other, much larger, federal agencies are trying to implement performance measurement and assessments.
What we're going to do after we collect all of these accomplishment nuggets is that we will sample them. You know, we won't be able to unload on these expert reviewers every single accomplishment nugget because if everybody, if every awardee submits two of each, that would be six, six times 69 is whatever it is, it's too much to distill and have them evaluate.
So what we're going to do is something that - we're going to sample and we're going to use - we were probably trained as social scientists, social and behavioral sciences, and purposeful sampling.
The term that NSF is using is judgmental sampling, and I'm developing kind of a template or guidelines for how we will implement that judgmental sample because each portfolio will contain different types of awards, will contain Field-Initiated, DRRPs, RERCs and potentially RRTCs.
So we have to have a way of sampling that covers the different types of awards as well as have it somehow proportional to not just the number of awards but the number of dollars invested in each program category.
And then that's where these clusters come in. We also want to make sure that the way we select these accomplishment nuggets, sample them, reflects the diversity within the portfolio. So it will be - so what each - what the cluster of expert reviewers will get will be the judgmentally selected sample of accomplishment nuggets.
That's another detail that I'm, you know, very interested in your thoughts about or your ideas and/or your reactions to. But that's how we're planning to proceed.
CYNTHIA SALZMAN: I have a question.
MARGARET CAMPBELL: OK.
CYNTHIA SALZMAN: This is Cynthia Salzman from the Model Spinal Cord Injury System in Seattle. And I don't know whether this is too specific but, you know, we're coming up too - I guess we're getting an extension, our five-year grant is almost up.
And I'm wondering - and we're trying to think in terms of the next cycle and doing logic models to plan our activities. And I'm wondering what kind of training or, if you have - what we can expect in terms of the request for proposals and how we should be thinking about our research projects and our activities.
MARGARET CAMPBELL: Right. Good question. And again, I may not be the best person to answer that question. It may be more Dick Melia or Ruth Brannon. But, so what I'm going to offer here is just kind of what I've heard between the lines.
I don't think, at least not in the near term, that NIDRR is going to require logic models per se, per se. I don't think that means you couldn't include them. But it's - a logic model is a tool, right?
CYNTHIA SALZMAN: Yes.
MARGARET CAMPBELL: It's a tool that you can use for identifying these well-constructed outcome goals that really come, you know, hit on what you think you're going to be able to accomplish and the steps that you need to take to get to them, et cetera, that are reasonably ambitious and yet measurable, et cetera.
So, I don't foresee NIDRR requiring logic models, per se, but I think, yes - I think it would certainly serve you well to use that approach as a tool to planning your proposal. How much you include the logic model, per se, in your proposal is a real judgment call because it's back to the other question we had, how much can you assume that the reviewers have been, have adequate training in that.
So I think, kind of separating out, using it as a tool to plan and construct your work, I think it's just very, very helpful in that. First is actually incorporating - how much of it you incorporate in your narrative, in your write-up, and how much you show your logic model, et cetera.
I think that's a judgment call. And I think somebody else reflected, I guess it was the very first question, Krista, on that this is becoming, obviously, more and more persuasive - not persuasive, pervasive.
So I think that as time goes along, you know, even if NIDRR hasn't given the training, and hopefully NIDRR will be providing some training, and our logic model and logic modeling, these reviewers are going to be encountering in their own work.
So I think as time goes forward, you know, it becomes less of a risk to talk in those terms within your narrative. But on the other hand, the big connecting the dots that has to take place on the NIDRR side is the selection criteria have to be conducive to that so you get full credit.
CYNTHIA SALZMAN: Yes.
MARGARET CAMPBELL:
I would not be surprised to hear stories from grantees in the past, last two to three years, that have adopted it and felt that the reviewers just didn't get it. In fact, I've heard one such story that's come to me. I wouldn't be surprised if there are others out there.
But I think going forward it's less risky.
JOANN STARKS: Cynthia, this is Joann in NCDDR. I wanted to let you know, if you weren't aware, we are hosting a workshop on outcomes planning and how to develop a logic model on June 1st.
And if you didn't receive an invitation to that, I can certainly make sure that gets to you.
CYNTHIA SALZMAN: OK.
JOANN STARKS: And any other grantees that would be interested, you know, we can certainly invite you. We do have, I think, a limit of 50, and I think we're very close to that. But we've had a few people drop out that had initially signed up, so we may have a few spaces left if anyone else is interested in participating in that June 1 workshop, which will be in Washington at the National Press Club.
UNIDENTIFIED PARTICIPANT: Joann, will there be any plans to have additional workshop perhaps later on in the year?
JOANN STARKS: I don't think we have any plans any further this year because, as you know, the NCDDR is coming up also. September is our ending date. We'll be writing a new proposal, as may others, and we probably will not have any other workshops this year.
But we will be providing technical assistance. Any grantees that want technical assistance in outcomes planning and dissemination activities, please contact us.
UNIDENTIFIED PARTICIPANT: OK. Thank you.
JOANN STARKS: We do have the RUSH project, and that also is very active with outcomes planning. So we can still provide some resources.
UNIDENTIFIED PARTICIPANT: All right. Thank you.
JOANN STARKS: Yes.
VATHSALA STONE: Joann, this is Vathsala and I know that time is running out, but is there a - is there time for a quick question for Margaret?
JOANN STARKS: I think so since we did start a little bit late. Margaret, if you can answer another one.
MARGARET CAMPBELL: Sure.
JOANN STARKS: And anyone who needs to leave, of course, please go. And those who want to stay...
VATHSALA STONE: I don't know if this is very specific, but too specific, but I'm thinking of the redesigning of the APPR, Margaret.
MARGARET CAMPBELL: Yes.
VATHSALA STONE: If we want to give you any input about our concerns whom should we be sending this to in future, in the future, knowing that Margaret Campbell is not there anymore?
MARGARET CAMPBELL: Well, good question. It is going to be - once we finalize the data elements, it will be up on the Web for you to make comments. And that will direct you to who to send those comments.
My hunch is it will be someone from RTI, who is the contractor, as well as someone internal to NIDRR. But we very much do want you to - grantees to look at these data elements and give feedback on both on the element itself, is it necessary, is it not necessary, but as well as how - if you have better ideas on how to collect the information related to the element.
So that is definitely planned as part of the process.
VATHSALA STONE: OK. Thank you, Margaret.
MARGARET CAMPBELL: Yes.
JOANN STARKS: Anyone else have one more quick question? No? Well, I thank everyone very much for coming. Margaret, thank you so much for giving us your time.
MARGARET CAMPBELL: Well, thank you for this opportunity. I hope we don't become, all of us become strangers with me taking a different position. So I hope truly to join your community.
JOANN STARKS: And thank you all for coming. Please do send me an email if you were participating today, and I'll make sure you get the transcript, or how to get hold of the transcript.
And also, if you have any other questions that you think of afterwards, please send them to me and we'll see if Margaret might have time to respond to any of those in writing later.
UNIDENTIFIED PARTICIPANT: Thank you.
JOANN STARKS: No promises, however.
MARGARET CAMPBELL: OK.
DOUG WATSON: OK, thank you.
JOANN STARKS: Thank you all very much for coming.
UNIDENTIFIED PARTICIPANT: Thank you.
MARGARET CAMPBELL: Thank you all. Bye-bye. Thank you, Joann.
UNIDENTIFIED PARTICIPANT: OK. Thanks.
UNIDENTIFIED PARTICIPANT: Good luck, Margaret.
UNIDENTIFIED PARTICIPANT: Good luck, Margaret.
MARGARET CAMPBELL: Bye-bye.
KRISTA KUTASH: Bye, Margaret.
MARGARET CAMPBELL: Bye-bye.
VATHSALA STONE: Bye.
UNIDENTIFIED PARTICIPANT: Enjoy your new position.
MARGARET CAMPBELL: Thanks.
[ END ]
Contact Dr. Campbell: Margaret.Campbell@ed.gov