news videos images websites wiki

Well good afternoon everyone thanks for joining us this is symposium six preliminary findings of the mental health Block Grant ten percent set aside national evaluation of coordinated specialty care and I'll be your facilitator but my main responsibility is to introduce our chair Abram Rosenblatt dr. Rosenblatt is from West. Stat and I am going to turn it over to him now thank you thanks Wow applause before we start talking that's a good sign well so it's after lunch thank. You all for coming on a whatever. Kind of date is out there I thought I'd harken back just to get started for a second on the valley of death that we heard about this morning from in. The first session this presentation comes to you directly from the valley of death that's where we live. Between implementation and research and it's okay because I grew up in Arizona and went to grad school in Arizona and prithi also went to grad school. In Arizona so as long as it's a dry heat we're fine in the valley of death it's a little bit better than it is outside right now so so it's a great privilege to. Be able to be here and talk with you all today I'll introduce some of our panelists as we get to that part of the slides so you know where they fit in and we've got three talks. And a discussant for you for this afternoon first off I want to thank our funders we were funded by the city of Mental Health Services Administration. The National Institute of Mental Health thank you National. Institute of Mental Health and the office of the assistant secretary for planning and evaluation and they funded this work where the company has been contracted to do this work West stat. It's a research company up in Rockville come on by and visit if you're in the neighborhood we'd love to hear from you all not all of you once a time No. And also the NRI and gnash but who are our collaborators so this is our evaluation team I'm the principal investigator. Of the project we have some senior investigators Lisa Dixon and Howard Goldman Tara is standing in for Lisa today because Lisa had some family issues that kept her from attending and she's gonna be our discussant and she's delighted to. Have you our project manager is Preeti George my fellow Arizonian and she's run the project and we also. Have a research methods group that includes the folks listed here many folks have worked in this area of early psychosis intervention Gary bond is leading our fidelity team and Valerie Noel is going to represent our fidelity measures so we have three presentations there we go nothing like hearing yourself live that's really good I'm gonna deepen my voice now. A little bit I don't want to see the video I think we'll wait till that gets wrapped up over there good okay so here's the objectives of the overall evaluation. Now we're gonna identify and describe coordinating special care program services being offered nationally this is a. National evaluation covers a lot of programs we assess the. Program fidelity to the NI MH coordinated specialty care model we're going to look at local environmental contextual factors related to how the programs are implemented and we also explore the outcomes of program participants so this is a national evaluation it's largely descriptive by design although we're. Going to be looking at the relationship through paint. Fidelity and outcomes. And some of the process indicators and outcomes we have 36 study sites for a 3-year evaluation we. Just completed the first. Year of data collection so you guys are going to be the first people in the world. Aside from us and our collaborators and the federal government to see some of this data so consider yourselves lucky or something like that and we selected sites based across the different regions we wanted to vary them in terms of program model types we vary they varied in terms of the. Services implementation status you'll see some of that variation on the ta that receives the start of the program in urban rural so there's a lot of natural variation across these 36 sites this was our site selection process you can see we went through a structured process of identifying them there were 216 sites and we narrow those down. From a possible list of 56 to get. To our 36 sites they had to receive mental help lock grant funding from their state so I believe most of you know but I'll just. Say quickly Sam supervised mental health block grants to all the states and the Block Grants are used by states for a variety of mental health services legislation. Passed at some I forget when but they passed legislation you probably know Susan to set aside some of those dollars to be used for early psychosis programs beginning with five percent and then it became ten percent hence our catchy catchphrase the mental health Block Grant ten percent. Set aside evaluation that's really fun to. Say I know and we also wanted to be sure that they're implementing the elements of coordinated specialty care in the model and that the site with certain clients and was willing to work with our study team so we're very appreciative that these sites were willing to work for us they received very little. And virtually nothing really to participate except the knowledge that they're contributing to the greater good and so that's really terrific so. We have four study components. You're going to hear from a couple of them today we have the data collection activity to site survey that's a two hundred and forty eight sites across the country so we'll be giving you some results from the site survey and for the. Sites that were participating in the National evaluation we have three data collection elements and outcomes. Assessment of fidelity assessment and a process assessment we will not have the outcome assessment data. For you today because we just finished the first round of data collection so you don't have outcomes usually until some time passes to get more outcome data so we'll stay tuned for that one you're gonna hear something about the fidelity assessment today and. Really mostly how was designed and what the fidelity assessment looks. Like we're quite proud of to our work in that area and then you'll see some results from our process assessment which was a qualitative set of interviews around implementation of coordinated specialty care services in the sites this is a timeline basically where we're at we just finished our time one data collection it's a. Mixed-methods design as we said we have an overview of the programs nationally we have. An outcome analysis fidelity assessment process assessment this is what the site survey does you're gonna see some data from this and it tries to look at the better understanding of the CSC programs nationally and get some information from them around participation treatment services duration of care what how communities are using there's a client outcome. Assessment these are the measures we're using on the outcome assess again you won't see data on these but you might be interested in them we we picked these to be as brief and also as good quality as possible because sites are. You know giving this us this data and. Also to be items that measures that were typically used in sites and in a coordinated specialty care programs so we have the modified Colorado symptom index the brief psychiatric rating scale if they had it the my rec gaffe symptom scale if they. Had it and then some quality of life functioning measures the global functioning social role. Functioning scales to layman. Quality of life scale the global scale I love this scale if you guessed Enoch was delighted terrible applies to almost all circumstances so at the end of this you can say whether the presentation was delighted or terrible or you were too loud or terrible and then you can think. About how you're feeling now how you guys now delight it terrible somewhere here. We go and then if available to my erect gap which is a social functioning occupational functioning scale from that the fidelity assessment which you'll also be seeing some more. Further description of it's conducted remotely by a trained Assessor and I don't think I'll go into that here because you're gonna cover that for us Valerie the process assessment are semi structured in person interviews with the program administrators at the. Sites the team leaders clients in the program and the state behavioral Authority representative we did that by phone Tara participated in one of those and. You can tell you how much fun that was in fact I did your interview so and so that's an overview of that now we have the most difficult task I'm gonna have in this presentation which is to get. To the next slide and if I do that I am delighted okay so. This is a look at the Samsung ended first program programs for first episode psychosis across the u.s. those that are using mental health lock grant funding and our goal here was again to get I already described this but to give us a better understanding of these programs nationally and the services they offer we did an online. Instrument 28 questions there were two conditions virology they had to currently be serving clients and they have to receive mental health ten percent set aside funding funds so pretty basic we sent out 250 invites we had. The survey available for a period from February to June this year and by the end two hundred fifteen sites completed the survey and the six were ineligible so our final response rate was eighty eight percent we are really happy to get an eighty eight percent response rate on a site survey going out to sites where they're. Completing this and again they're not. Receiving any Picard dispensation for doing. This other than to participate so we're really quite pleased with that and it prices a lot about how dedicated these sites are to these programs these are the locations of the responding programs so you can find your favorite. Location and see where the where it is because here they were across the country fairly widely disputed dispersed here's who responded to the survey we had mostly leaders and administrators team leads clinical directors and supervisors or other. So we had a range of different respondents but by and large they were program leaders or team leads so this is an interesting slide it's also. Colorful which is a plus and you can see that this is the year that the program's began serving people with first episode psychosis against the time they began serving people with mental illness you can see that most of these programs I've got a nice little arrow here yeah most of these programs actually began and opening and serving. People with mental illness way back in nineteen eighty or eighty two so they've been around a long time but what's significant here is the growth and first episode psychosis programs and that growth if you like reading these does tend to coincide with the funding of the. Guidance provided by States by NIH and I am Asian Samsa the set aside for first episode psychosis and the coverage of the first episode psychosis intervention services and then the set aside again that was done. Later in 2015 so you can see that a huge amount of growth right around here 2014 25th and when these sites began so this is obviously one case where federal intervention. And federal training and all those kinds of things led to a real explosion in services for individuals with first episode psychosis experiencing first episode psychosis this one is which of the following models did. You receive technical assistance or training from and you can receive training on more than one models who those of you and we have a research audience your priority doing the math in your head and going why is it more than 100% and that's generally the case for many of these items they can receive multiple. Training most receive training from on track which is that program out of New York navigate Orissa out of Portland and then we had. Models scattered around with other first peer or some. That said one or two that said they didn't have any training so on track. Navigate and he said were the most popular the number of active clients varied quite a bit some programs were very. Small from 0 to 10 some were quite large so a majority 51 clients so there's a tremendous amount of variability in the size of these programs nationally the programs also vary as to how full they are how they are running to capacity some are obviously 6% over 100 percent that doesn't sound like. A very fun program to have to work in by very very busy but many many are running in the 50 to 99 percent and some are still have a lot of room. So one of the things that we do ask in our process interview that you may hear somewhere about later is how sites recruit folks into their into their services the average duration of care varies some are more than three years and some are less but. Generally they're in the one to two to two to three year range and our values didn't come out for some strange. Reason when we made these wide-screen but it's about what do you guys think are all research was 45 55 I can't remember something like that as far as referral. Sources go they come from a lot of different places psychiatric inpatient facilities outpatient clinics you can go ahead and take a look down the list I don't need to read them all to you but you can see a wide range from Corrections to inpatient facilities so. A lot of different referral referral sources most were funded from State Block Grant early intervention set aside funds that's the blue I do know that and the other ones were funded by other means. And again we have this. Money no way we had that problem that's weird it was fine when we looked at. It before but whatever and did the receive support from other sources other than help block grant funds is about a third that did treatment services and supports offered again very these are typical CSC type services they include family education and support all the way down to neuropsych assessment down at the bottom and occupational therapy but by. And large if you take. A look at you know 71% or above you can see 7s use services primary care coordination peer support a wide range of different services are being provided across the country in these programs ones that are consistent with. Courtney's specialty care especially do they have a team lead that means yes the blue so yes. Most programs do have a team lead do they have enrollment criteria it's a good thing I looked at these beforehand the blue is yes most do and if so. As far as enrollment criteria what is the time period that they've been serving these youth and you can see most have criteria for about. Two years for duration of first onset so the time period generally runs two years to three years but mostly two to one-and-a-half years and some about one year I'm do they have enrollment criteria about where the person was prescribed antipsychotic medication I. Can tell you that the purple is yes do what kind of training was provided to staff there was a lot of different kinds of training rationale for early intervention client family engagement share decision-making components was coordinated specialty care so a wide range of different kinds of training. Was provided to the staff as far as how they're staffed a lot of variation here three or fewer FTE oh it's a common response of 45 percent so that's. A very small number obviously for the program and then some programs had seven. Or more FTEs so there's quite a bit of variation though most actually we're fairly small in terms of the number of FTE full-time equivalent positions dedicated to the program as far as family engagement you can see a number of strategies including education family members invited to participate psycho-educational services but the main takeaway here is all of these numbers. Are well above 80 percent so a lot of different strategies a lot of things are being used to engage families and then these are some of the outcome measures I think most of these who look familiar to you all but again a lot of outcome measures being used. A lot around psychiatric hospitalization employment school participation justice involvement homelessness those kinds of things relatively fewer physical health as you might expect but only three percent are not collecting outcome measures there's only like one or two respondents I'm is there. A designated person at the site looks at client outcome data I believe the blue is yes so yes if yes are they tracked systematically at the caseload level most are tracking them at the caseload letter and comparing over time not as. Many compared to benchmarks benchmarks are hard to come by so a lot of outcome data collection going on in these sites and a lot of monitoring of data of outcomes I do they use a fidelity tool the blue is yes but the purple is no and so a lot of sites are not using an actual coordinator. Specialty care specific fidelity tool like the one that we're going to cover and the rest of our presentation most accept a wide range of insurance you can see 93% accept Medicaid 62 Medicare 76 private so many of these programs really rely on Medicaid which we also actually did encounter in our process interviews so some takeaways a large increase. In programs providing first-episode services coinciding with set-aside funding and CMS coverage most programs are implementing navigate on track Orissa models and they do receive ta directly from those models the active clients as you saw do vary by capacity many. Programs do rely entirely on Mount Hope Block Grant funding so that's a very important funding source obviously for these programs if they're unable to find other funding they'll have a challenging time to continuing programs do provide a wide range of services they use a whole host of different referral sources about half the program's use the fidelity tool so. Not not a huge number about half and almost all of them accept Medicaid and about three-quarters accept private insurance so they do use other funding mechanisms but they really rely on them and help block grant dollars so that's a look across 200 some odd sites across the country you can see we have that's. Not all of our data we have a lot of descriptive. Information about these programs that I think we haven't had before so we're going to be very interested as we dig a bit more into all of this and also compared to the sites that we actually. Then collected data from so pre fee now I can that's right okay great okay well good afternoon everybody thank you. So much for joining us we're so glad that you showed interest in our session and we're delighted to share information about our study as we learn. About it so you've just heard from Abram sort of a broad picture of CSE programs that were sort of reviewed and surveyed nationally what I'm going to do in this presentation is focus more and definitely on the 36 sites that are included in our mental health Block Grant 10% study. So in general we are interested in understanding a little bit about the local context in which these coordinated specialty care programs are delivering services and in particular for our 36 sites we want to know about whether that local context in which they're operating has sort of an influence on the way that treatment services are being delivered and how programs. Staff perceive that local context to affect their work so in this particular presentation I'm. Going to be talking about the process assessment which is what Abram just mentioned in his overview and that is the qualitative component of this evaluation that sort of looks at the local context of the different programs. And asks them to sort of discuss the implementation issues that they might be experiencing and delivering CSC and. In particular for this presentation what we'd like to do is talk about the local accommodations to the CSE model that sites reported that they are using in. Response to that local context so first what I'll do is I'll just provide you some brief background information about the 36 sites that are in our program and. Then just deliver to sort of a little summary of the. Process assessment data collection and then we'll actually talk about the accommodations that sites reported that they're using to implement CSE okay so as abram mentioned there 36 sites all of them are. Using mental health Block Grant set-aside funds for some aspect of their coordinated specialty care programs and the sites that are in our study they reported using seven different CSC program models all of them implement coordinated specialty care but they have different types of see se models that are in use for example first we have on. Track ISA navigates some of them use hybrids of those models and so this is a map of the 36 sites that are in our study you'll see we have some geographic spread here we have sites that are represented in nine of the ten dhhs regions of the United States and. Just a few background slides about the sort of local context of these. Programs this is the catchment area size so for the programs that indicated that they had a catchment area using so using US census data we. Looked to see sort of what the catchment area size was of those local programs and we have. Some programs that are offering services to those areas for less than one hundred seventy five thousand I think the lowest number of the catchment area sizes for a community serving fifty. Thousand and then we have sites that are serving catchment areas of three million or more so we have a wide range similarly we have different levels of catchment areas served by these sites the majority almost half of the programs that are in our study indicated that they serve a single county others served multi County areas and. We have some that are serving combinations of cities and and states and local jurisdictions as well the number of active clients varies - sort of we have some programs that are offering services to a very small number of clients less than ten others that are serving 55 or more so we have a broad range most. Of them serving the numbers between 10 and 40 clients so just more in depth about the process assessment so like I mentioned we're interested in understanding not just the fidelity and the outcome data clinical outcome data but we want to learn more. About the implementation and the local context in which these programs are operating so our process assessment was designed to examine that and the way that we use the process assessment is to do site visits so we visited all 36 sites and did interviews with various staff at each of. The sites so we spoke with program directors administrators as well as the clinical team we spoke with some clients that were recipients of the services or recent graduates of the program and we also spoke with state mental health directors for the states in which our programs are located so in. General overall after this first year of data collection we spoke with 210 clinical CSE staff we interviewed 35 administrators 22 state behavioral health authority representatives and 58 clients so it was kind of a fun experience to visit all these different sites we. Got to travel as far as Puerto Rico Oregon all over you got to go there it was a very fascinating experience and Tara was one of our sites we visited too and so it was a great experience to actually see the context in which these programs are being implemented and sort of see on the ground what clients and what. Staff are seeing as they provide services so we audio recorded all the interviews and then had them professionally transcribed and we used the data from those transcriptions with a qualitative software programming analytical tool and so in our particular interview what we are focusing on here are sort of the local. Context contextual factors that sites sort. Of have to face and how do they sort of provide accommodations to the CSC model to address those local factors so we asked sites and in particular for this presentation and we'll talk about interviews. We did with the CSC clinical staff as well as the administrators so we asked them about what are some. Of the ways that you've modified your services to address local contextual factors including. Things like the language of your clients the demographic characteristics of your clients socio demographic socio demographic factors of the community and others sort of factors and so in. Order to do our analysis what we first did was sort of looked at the responses to those questions and identified any other places within our interviews where they talked about imitations to. The model and then we extracted. The key responses that sort of described the accommodation that they were discussing and then we did some coding of those data and. We looked at the type of accommodation and the general driving factor for that accommodation as reported by the sites so these are all self-report data so using the coding what we essentially want to. Know is we want to know what type of accommodations did sites reportedly make to their CSE models to accommodate local factors and why were those accommodations being made so let's just talk about the types of accommodations there were basically five different. Categories that we found most prominently aspects of treatment provision was the era in which most of the accommodations to the. Model were made locations or the modality of work was also represented the cultural awareness and language of the populations they're serving staffing issues as. Well as some other accommodation factors which we'll talk about in just a moment so let's talk about that first one aspects of treatment provision so that was by far the one that was most prominently named by clinical staff and it basically fell. Into four different areas the first one being the most prominent was to add an approach or therapy that wasn't necessarily part of the model or increase some use or aspect of the model so for example. Add dialectical behavior therapy to them to your clinical treatment plan or add the use of peers in providing services or increase case management or psychoeducation depending on the needs of the client other ways that treatment provision was addressed was to several sites said that they use individual psychoeducation. With families rather than providing psychoeducation through multifamily groups and that may not necessarily be a specific accommodation to the CSE model but sites perceive that to be an accommodation that they were making another was. To change the language or the order of the modules to try to help clients understand the the content and also to take away an approach or a therapy for example don't always use cognitive-behavioral ii oriented therapies the next. Type of accommodation was local at the location of treatment and for the majority of the individuals who responded they indicated that visiting clients in their homes and in the community was something that they practiced and it wasn't just their therapist but it was. All members of their team to try to help provide transportation meet community members in in the community and also to use technology to work with clients so. For example one of the respondents indicated they're more creative in terms of using telepsychiatry to address clients that may not be able to come to their locations we also heard people talk about cultural awareness and language. Supports so much of the discussion was here related to attention to language providing translation bilingual therapists other. Types of things to sort of accommodate people with other language and also to address cultural factors such as religion or gender that could be sort of something relevant to their treatment staffing or staff patterns was another accommodation that's that were named so for example some of the examples people talked about were to. Have staff fill multiple roles so given the context of. Their their team they had some folks doing both case management and being the recovery coach or doing supported education duties as well as case management other areas that people mentioned that they did in terms of accommodations to the model were related to policy training or outreach and some of. These are sort of related to having flexibility in terms of when individuals would receive services having additional trainings if they needed additional trainings to there for their staff and also to be attending to outreach and making sure that they provide outreach to populations that were historically underrepresented so now that we. Talked about the types of accommodations here are information about the driving factors that individuals serve responded about why they do these accommodations and they. Fell into basically three categories we have characteristics of the population served factors related to the. Agency or team and then characteristics of the community so in terms of characteristics of. The population served individuals reported that they made these accommodations in order to address cultural and language issues of their clients to accommodate personal preferences so for example if they were family members that didn't like sharing in groups that they would do individualized psychoeducation to try to address the clinical presentation of the individuals they were seeing so to. Add trauma focused treatments or social. Skills training for example to address those issues the economic backgrounds of their clients so if clients couldn't afford transportation if they had challenges in terms of missing work in order to be with the therapists they would be very flexible in terms. Of accommodating those needs and also trying to incorporate or at least acknowledge beliefs about religious beliefs about causes symptoms and treatment and sort of incorporating that in their treatment the factors related to the agency or program a lot of these sort of examples that people discussed. As sort of driving forces for these accommodations was sort of to address based on clinical clinicians experience what did they think would be most helpful in Zoar facilitating engagement with their clients so some of the things they talked about was well we have peers who work. On our team and that shared experience is very helpful in terms of helping with engagement and participation in the program so those were some of the factors they talked about there and then also characteristics of the community so there were some sites that. Indicated that there was limited bus service little or no low-cost housing. And so they tried to make accommodations. In the types of services they provided to address those needs similarly if there was a very large catchment area families might be dispersed across large areas and so one of the input challenges for example that rural areas some sites indicated that they faced having a very large. Population a graphic area but the population based on which to sort of have clients was smaller so there was a lot of geographical navigation that they had to do to get to clients that they needed to see so what are some key takeaways from all of these these qualitative data some. Of the keys or accommodations to. The model kind of just reflect a good clinical practice things. Like translation or bilingual therapists or interpretation those are all things that I think all good clinicians would try to do to try to help facilitate clinical practice some accommodations seem to reflect an extra. Effort of staff to be sensitive to the economic and other barriers that clients faced and other accommodations seem to provide sort of an accumulated experience of staff and what they thought. Would be helpful engaging keeping clients engaged so. Those are some of the key takeaways we learned from our first wave of qualitative data we're going to go out into. The field next year and go and do site visits again and so we hope to have more information to contextualize some of the quantitative work that we're going to be collecting as well so our next presentation is on fidelity with Valerie Noel I want to just really quickly though before I forget acknowledge that while we're up here speaking our. Colleagues at Westat and elsewhere a lot of work on this presentation particularly Jennifer O'Brien did some of the work on the site survey. That you saw and Tamara Daly did a lot of the coding and work with us on the qualitative interviews and you can imagine there's a lot more data in those qualitative interviews and also I want to thank Lisa Dixon and Howard Goldman who gave us a lot of feedback and help in framing the presentations just wanted to be. Sure we acknowledge some of our colleagues and our next one's on fidelity so Valerie thank you April I'm sorry you are for okay great five okay all right thank you so much so with the 36 sites spanning across the u.s. we were faced of the challenge as to. How we would conduct a fidelity review at each site with the limited people power that we have which I think was me so so the method we chose to overcome this challenge. Was to use a remote fidelity review method which I'll describe along with I'll provide you some preliminary inter interrater reliability. Statistics and discuss some of the benefits and limitations that we experienced and identify throughout the process so firstly our first step was to identify the fidelity scale so around the world there have been currently there are many endeavors - aimed at providing services for those experiencing a first episode. Psychosis and so researchers have developed fidelity scales in order to assess the level. Of delivery of these services and so the development of scales can be informed by research evidence it can. Be informed by the practices of a program model such as what we have here is the the opus which is from Denmark and epic which is a fidelity scale made in Australia they can also be informed also in addition by expert consensus which we. Have the Eden which is from the UK and ISA which is from from here so as Abram and prithi had described the programs that we were evaluating were diverse. They included they used different models such as the navigate model on track ISA some programs used elements from a few of those some programs didn't use any of their of any model and kind of made their own model so we weren't able to use a fidelity scale that was based on a specific model so what we ended up. Choosing in the end was the first episode psychosis fidelity scale and so this is a 31 item evidence-based scale it's rated on a five-point scale and it is it was developed through knowledge synthesis of so it's researched from empirical studies meta-analysis systematic reviews and also incorporating expert opinion to the. Developing it and this scale works because it wasn't developed based on a model and so it will work. Across all models another reason why we've chosen. It it has shown a good inter-rater reliability and there is evidence also for face validity and so in addition these 31 items also have the greatest content overlap over the other I believe five fidelity scales that I have shown you so this fidelity scale almost encompasses all. Of the content areas that. The other five fidelity scales currently examine so the programs that we were evaluating follow follow the principles and components of the. Coordinating specialty care model and so the fidelity scale that we chose needed to assess these principles which the FE PSFS does these principles of the coordinated specialty care. Model being that it's a team-based approach levels of certain characteristics and team leadership there has to be case management provides supported employment and supported education services psychotherapy also family psychoeducation and pharmacotherapy and primary care so now to the evaluation so in this first year we made several modifications the first. Modification was modifying the procedure because this scale the FE PSFS was used for fidelity reviews for at on-site visits we needed to come up with a procedure as to how to do this remotely we also modified the interviews these interviews typically done in a. Site visit we had to modify them for doing them remotely and we also remove three items from the fidelity scale in the past year and we also modified some of the anchors on the fidelity item. So I'll give an example of an item and later on so to conduct the remote fidelity assessment we developed a process for data gathering and in that we required a certain level of training so we held two. Training sessions one training session is for the site team leader and this is an opportunity for us to tell them what to expect in a remote fidelity review and then we needed to hold a second training session and this would be for the person that we would identify at the. Site who would be responsible for collecting client level data because we are dependent so dependent on the sites for providing us with quality level data we felt providing them with a manual wouldn't be sufficient and so we had one-on-one trainings. With each team leader and each health record reviewer at each site then. The data gathering methods for conducting a remote fidelity review where we conducted telephone interviews we reviewed programmed documents that were sent to us from. The sites and we reviewed client level data so the first is the telephone interviews where we interviewed four staff members that would be the team leader the therapist. The support employment of spoted supported education specialists and the prescriber and this isn't saying that these are the most important team members or the key key roles on the team rather we wanted to keep the burden as low as possible when doing the remote fidelity review and we. Felt that interviewing these four staff members would give us the the spread of information that would be required to to complete the review in some cases some rules were split and so we would have to interview a few more staff members or some staff of that there. Have been times when I would interview someone and they wouldn't be able to speak to certain questions and so I'd request to speak with another staff member to to get information. On those certain areas so. Next is the documentation we requested programs to send this. To us electronically and we also asked them to send us this information in advance it allowed us to prepare for the interviews and I should say that this also would be used to corroborate information that would be given in the interview and also vice versa then the third area of information. Collected for the fidelity review was the health record review and so we would have someone a staff member who doesn't provide clinical services to any clients on the team but who is familiar. With the sites electronic medical record to complete the health record review and it would be providing us with information on about 13 of the 31 items on the scale and so I should say so these three sources of. Information would co-operate each other for example an example it is a family psychoeducation we are expecting that the families should be receiving about 8 sessions of family psychoeducation. At least during the first year of a client being enrolled in the program so on this the health record review the health record reviewer would tell us for each client how many sessions of family psychoeducation they received in the program documentation we would be able to see the family psychoeducation materials and then in the interviews we would. Ask them what method or what form of family psychoeducation would you provide and the reason this was important was for example the health record reviewer may complete the list. Which was which would be correct but then when we go and do the interview we find that the clinician or the people. Who are responsible for about providing family psychoeducation never had any training or that they maybe don't provide any family psychoeducation and so with that information we would then know well actually the site even though. The health record reviewer has indicated that a client received so many sessions the site isn't actually providing family psychoeducation and i want to briefly show you a sample of an item this is. Our supported employment item as you can see it's rated on a five-point scale some of the items we made a lot of improvements to these because the. The initial scale items the anchors were more qualitative in description we found that it's a lot easier if you actually just have a list of expectations that you have for a given practice so in supported employment we had a list of eight expectations and then we made. The anchors as to how many items expectations I should say did each did the support employment or supported education specialist meet in order for us to do our rating which made it much simpler so then we. For five of. The sites we had four raiders in order to test interrelate greater reliability and in general the reliability ranged from moderate to excellent and the site's actually had a very good spread in delivery of services which the fidelity scale captured as you can see in the range of the total. Scores so the remote fidelity assessment revealed several benefits one the first. Being that it was cost-effective there's no travel no flying time there also there is fewer staff hours lost so fidelity reviewer isn't losing a day. To travel to conduct the fidelity review also at the site oftentimes when you're doing a site visit. You need to have a staff member sort of babysit you a bit and make sure that everything is all right with the fidelity reviewers that's. Not necessary if you're not. At the site it was also accommodating to staff work schedules we I had interviewed someone while they were in the car I had interviewed people at the end of the day of their work after they were finished work whatever was most convenient for them I actually conducted a training while I was in the car driving which. Wasn't very good but it only happens yeah that wasn't bhaiyya that was not by choice but that only happened once and also that it was another benefit is that it's efficient when we have to do. Client chart reviews when we go to a sites we're navigating through paper records and electronic medical records that were not familiar with if we have a staff member do it it just reduces the amount of time because they. Know where to find everything and also another benefit to doing it remotely is that we had information in advance of the interview so I was able to prepare for the interviews and and kind of have more focused questions because I already had. Quite a lot of detail in advance but I should say this last point also can be seen as. Limitation because there are instances that we didn't get the information in time for the interviews and and sometimes we didn't get the information for a couple of weeks which delayed the rating. Of the in completing. The fidelity review which you wouldn't have if you went to a site visit but then there are limitations and the largest limitation we saw for a remote fidelity review is our inability to make any observations we couldn't make any behavior observation as to the how well sight is actually providing the quality that which. They're providing the services we didn't have any client or family perspective so it was very difficult to assess the level of shared decision-making that was happening at the site it was difficult to assess their also their perspectives whether or not their. Goals are being in corporate and they feel their goals and needs are being incorporated into their treatment plan and also for the client charts we don't have a first-hand review of the charts yes we have someone. At the site doing the review but we don't know if maybe. The team leader reviewed it before they sent it to us so we just that bias could still be there there was also some scale limitations which we are working through and that some of the items do require some subjective ratings and some of the the ratings required. Us to be proficient in in the different manuals and for me I just started so I wasn't proficient in the on-track navigate East and manual and the different terms they use for different therapies is IRT considered cognitive behavioral therapy does that work for this item so it required this extra knowledge that I just. Did not have so those are those are some things that we need to work on with the scale and also we don't have evidence for predictive validity but this. Moves to our future directions in that in year 2 we're going to be modifying the scale and also in year 2 as prithi mentioned. Is that when we're good we're going to be collecting the outcome data and that's won't be able to develop evidence for predictive validity and I should say so these were the limitations that we observed but I would be very interested later when we have it's open for. A discussion for you if you observe any other limitations or things that we should be. More aware of when doing a remote fidelity review and I'd like to acknowledge our fidelity team we have Gary bond and. Matt Landers and also Don Addington who's based in Calgary he was the developer of the first episode psychosis fidelity scale and was all these members were were key in in making this successful so thank you all right Thank You Tara we have time for you to give your comments and then we'll be able to open up to. The audience for discussion and questions okay well thanks everyone for hanging out and thank you for allowing me to pretend to be at least at Dickson for the day it's amazing its vigor rating to be her so as was mentioned during the. Various presentations you know my view here comes partly as a researcher but also as a stakeholder in this process so my perspective is informed by. Both of those things and I was interviewed I think as two roles because I am the executive director which means I'm an administrator but I'm also the team lead so I'm a supervisor and trainer as well for our team so I want to put that out there as a stakeholder in this process. I want to highlight. My appreciation for the approach that West that took there mixed models approach I thought it was really nice to not just feel like I was giving. You numbers but that you were gonna talk to me about not just the fidelity but the experience and why we do what we do and I think today it was really nice to see those relationships Illustrated. So I think there was a lot of value and the way they set the study up and so they did it with this idea of trying to. Capture the richness but also to make it doable for sites to participate they you know the the outcomes that Avram showed they're not terribly complicated and it was. Easy for us to do which I think for all of us who are trying to serve as many people as we can in our community and yet also wanting to contribute to a large study it made it more feasible for us and I really appreciated the telephone engagement pieces that we're done I think that it all. Of the reasons you pointed. Out in terms of the flexibility for staff to be able to engage when. People were available like if you want to talk to my psychiatrist. I can tell you that's a very specific time so it allowed us to make more people available to provide information and I also really appreciated the way the chart review was done I understand from a researcher perspective how you were like we'd like to look at the charts ourselves you know. And really try to confirm if this is what we're trying to measure but for us with IR B's and all of those things. It was so much easier to keep the pH I within our system and to just provide that data for. You so I thought they had this really great balance of trying to get as much data as they possibly could while also respecting the clinical process that was happening at the sites so in terms of the data that I saw and this is my first time seeing this data - I was really struck on the survey data the. Graph of the increased number of sites that really correspond it to the increase in the money and. It made me think about this morning's presentation about how money really don't drives the delivery of care and. I know in California and we already had money for this work and so there were already a lot of sites doing it but then there were more sites doing. It because of the Block Grant dollars and that the explosion across the country as a result of that I thought that was interesting so there's this money that's driving care but then the survey data also. Showed that there were three FTEs in general usually serving 10 to 40 patients across these giant catchment areas so for all it to me it really spoke to what I see in Sacramento County as well as in. The adjacent counties that we collaborate with this how. Do we do as much as we can for as many people as we can with as little money as we can the mental health Block Grant dollars are actually very small and I know for some California counties the rural and very very small counties the portion that they get from our state it's really hard to do anything. With that and so what we've seen programs do is divide it up in teeny tiny chunks across multiple staff members and I think that this is another way in which money. To me is diluting care because it is hard to create that team-based flexible approach and to make sure that people are delivering the services with fidelity if they only have point one of their FTE you know devoted to the service so that was sort of an interesting piece I also it that the. Small number the small patients with these large catchment areas really highlights to me that we're probably not serving as many people as neat. Service I know that's the case in Sacramento County and so they're you know the mental health Block Grant dollars are great and I think they're not sufficient to really serve all of the folks who need these services in the survey data I was also struck by how there were fewer programs who were integrating physical health or cognitive remediation those. Are at the low end and those are evidence-based practices in early psychosis care we saw. Some really great data on physical health and serious mental illness today and they aren't. Coming in in to these programs and I really think that speaks to the divide between research and practice that we were. To have been talking about today it also we saw the same thing in we did a survey of California counties and asked them you know what are you doing and we saw the same thing this is what's published in early intervention in psychiatry with mark Savile is the first author and those services fell to the bottom in. California so it's consistent with what we were seeing okay in terms of I sort of combined the survey and process data in terms of talking. About the catchment areas. So one thought that I was wondering about pre thing with your. Data the staff being split over the roles and. All of the accommodations they were making when I talked to providers in the space there's a hypothesis that that's contributing to increased turnover and so I was wondering if. You have data. About turn I don't remember giving turnover data but I can't remember and so I was just wondering if there's any association there that might be helpful to look at because that's something we're talking about a lot I've mentioned that I really appreciated the fidelity approach. And I was not surprised to see that less than or about half of the programs do regular fidelity measure I think that is challenging I. Think it requires a whole nother level of data. Collection that's that's hard to do when you're trying to drive your money towards care and so I I agree with you I think the faps is. A very nice model to really capture components of care that isn't modeled driven but then I noticed in your the one item you put up which was the supported employment that it had IRT it had IRT is that one of the did you or IPT individual placement and training IPs it was hard to see from my corner yeah. It was it started with a knight IPS individual placement and support and so that is a particular. Model right and so I looked and it was like that entire model or item was rated on my PS and. I was like well we didn't get that model because we didn't get that item because that's. Not the model we use and so while you picked something that was not model driven you then put models in and assumed that that was the right way to go so I'm just wanted to point. That out we because of our experience doing this evaluation and the way we did we're using the remote assessment fidelity model in the California evaluation that we're developing right now and so we're working with Don Addington to. Be able to translate that into an evaluation of more California programs I guess in summary I was sort of thinking about how do we take this information and translate it into practice now how do we you know for example in. In previous presentation how do we take this knowledge that you've captured about the ways in which programs have successfully modified CSC and disseminate that to more places and I've bet had the privilege of being part of some of the Nashville and Samsa. Webinars and I'm like oh is this something we should be telling. People if you live in a rural area here's. One way in which you could serve more people or modify things I think that's something that many programs are hungry for I'm also interested to see your future analyses related to whether specific the. Presence of specific components is related to specific outcomes I think that'll be a very interesting analysis I'm also curious as to how this this project will impact the disparity that we still see in the provision of CSC care for individuals with insurance because. You pointed out in your slide that CMS has approved first episode care and the majority of the programs that you surveyed are Medicaid programs but I also have an insurance based program. And I can't bill for the majority of the services so we aren't seeing it translate into private insurance paying systems and that is creating a disparity in services for people so I'm wondering if. This data is gonna help to drive that conversation forward and then. Finally the EPI net RFA was posted this week and we were sort of talking about like how is this going to drive the the proposals that may come in for epi net and I. Think one piece that I would like to suggest as I think you guys have developed a fidelity assessment approach that. Is really great and. I know I would use that and I really hope that respondents to that RFA also really think about how to best include the clinics and get data from the clinics in a way that was as unobtrusive as possible which I think you guys also demonstrate it so those were just two points that I thought I would make so. Terrific thank you I will say one quick thing and open it up one of the things out of the. Process interviews obviously as. We spend a day usually in each of the places and we were very fortunate I did some pretty just some we had Howard Goldman and Dave Shirin and Lisa Dixon doing these interviews when we sit down and. Talk about the interviews and what we learned it has many of the things that you touch upon came up in those interviews and how we're gonna code and capture that's going to be really very interesting so you saw sort of the tip of the iceberg that won't sink any ships but then. You know has a lot of information to it and you raise some really important points that we want to try to pull out of there and also linked to our fidelity process so I don't need to say more I just wanted to mention mention that briefly we are. Happy to and retained questions or have any discussion from anyone that would like to do so thank you for the very great presentations and thanks for sharing early data with us very informative and also appreciate the great discussion my name is Johan from white Cornell Medical College and I started working. With Lisa Dixon's group a few years ago I came from a health economics background and so very concerned about our policies in particular payment. So we are presented some early you know a prototype of a payment design support tool yesterday to design the p-pardon and especially a case rate based payment for CSC so you know where I came from so my questions are related to the financing side. Of CSC so the first question is in your you know site interviews did you ask about their their revenue sources other than the manor-house Block. Grant what what is the general sense of where they kind of get coverage or financial support to run the team the other question is regarding the. Staffing side so we've talked about a lot of the accommodation or you know like split studying rose among different clinicians the different you know FTEs but I wonder if there's also large variation in terms of what. Credentials what types of professionals are serving a particular role you know for example for the team leader can be you know that role can be played by a clinic. A licensed clinical social worker or you know some kind clinical social worker right and you know etcetera so I wonder if you have any insights gleaned from your visits to share thank. You those are great questions and I'm gonna start an answer and let anyone else chime in how I know you run one of these programs or a couple of them so you have a few comments on the financing side. We did ask about where the dollars came from you saw some some initial indications in the site survey you know the usual suspect Medicaid private insurance and then in the process we had to be asked more detail so for example in California there's a whole funding process that's not replicable elsewhere through the Mental Health Services. Act in that state that's very important. To the programs that we interviewed in. California but there's no such thing in other states and so you can't ask always canned questions because you never know what you're gonna find when you go out there which is why we did the process never but we did ask and I don't want to speak out of turn because we haven't really consolidated our results there yet but. I think it's safe to say and chime in if I've got this right that we saw a wide range of different ways that these services are funded they tended to rely if they didn't have special things like California did on Medicaid private insurance created a lot of challenges usually for the sites depending on how they were set. Up although we encountered some sites that had no Medicaid access virtually none and we're relying on their private insurance or their Hospital payment systems of various ways to fund it so one of the problems with this and. And Howard I just finished a paper about this is the funding mechanisms for all of this are very complicated because you're moving. From the child system right to the adult system that requirements on the child system are different than the adult system people transition and magically turn into adults according to the funding mechanism but they don't for the programs and then they have different qualifications and Medicaid on different sides depending. On whether you're a kid in its early period I've screen detection treatment or you're. An adult and you has to be SMI based. And so there's a real hodgepodge which is I'm sure what you have seen in. How these places are funded and they've generally done what they can if they don't have something like the Mental Health Services Act to braid these things together in these sort of I'll just say editorializing a bit kind of fragile feeling ways you know because there isn't a set way of. Paying for this that transcends the age groups and goes forward so that's item one and I'll let you guys chime in on that if you like yes so I can speak to the. Second so just to clarify you're asking about diversity of staff roles and and training and whether or not we saw differences across the sites is that correct you want sites to be to accommodate their local you know situations right and it is teams are doing that in fact but on. The other hand you don't want them to. Deviate too much from evidence-based approach I mean you think you know how you staff a team so I wonder if you know if you see if there were like called a typical team makeup that you saw from your okay well I didn't say that well starting with the the team. Leader most were it's a master's level couple of PhD level counselors I think yes so most had like a master's in counseling for fidelity but you get the highest level of fidelity we expected the team leaders not only to provide. Supervision but to also provide direct services to clients so to be quite completely connected with the with the clients that they are serving there were some instances where we had the prescriber who acted as the team. Leader or like at some sites you'd have psychiatrists other sites had nurse practitioners for the. Clinicians there was some variability there as - you'd have mostly the social workers I would say like master's-level social workers acted as clinicians I think. What really would Oh in an addition sometimes he's also had nurses not all the teams did but that would that that that that would speak to the principle of primary care services in the coordinated specialty care. Model the nurse would be able to to address to dress those and then there also appears on the team and family support workers. But the thing that really would differentiate and keep the quality of that the services high would be the level of supervision who are you receiving supervision from what sort of training did you receive are you receiving like Abram was talking about a lot of sites continue. To receive ongoing consultations from the on track or navigate or ISA I would say that was was where we did see some some distinctions that some sites were having that high level of supervision from external sources and also internal supervision whereas other sites less. So yeah I would I would say the only thing I would think about from a health economics perspective because we did a pilot analysis in Sacramento County look at looking at cost utilization and outcomes associated with the care that we provide in Sacramento County compared to two outpatient clinics and. One of the things we had a very hard time capturing the cost of was the training and the ongoing training that's not something that or you capture in a billing code. And so it makes it really hard and sometimes. It's reflected in a budget and then sometimes it's not and so one of the things that we see often times in our contracts with other counties is a challenge like a struggle for them to keep their staff while also needing to support this ongoing. Training and supervision and so I think that's in terms of understanding the economics of these programs that's one cost that we really need to think about how to quantify because it's sort of. Hidden in the program but it's it's what makes or breaks the program yeah I just real quick we did see challenges for sure and the training side there are only so many people from on track in these places and they would come out. And do the training staff turned over which happens they had to read train and that was a challenge for. A lot of sites but just the reasons you spoke at they had to allocate resources and then they couldn't figure out how to pay for it again we did see some of that Susan observations fidelity assessments and maybe Valerie and what you'll do in the do with the findings. In terms of providing feedback to those sites like will they get their fidelity ratings and what would they be presented to them and in the context of here's how you did and here's the aggregate or how will you do that so that it's you know informative and quality improvement way that's such. A lost art that's such a great question and we're kind of laughing. Up here because we just had a extended discussion with with Gary bond about and Valerie and the team about what we could give back to the sites and what we could afford to give back to the sites which. Are sort of different things and how we could do that we sort of decided and tell me if I got this right so we would do this at time too because we felt like go go ahead Valerie yes in terms of detailed feedback for each specific to each site we thought that we would do that we will do that. In in year two we in year one what we're going to do now I think we're gonna put together just in an overall summary of the range of services that that we saw and provide that to the site so then it will be on for them to reflect looking and seeing where they would fall into how're we described what. We observed another reason we didn't we we aren't quiet the stage to give individual sight feedback because of the we're still developing the VF EPS scale so there are no benchmarks we couldn't tell the site you know you met fidelity or. You did not so it wouldn't be quite appropriate to provide a fidelity score at this at this time in short we're proceeding with an abundance of caution while remaining enthusiastic you know that's great so I can talk about so just. Time burden overall just with the interviews team leader interviews were about an hour to an hour and a half the prescribers and supportive employment specialists. Were about 30 minutes each and the therapist interviews were about 45 minutes to an hour so that adds up to what three and a half three and. A half hours for interviews then we surveyed for the health record review. They need to review 10 10 charts and that ranged on average about three hours to. Do and then collection of the program document so these needed to be submitted electronically that really ranged so I had one site tell me that they did in 15 minutes and probably the reason that is that they already had everything on their computer in electronic form whereas on the other hand one site it said they. Took them two hours to compile compile so in general I thought that the time burden was pretty pretty reasonable mm-hmm oh and then the training so the team leader I met with the team leader for half an hour and then the health record reviewer for half an hour and then there. Was time on the the team leader side working with our research assistant to schedule everyone. So the scheduling I'm sure it takes this takes a lot of effort I'm very fortunate for Matt who's up there he organized all of. That yeah and just one add-on real quick one of the things that really drove some of our process around the. Fidelity interviews at the beginning at least was the IRB concerns very difficult you know for us to have people seeing charts and we talked about. You know sort of D identifying them at the. Minute you don't de identify something you should have we've got all kinds of issues and so we just decided we were better off not seeing anything which did shift some of the burden to the sites but probably I think Terrier sight in the long run didn't because we didn't get into these. IRB problems that would have just probably waylaid us and so that was part of how we ended up with this sort of remote control version of doing this and us not seeing the charts it just on a study with 36 sites you can imagine it's like anything that can go wrong will. And would have and then we would have had to have reported that and dealt with all the consequences and stuff yeah my name is Victoria doyon thank you so much this has been a very. Very rich very contextual session happy to hear it is such a right formative stage and since i if i heard correctly we're hearing it for the first time and the question was what could you give back and i would say that this discussion itself may not be though it's being recorded and. Will be available for digestion that may. Be that that might not be a bad tool for outreach hmm an 88 percent response rate in the science that we. All are about deep bow and it suggests. A really vested audience and coming up through the public health system. From county to stay to regional national level it's also at are really helpful to hear about that diffusion of the funding stream i cut my teeth in public health in minnesota and i know what you know the northern reaches you. Know inner county collaboration looks like to try to spread the wealth and so this is maybe just a comment but a thought for the push/pull in the data sharing and when you have this qualitative investment Valarie that you've spoken so clearly too often times that audience will have things will resonate downstream as a consequence of what you've what's. Been asked of you so i think you've asked a really really provocative question what can we give back even in these early stages and if it's not too idealistic you know this breakout session the kinds of questions that have been shared the feedback from tarah even if it's for that. 88 percent responders you know here's some here's some food for thought and just to invite the comments the technology presumably is there whether there's remaining funds in the grant to take a look at what might come back qualitatively from your stakeholder. Engaged participants at this early stage for time to if that makes sense because I think that there's real opportunity in. Where those dollars are spread out at the 0.01 FTE a and then B is it's been rolled out through this extraordinary investment from the federal agencies at a national snapshot. But maybe there's some synergies that are regional that could pop up for exactly that would kind of get at that the the economics of it right thank you so much thank you for the comment thank you very helpful we're just about at a time we've got five minutes if anyone else would like a last final question for the. Afternoon okay if not. Thank you so much for joining us this afternoon really appreciate it and have a great rest of your day and we'll be paying around if you have any of further. Questions thank you .


Westat Company News

Fri, 12 Apr 2019 07:00:00 GMT
Sexual Misconduct Climate Survey Firm Paid $1.5 Million in 2014 Employment Discrimination Settlement | News - Harvard Crimson
Sexual Misconduct Climate Survey Firm Paid $1.5 Million in 2014 Employment Discrimination Settlement | News Harvard Crimson The Association of American Universities released its second national sexual misconduct survey April 2 in collaboration with Westat — a research firm that faced ...
Thu, 25 Apr 2019 07:00:00 GMT
Rockville Restaurant Inspections: Potomac Pizza, Robert Frost MS - Rockville, MD Patch
Rockville Restaurant Inspections: Potomac Pizza, Robert Frost MS Rockville, MD Patch Rockville Restaurant Inspections: 5 Eateries Cited - Rockville, MD - Five eateries were out of compliance. Here are the latest restaurant, school, and business ...
Fri, 01 Feb 2019 08:00:00 GMT
Texas A&M Encourages Students To 'Take The Survey' To Help Fight Sexual Misconduct - Texas A&M University
Texas A&M Encourages Students To 'Take The Survey' To Help Fight Sexual Misconduct Texas A&M University Texas A&M University students have an opportunity this month to help university leadership gain a better understanding of each student's experiences and ...
Mon, 04 Feb 2019 08:00:00 GMT
2019 student survey will assess campus climate on sexual misconduct - Yale News
2019 student survey will assess campus climate on sexual misconduct Yale News In collaboration with the Association of American Universities (AAU), Yale today announced the launch of its 2019 Campus Climate Survey on Sexual ...
Wed, 05 Dec 2018 08:00:00 GMT
Survey finds obesity, mental health are continuing problems for wounded warriors - Stars and Stripes
Survey finds obesity, mental health are continuing problems for wounded warriors Stars and Stripes A new survey of 33000 wounded warriors has alerted advocates and government officials to ongoing problems with veterans' mental health and obesity.
Thu, 04 Oct 2018 07:00:00 GMT
UChicago to participate in AAU campus climate survey on sexual assault, misconduct - UChicago News
UChicago to participate in AAU campus climate survey on sexual assault, misconduct UChicago News The Association of American Universities on Oct. 4 announced that 33 universities, including the University of Chicago, will participate in a national campus ...
Fri, 07 Dec 2018 08:00:00 GMT
Do You Work for One of Washington's Most Charitable Companies? - Washingtonian
Do You Work for One of Washington's Most Charitable Companies? Washingtonian Within days of the inauguration of President Trump, VolunteerMatch—an online platform pairing volunteers with nonprofits—experienced one of the busiest days ...
Mon, 01 Apr 2019 07:00:00 GMT
MIT launches 2019 AAU survey on sexual misconduct, responds to National Academies report - MIT News
MIT launches 2019 AAU survey on sexual misconduct, responds to National Academies report MIT News Continuing its commitment to preventing and responding to sexual misconduct, MIT is taking important steps during Sexual Assault Awareness Month (SAAM): ...
Fri, 05 Apr 2019 07:00:00 GMT
Campus climate survey on sexual assault to begin April 8 - AroundtheO
Campus climate survey on sexual assault to begin April 8 AroundtheO University of Oregon students will receive an email on Monday, April 8, inviting them to participate in the Association of American Universities' 2019 Campus ...
Fri, 25 Aug 2017 07:00:00 GMT
Perspective | Researcher's fight to close achievement gap irked school board for years - The Washington Post
Perspective | Researcher's fight to close achievement gap irked school board for years The Washington Post Data gatherers usually go unnoticed, except for Joe Hawkins.