The purpose of this evaluation as described by North Dakota Guardian ad Litem (NDGAL) Project administrative decision makers was to determine the value and effectiveness of the NDGAL Project and the levels of stakeholder satisfaction with the Project. These administrative stakeholders informally posed a core evaluation question--"How are we doing, and what can we do better?"
An external evaluation team conducted this evaluation. That team selected a standard formative, decision-oriented evaluation design to guide the conceptualization and implementation of this study. That design focused on four Project components--the NDGAL context, resources, processes and products. The evaluators chose methods and procedures that would generate information for use by NDGAL Project decision makers seeking to improve their established program.
This evaluation was conducted as survey research. Both quantitative and qualitative perception data were obtained via printed questionnaires and telephone interviews. Six groups of stakeholders participated in the evaluation:
1) Fifteen judges and judicial referees,
2) Eleven state's and defense attorneys,
3) Sixty-five child welfare workers consisting primarily of county social workers but also including two foster care workers and one Social Services supervisor (CWWs),
4) Twenty-seven guardians ad litem (GALs),
5) Three current or former GAL Project administrators, and
6) Two Supreme Court administrators (Project administrators).
The evaluators sent 272 questionnaires to four stakeholder groups--judges/judicial referees, attorneys, CWWs and GALs. There were 118 questionnaires returned with useable data, for a return rate of 43.4% ranging from 60% for the GALs to 37% for the attorneys. The evaluators conducted telephone interviews with 32 stakeholders representing six respondent groups including: three judges, four attorneys, ten CWWs, ten GALs, three project administrators, and two court administrators.
The quantitative data consisted of the respondents' ratings of the variables included in closed-ended questionnaire items. The ratings were elicited with five-point scales anchored at points 1,3, and 5 (for example,1 = Not very useful, 3 = Somewhat useful, 5 = Very useful; and 1 = Not satisfied; 3 = Somewhat satisfied; 5 = Very satisfied).
The quantitative data were analyzed for percentage of respondents selecting each numerical rating 1-5 and for mean ratings by all respondents combined for each item.
Qualitative data were obtained from two sources, telephone interviews and open-ended questionnaire items. There was a high level of overlap in the respondent feedback obtained by these two procedures. Therefore, the qualitative data from the two sources were combined for analysis and interpretation.
This evaluation resulted in the following findings, recommendations, and conclusions about 1) the value of the NDGAL Project and GALs; 2) the effectiveness of the NDGAL Project and GALS; and 3) stakeholders' satisfaction with the NDGAL Project and GALs.
The Value of the NDGAL Project and GALs
Finding #1: Each of the four groups of stakeholders perceived the NDGAL Project to be of value for child deprivation cases involving abuse and neglect with the level of perceived value being greatest for GALs and least for CWWs.
Finding #2: Each of the four groups of stakeholders perceived the NDGAL project to be of value in cases involving: 1) conflicting case information/highly adversarial parties; 2) the status of a case in relation to ASFA; 3) issues related to reunification plans; and 4) concerns about the implementation of services. Judges/judicial referees and GALs provided somewhat higher ratings of usefulness than those offered by attorneys and CWWs.
Finding #3: Each of the four groups of stakeholders perceived that the input of GALs is of value in informing court processes and judicial decision-making in cases that involve placement, service provision, visitation, and safety issues. Again the judges'/
judicial referees' ratings of perceived value were the highest. Further, the judges/judicial referees indicated that GAL recommendations regularly become incorporated into court orders.
Finding #4: According to 20% or more of the respondents, the GALs have relatively less impact on decisions about 1) location of placement, 2) frequency of visitation by family of origin, and 3) safety of children/youth while in placement and after court dismissal.
Recommendation #1: That NDGAL Project decision makers explore this finding further to determine 1) the reasons for it, 2) the practical importance of it, and 3) any course of action to be taken.
Finding #5: There was a high level of stakeholder agreement that the personal knowledge that GALs have about children and youth and the quality of information that GALs provide are beneficial to court-related decision-making.
Finding #6: There was a high level of stakeholder agreement that children and families are better served because of GALs' involvement.
Finding #7: The levels of agreement of the judges'/judicial referees' and GALs' relative to the three types of GAL contributions in Findings #5 & #6 were consistently higher than the levels indicated by the attorneys and CWWs.
Conclusion #1: As measured against a criterion of needs fulfillment, the NDGAL Project and the GALs are making valuable contributions to meeting the needs associated with 1) different types and issues of deprivation/abuse and neglect cases and 2) various court processes and judicial decisions.
The Effectiveness of the NDGAL Project and GALs
Finding #8: Apparently, the NDGAL Project does not have written statements of mission, goals, objectives, a strategic plan, policies and guidelines, and procedures for ongoing personnel evaluation.
Recommendation #2 :That the NDGAL decision-making leaders formulate, document, and disseminate among stakeholders the following infrastructure as a context for project management and evaluation:
An NDGAL Project mission statement;
Lists of Project goals and measurable objectives with an associated strategic plan;
A policy manual and associated guidelines that address, but are not limited to: 1) the roles and expectations for GALs, 2) the non-roles and non-expectations for GALs, 3) ongoing personnel and project evaluation, and 4) periodic assessment of these contextual components for their relevance and currency.
Finding #9: There is apparently no Project documentation that identifies the existence or identity of the Project's decision-making or advisory individual or group.
Recommendation #3: That the individuals or groups currently responsible for Project decision-making, advisement, and/or oversight take action to formalize, document, and appropriately disseminate their identity, the nature and extent of their Project responsibilities, operational procedures, and contact information.
Finding #10: Relative to administrative Project personnel, there is a high level of satisfaction with the performance of individuals. However, there are concerns that to strengthen and improve the Project leadership, roles of the director need to expand in the following areas: 1) representation and promotion of the Project with other organizations and state agencies; 2) data analysis and research on GAL Project effectiveness and impact; and 3) fund raising for special projects.
Recommendation #4: That NDGAL Project decision makers 1) establish the desired target roles and responsibilities of the Project director; 2) determine the nature of the director's current workload; 3) analyze and resolve discrepancies, as reasonable within the limits of actual and potential Project resources; and 4) establish evaluation procedures for the directorship.
Finding #11: The data do not clearly support the need for additional GALs except for the concern that Native American GALs are needed in some areas of the State.
Finding #12: The data reveal inconsistent preferences, expectations, and practices across the Project relative to the duration of the GALs' service on cases raising concerns about whether cases are being consistently, appropriately, and economically monitored.
Recommendation #5: That the NDGAL decision-making leaders 1) formulate, document, and disseminate a Project policy regarding duration of GALs' service on cases including procedures for appeal and/or exclusion; and 2) adopt procedures for determining the impact of the policy on "children's best interests" and Project funding.
Finding #13: Non-personnel resource issues addressed by stakeholders included: 1) funding for the project; 2) administrative costs of the Project; 3) the limit on the number of billable hours; 4) reimbursement for work-related expenses incurred by the GALs; and 4) non-electronic data recording and analysis.
Recommendation #6: That the NDGAL Project decision-making leaders continue to seek increasing fiscal support for the Project from both traditional sources as well as new sources to supplement the traditional sources (e.g., benefactors, sale of educational services, and interagency cost-sharing).
Finding #14: A high percentage of the stakeholders rated the GALs to be effective in three areas of performance and activity: 1) GAL understandings and work performance; 2) GAL contacting and interviewing various parties; and 3) GALs' case-related activities. Mean ratings of effectiveness were generally higher for the judges/judicial referees and GALs than for the attorneys and CWWs.
Finding #15: The extensive stakeholder qualitative data relating to GAL effectiveness offer excellent insight into 1) reasons for participant ratings; 2) majority and minority points of view; 3) perceived strengths and weaknesses of the performance of GALs; 4) GAL performance that is valued and should be sustained; and 5) opportunities for improving the performance of GALs.
Finding #16: Throughout this report the comments of respondents from all of the stakeholder groups include perceptions that GALs have provided services of higher quality than those provided in the state by their attorney predecessors.
Recommendation #7: That over time, the Project decision-making leaders revisit the stakeholder qualitative data relating to GAL effectiveness to identify and prioritize Project objectives for 1) sustaining valued GAL performance and 2) pursuing opportunities to improve the performance of GALs.
Finding #17: Stakeholder feedback on the training of GALs addressed the following areas: 1) strengths and weaknesses of the present training and 2) the need for training on court and legal systems, on types of cases and issues, and on a variety of other specific topics.
Recommendation #8: That the Project decision-making leaders immediately review the stakeholder feedback on the training of GALs to determine if there is input that has relevance for the planning of the July 2007 NDGAL Conference.
Conclusion #2: Through context, input, and process evaluation based on stakeholders' perceptions, it is concluded that the NDGAL Project is supporting GALs at a level and in ways that are resulting, overall, in their rendering of effective services to children. However, this report provides several recommendations addressing contextual issues, needed resources, and aspects of GAL performance and training that merit attention for the improvement of the Project.
Stakeholders' Satisfaction with the NDGAL Project and GALs
Finding #18: Overall, both quantitative and qualitative data indicate a high level of stakeholder satisfaction with the Project and the GALs.
Recommendation #9: That over time, the Project decision-making leaders revisit the qualitative satisfaction data to identify and prioritize Project objectives 1) for sustaining high stakeholder satisfaction with the Project and the GALs and 2) for opportunities to improve the Project.
Conclusion #3: Product evaluation based on quantitative and qualitative measurement of stakeholders' perceptions revealed a high level of satisfaction with the NDGAL Project and the GALS but was accompanied with additional suggestions for improvement of the Project.