Evaluating Programs and Human Performance

Thoughts on Section III to evaluate my instruction:

Well, isn’t this a timely assignment with the release of the STAAR scores? It certainly is a time of reflection… what worked, what didn’t. And why for each of those reasons. After many hours of searching for other evaluation models, I decided to reflect on the logic models and outcome based models.

Logic models look at how one component of a program affects another. “Educators can use these representations to plan evaluations, monitor the implementation of program activities, and determine the extent to which programs have their intended effects.” (Lawton, B., Brandon, P.R., Cicchinelli, L., & Kekahio, W. 2014) Logic models help programmers understand the program’s activities and intended outcomes. Because the budget is always a concern, the logic models help educators decide “which program features are most crucial and then develop evaluation questions that address the program features and their interrelationships.” (Lawton, B., Brandon, P.R., Cicchinelli, L., & Kekahio, W. 2014) Sometimes the evaluation questions become a starting point for the logic model. Other times they arise during the process and might need to be clarified into clear, specific, and actionable evaluation questions. These often help to determine what resources are needed as well. The logic model is a series of inputs (program design and goals), outputs (interventions or processes), and outcomes (results or impact.)

The following chart is helpful in understanding the logic model of an elementary science program.

There are four main types of Outcome-Based Evaluations (OBE): program evaluation, effectiveness evaluation, impact evaluation and policy evaluation. I felt that the effectiveness evaluation was a better tool to evaluate my instruction. The effectiveness evaluation asks, “Is my program meeting its goals and objectives?” There are many factors to consider when using an OBE. One is taking into consideration the intended audience and what they already know. OBEs make sure that there is a clear goal or intended outcome of the program. There should be inputs and resources as part of the program. (training, materials, staff, etc) The output of the program should be measurable and directly affect the outcome which looks to see if the target audience was changed or improved as a result of the program.

When I reflect on my own instruction I was well-equipped in terms of training and resources. My principal spent additional funds this year to have all lower elementary trained in guided reading. We attended workshops, had experts brought in to coach us, and professional and student books were purchased to make sure we always had what we needed. When I planned my lessons, I first made sure that I knew what the goal was of the TEK. With certain skills, I gave a pre-test to survey what the students already knew. Then I consulted my resources of some of the experts in the field of reading: Jennifer Serravallo, Franki Sibberson, Stephanie Harvey & Anne Goudvis, Kylene Beers & Robert Probst, etc. I used varied the components of in my lessons from the multiple intelligences to technology. My lessons typically began with a direct-teach mini lesson followed by small group or partner work before independent practice. During the practice phase I used formative assessment to determine which students “got it” and which students needed more practice.  At the end of the unit, I would assess how much the students had learned either through a reading passage, or authentic assessment where I observed if the students were actually using the new knowledge. For those that needed additional support, I provided extra small group practice during class and in addition to the school day. I also had the students track how well they did on each TEK with a chart so that they could take ownership of what they needed to work on and be proud of their progress. I am very passionate about reading. I feel like my students increased their love of reading this year. I know that because I had several parents tell me how much their child loved to read after this year, and I could see the excitement on a child’s face when we talked about the books (s)he was reading. I don’t love teaching to the STAAR test. While I know the students and teachers have to be held accountable for their learning/teaching, I don’t feel that drill and kill of reading passages are the way to teach a child how to read and more importantly, instill a love of reading. Did the output meet the goal? For the most part, yes. My “mastered” scores were 10 points above the district average. Was there room for improvement? Yes. It’s a constant struggle to achieve higher performance outcomes from the kids for whom reading is a challenge. I am constantly seeking what I can do better to work with these students.

I believe there are questions that need to be asked in the evaluation of standardized testing. Do the  high stakes tests promote a healthy learning environment? If not, how can we assess the learning differently in a more authentic yet measurable way? Are our schools producing creative, analytical, problem-solving, collaborative thinkers? If not, what do we need to do to change that? Do ALL of our schools have the resources they need? If not, how can we change that?

Of course return on investment should be a consideration of any program. However, that may look different in the field of education. Value is not only determined financially but also by data points or intangible benefits. Intangible benefits actually show a return on investment by creating happy employees (or students) and should be an important consideration. If the employees are happy, they are more likely to stay and yield higher productivity. Employee benefits at various companies such as Google, Netflix and Microsoft offer benefits such as unlimited vacation time, massage spaces, free commutes, recreational spaces, and bring your pet to work, just to name a few. All of these companies have realized that when they create an environment where their employees are happy, they can do their best work. Companies have then seen an engagement in work which is associated with feeling valued, secure, supported, and respected thus resulting in less employee turnover which equates to less money spent on training, improved teamwork, and fewer complaints. Similarly, when our students and teachers feel valued, secure, supported, and respected, there is better engagement and higher performance outcomes.

Performance Problem

Year after year I struggle with how to help my lowest readers. I read books every summer and search online for best practices. I have conversations with my colleagues. And I feel like I have some good instructional tools in my tool belt. But nothing makes a difference like engagement. And all too often I get away from this important component by focusing on the strategies. After reading Section IV, I was reminded of how critical the social part of informal learning is. The biggest obstacle I see with my lowest readers is that they don't seem to want to learn to read. But I know better - if these students were able to seek out what they are interested in and given time to talk about it with their peers, I would learn what interests them and their interests would propel them into engagement which would further their learning to read. Every year I have wanted my students to blog. However, I have shied away from it for two reasons: afraid that we don’t have enough time and would the parents support a “public” social venue. But now I see how blogging could provide just the motivation that these students need. In order to read and comment on another student’s blog, they must know how to read and spell. In order to read a blog on a topic that interests them, they must know how to read. Providing a social time to talk to classmates about what they are working on generates a shared interest and furthers motivation. Informal learning and on the job training at its best! I can’t wait to implement this next year!

References

Reiser, R. A., & Dempsey, J. V. (2012). Trends and Issues in Instructional Design and Technology (3rd ed.). Boston: Pearson.

Lawton, B., Brandon, P.R., Cicchinelli, L., & Kekahio, W. (2014). Logic models: A tool for design- ing and monitoring program evaluations. (REL 2014–007). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Region- al Assistance, Regional Educational Laboratory Paci c. Retrieved from http://ies.ed.gov/ncee/ edlabs.            

Wang, V. C. (2010). Assessing and Evaluating Adult Learning in Career and Technical Education. Retrieved June 18, 2017, from http://web.utk.edu/~ewbrewer/pdf/books/Evaluations%20Models%20for%20Evaluation.pdf

Voelker-Morris, R. (2004). Outcome-based Evaluation: Practical and Theoretical Applications. Retrieved June 19, 2017, from http://pages.uoregon.edu/culturwk/culturework28.html

Winsor, K. (2008). Top Perks: Working at Google. Retrieved June 19, 2017, from http://saleshq.monster.com/careers/articles/1012-top-perks-working-at-google?page=12

Maher, L., Special to CNBC.com. Updated by Marguerite Ward, & CNBC.com, S. T. (2015, August 20). You won't believe these company perks. Retrieved June 19, 2017, from http://www.cnbc.com/2014/12/18/you-wont-believe-the-perks-some-companies-are-offering.html?view=slideshow&%24DEVICE%24=native-android-mobile

Barr, J. J. (2016, October). Developing a Positive Classroom Climate. Retrieved June 19, 2017, from https://www.ideaedu.org/Portals/0/Uploads/Documents/IDEA%20Papers/IDEA%20Papers/PaperIDEA_61.pdf