Rapid Hub Analytics: A Focus on Continuous Improvement

Carnegie has made considerable progress on developing an information system that integrates real-time faculty and student data with institutional records to inform continuous improvement as we address problems of educational practice and constantly improve our ability to serve developmental mathematics students at community colleges.

For example, we are designing a system that will incorporate institutional records going back to 2008 on the longitudinal performance of cohorts of students designated for developmental mathematics at each of the 30 colleges participating in our community college mathematics pathways initiative. These data constitute a baseline for understanding institutional performance over time, for establishing college-by-college improvement targets, and for exploring the antecedents and conditions of performance going forward. Building off of this, these two pathways — Statway™ and Quantway™ — draw additional detailed data from weekly faculty facilitator calls, faculty forum discussions, two-minute faculty and student surveys reported on a lesson-by-lesson basis, supplemental background data on students, dispositions toward success and math tests of fundamental concepts, follow-up student surveys, data feeds from the out-of-class digital platform that include student homework and other out-of-class activities, common topical and end-of-module assessments, as well as end-of-course assessments and course grades. It is all data for the improvement of teaching and of the materials.

Analytics is now prototyping continuous data feed reports to faculty on their classroom context and individual student progress. Identifying places where rapid interventions might occur is one driver in our overall plan for advancing student success.

Specifically, analytics address four broad purposes:

1. To inform faculty so they can more effectively address the needs of students – Routine reports describing the “classroom ecology,” including students’ language background, interest and engagement in mathematics, and entering levels of basic mathematics skills can inform faculties’ curricular and instructional planning from the very outset of the course. Predictive models developed through analytics can forecast students’ needs (including real time indicators of engagement or disassociation) to enable faculty to tailor interventions and support services for individual students. 
2. To identify needed improvements in the instructional system – Data from numerous sources, especially those that track student engagement and achievement, but also including frequent faculty and student feedback, are analyzed to determine where instructional improvements may be needed. This includes possible changes to the instructional kernel itself — curricular materials, the out-of-class platforms and assessments. It also includes identifying areas of faculty knowledge and skill that may require attention in the advancing quality teaching strand. 
3. To test proposed changes as potential improvements – As modifications to the instructional system are proposed, they are prototyped and rapidly tested using the techniques and methodologies of improvement research to ascertain whether these changes can be warranted as improvements. In this manner, Carnegie aims to ensure effective implementation across locations and contexts. 
4. To examine and test for impact and accomplishment of the whole effort – The data that are gathered to enable improvement also allow us to examine the overall impact of the Pathways initiative. We are now in the process of developing value-added estimates for each classroom, college and sub-group of students and will compare these against benchmarks established this first year and against results for similar students not enrolled in the pathways in each college. We intend to continue to track the success of students post-pathway through the following academic year. Understanding “what happens next” for students is another key to improving what we do with them while they are actually enrolled in a pathway.

Our overarching goal is to enhance the capacities of participating faculty to engage in systematic program improvement. We pursue this both for the sake of their fullest participation in the Statway™ and Quantway™ Networked Improvement Communities and to develop the overall capacity of member institutions to use the approaches, tools, and techniques of quality improvement to enhance the diverse program implementation efforts in which they may engage. Ultimately, the biggest payoff of all may reside there.

————

This post was written with contributions from Carnegie colleagues.

Learning Opportunities: Productive Struggle, Explicit Connections and Deliberate Practice

If you were able to participate in the webinar, “It’s Not Just the Curriculum: Developing Pathways for Student Success in Community Colleges,” in January, you heard that Carnegie’s work in developmental mathematics in community colleges — the Statway™  and Quantway™ pathways — is not just another curriculum product or educational technology. Instead, the instruction includes six components. First, we are teaching the math that matters in students’ work, personal and civic lives. Second, given the diverse backgrounds of students in this sector, there are language and literacy supports interwoven through the materials and classroom activities and embedded in the instructional design. Third, we have introduced something called productive persistence to the pathways, science-based activities packaged to increase student motivation, skills and will to succeed in college. The fourth “ingredient” of this mix is learning opportunities: productive struggle, explicit connections and deliberate practice. Fifth, the advancing teaching component aims to provide instructors with the knowledge, skills and habits necessary to experience efficacy in initial use and develop increasing expertise over time. This dimension is essential in seeking to reduce the variability in outcomes among classrooms serving similar students. Finally, we seek to exploit how rapid informative analytics can inform continuous improvements by students, faculty, colleges and network-wide. Operating throughout the instructional system, we seek to tap technology as a powerful tool to advance efficacy, efficiency and personalization in the work of students and faculty alike.

These form a foundation for the lessons within these new innovative pathways for developmental math students. During last year’s Summer Institute, participants in Carnegie’s  Networked Improvement Communities attended a session on learning opportunities. In this session, Kay Merseth, a senior fellow working with Carnegie on a thread of work called Advancing Teaching, prepared an essay that beautifully explains what the three learning opportunities are and what they mean to both students and faculty.

Briefly, in Kay’s words:

“The focus of the productive struggle is on the mathematical learning goals embedded in the problem or situation — it’s not about guessing what the teacher wants to hear or about finding a particular answer. It is about the process of thinking, making sense, and persevering in the face of not knowing exactly how to proceed or whether a particular approach will work. Exploring, investigating one or multiple approaches, and articulating a chain of reasoning behind the approaches also characterize productive struggle.

Within the Pathway materials the idea of explicit connections refers to the linkages or relationships among and between mathematical and/or statistical facts, procedures, and concepts. Explicit connections generally reference math ideas and concepts and may be about context as well. Connections may be drawn by students or faculty, but most often are presented and reinforced by faculty.

Deliberate practice consists of a set of tasks for students that are created to overcome gaps in understanding, apply what has been learned, and/or deepen fluency with key concepts.”

To learn much more about these learning opportunities, read Kay’s essay, “Learning Opportunities for Pathway Classrooms.”

_______________________________________________
Ed. note: This post has been revised from the original.

The original post did not include the fifth and sixth components of the pathways. These components — advancing teaching and rapid informative analytics — will be discussed further in future posts.

Lesson Study Revisited

You might find this article about lesson study from The Hechinger Report interesting. Carnegie is using lesson study, not exactly in the way outlined in this article, but to improve our mathematics pathways. Statway and Quantway faculty teams at each community college site will be organized into lesson study groups. They will work together to plan instruction, observe each other teaching, and identify the most difficult and high-priority obstacles that stand in the way of success of the pathways. The pathways curriculum materials are built upon three “learning opportunities” in the instructional design of the pathways — productive struggle, explicit connections to concepts, and deliberate practice. These learning opportunities are interconnected and not necessarily exclusive of each other and are designed to support a deep and more meaningful understanding of mathematical and statistical concepts. Lessons study provides a framework for rigorously and carefully improving the pathway in its entirety from curriculum development to implementation in the classroom.

“LESSON STUDY,’ JAPANESE STRATEGY FOR IMPROVING TEACHERS, CATCHING ON IN U.S.
Lesson study is a professional development strategy used extensively in Japan that essentially dissects a teacher’s lesson and the way it’s delivered. Here’s how it works: teachers come up with a detailed lesson plan and explain ahead of time to colleagues the goals of the lesson. Then, one teacher tries the lesson out on a group of students, while dozens of other teachers watch what happens. Finally, the observers offer feedback and ideas for improvement. “[We’ve been] doing lesson study more than 100 years in Japan,” says Toshiakira Fujii, a premier professor of math education in Japan who was among those teachers observing at Jorge Prieto Elementary on Chicago’s’ northwest side. “But lesson study in the United States is quite new.” “Traditional American professional development is somebody outside comes and then does for teachers,” says Takahashi. But he argues there is a lot that teachers can do on their own. “My goal is in every school teachers gather and then find a new way to improve lessons by themselves.”
Read article >

Learning From Practice to Improve Practice

In a keynote address at the annual meeting of the Association of Community College Trustees in Dallas recently, Carnegie President Tony Bryk outlined for the Trustees how Carnegie is using improvement research in our work to improve the success rate of students in developmental math.

“We need to rethink how we innovate,” he said. “As educators, we are pragmatists. We see problems and we want to move quickly to solutions. But we also know from past experience that many solutions are rarely tested against evidence and we rarely rely on evidence to continuously improve them. We tend to put new programs in place and then move on.”

Bryk explained that Carnegie is challenging this way of working. He said that we know from 50 years of history at educational innovation that few things actually work as originally designed. That failures may occur is not the problem; that we fail to learn from them is.

In response, Carnegie embraces a quality improvement orientation, encouraging rapid cycles of change. If something doesn’t work, the process is to change it until we find something that does. “This entails a mind shift from seeing change as principally about managing the dynamics of large scale roll out toward seeing change as opportunities to learn to improve,” he said. “This learning from practice to improve is the surest mechanism for success.”

Resources

• Read the speech
• Download the presentation slides
• Listen to the speech:
  

There Is More to College Success Than Test Scores and Lesson Plans: Carnegie’s Focus on “Productive Persistence”

Why do some college students persist while others don’t? David Levin asked this question about the alumni of the Knowledge is Power Program (KIPP)—a network of urban charter schools.  Instead of focusing on students’ test scores or college professors’ instructional styles, he found something different. Students who were successful “were the ones who were able to recover from a bad grade and resolve to better next time. …” That is, they weren’t just the ones who were “naturally smart.” Levin, who is co-founder of KIPP, and his ideas were the focus of a recent New York Times Sunday Magazine article, “The Character Test: Why our kids’ success—and happiness—may depend less on perfect performance than on learning how to deal with failure.” Optimism, persistence and social intelligence, according to the article, are the habits and mindsets that “seemed indispensable” to enable students who were from families without a lot of family resources to graduate from college. These characteristics help students navigate the transition from high school to college by helping them to bounce back from the new challenges that inevitably come up in college.

What are the characteristics that help students succeed and how can we promote them? Many psychologists have contributed to this question. Carnegie uses the term “productive persistence” to summarize these good ideas and to apply them to our initiative to improve student success in developmental mathematics in community colleges.

We do this because the work of social and developmental psychologists has convinced us that improved curricula and instruction are not sufficient to dramatically improve college-level math completion rates. Many students work hard in developmental math classes—studying long hours, nights and weekends—yet do so using ineffective strategies. Other students simply withdraw effort soon after the course begins. To help more students successfully complete the Carnegie initiated math pathways, we want them to both persist in their studying and attendance (tenacity) and to do so efficiently and effectively (good strategies). Productive persistence is core to our work.

We recognize that in addition to pedagogical improvements, student motivation, engagement and skills must also be attended to. Students placed in developmental math often come to the classroom with what our colleague from UCLA, Carnegie Senior Fellow Jim Stigler, calls “math scar tissue”—or the residue of years of failure in mathematics courses.  Such past failures can ossify and produce in students a belief that their mathematical ability is fixed and not improvable, thereby undermining their motivation following setbacks or failures. Students may also have crippling math anxiety that results in worsening cycles of procrastination and low performance, or they may lack basic study skills for successfully completing their work.  Moreover, because many community colleges are heterogeneous commuter campuses, students may feel only a weak sense of connection to their peers, to faculty, or to their institution. In the absence of these ties, motivation to attend and persist may suffer.

We believe that these mindsets, skills and social relationships can prevent student success. We also believe that when efforts are made to improve them, then the impact of otherwise effective curricula can be made manifest. In effect, we aim to do what our Senior Fellow Uri Triesman calls “reclaiming students’ mathematical lives.” We are committed to understanding the social and psychological drivers of developmental math student success and using our improvement processes to design and deliver effective interventions to address them.

In a recent Gates-funded white paper on “Academic Tenacity” by Carol Dweck, Geoffrey Cohen, and Gregory Walton of Stanford University, successful motivation is described as:

Students are engaged in learning, view effort positively, and are able and willing to forego immediate pleasures for the sake of schoolwork. For example, they seek challenging tasks that will help them learn new things, rather than tasks well within their comfort zone where they do not have to work hard or risk failure. Next, difficulty (confusion, setbacks, failures) does not derail them. They see a setback as an opportunity for learning or a problem to be solved rather than as a humiliation, a condemnation of their ability, a symbol of future failures, or a confirmation of their identity as a non-student. This is true at the level of a given task and at the level of their studies in general–they know how to remain engaged over the long haul and how to deploy new strategies for moving forward effectively.

Keeping these factors in mind, Carnegie’s new math pathways include specific activities, support systems, and pedagogical approaches designed to encourage and build these skills and mindsets in order to increase a student’s self-efficacy, motivation, persistence, and ability to navigate college.

A Brief History of the Quantitative Literacy Movement

Background Information for Faculty Arithmetic and Algebra Skills Aren’t Enough Any More!

“Despite its occasional use as a euphemism for statistics in school curricula, quantitative literacy is not the same as statistics. Neither is it the same as mathematics, nor is it (as some fear) watered-down mathematics. Quantitative literacy is more a habit of mind, an approach to problems that employs and enhances both statistics and mathematics. Unlike statistics, which is primarily about uncertainty, numeracy is often about the logic of certainty. Unlike mathematics, which is primarily about a Platonic realm of abstract structures, numeracy is often anchored in data derived from and attached to the empirical world. Surprisingly to some,this inextricable link to reality makes quantitative reasoning every bit as challenging and rigorous as mathematical reasoning. (Indeed, evidencefrom Advanced Placement examinations suggests that students of comparable ability find data-based statistical reasoning more difficult thansymbol-based mathematical reasoning.)”

– The Case for Quantitative Literacy, in Mathematics and Democracy
It has always been important for individuals to have the capacity to do arithmetic and algebra, however, in today’s global and technological society, doing calculations is not enough. An individual’s capacity to identify and understand quantitative situations, reason quantitatively, and communicate about the role mathematics plays in the world is essential. This quantitative literacy goes beyond basic computational skills. The quantitatively literate individual should be able engage in mathematics and solve quantitative problems from a wide array of authentic contexts and everyday life situations. These “habits of the mind” lead to making well-founded mathematical judgments that are useful in an individual’s current and future life as a constructive, concerned, and reflective citizen. Quantitative Literacy (QL) is more than just arithmetic skills and as fundamental as language literacy.

Conversations within the mathematics community about the importance and content of quantitative literacy have occurred for at least two decades. The following is a partial list of the many activities and documents that have brought the conversations and courses to college campuses:

In 1996, the Quantitative Literacy Subcommittee of the Committee on the Undergraduate Program in Mathematics issued guidelines for quantitative literacy programs (Sons, 1996). An excellent description of quantitative literacy and a summary of the CUPM recommendations appeared in Assessment Practices in Undergraduate Mathematics (Sons, 1999). These reports argue that a college graduate should be able to:

• interpret mathematical models such as formulas, graphs, tables, and schematics, and draw inferences from them;
• represent mathematical information symbolically, visually, numerically, and verbally;
• use arithmetical, algebraic, geometric and statistical methods to solve problems;
• estimate and check answers to mathematical problems in order to determine reasonableness, identify alternatives, and select optimal results;
• recognize that mathematical and statistical methods have limits.

In 1997, quantitative literacy was defined as “five different dimensions of numeracy: practical, for immediate use in the routine tasks of life; civic, to understand major public policy issues; professional, to provide skills necessary for employment; recreational, to appreciate and understand games, sports, and lotteries; and cultural, as part of the tapestry of civilization.”(Steen, L.A., (1997). Why Numbers Count: Quantitative Literacy for Tomorrow’s America. New York, New York. The College Entrance Examination Board, p. xxii.)

In 2000, The National Numeracy Network (NNN) began with a vision of a society in which all citizens possess the power and habit of mind to search out quantitative information, critique it, reflect upon it, and apply it in their public, personal and professional lives. The National Numeracy Network has continued to promote education that integrates quantitative skills across all disciplines and at all levels. The NNN website (http://serc.carleton.edu/nnn/about) presents three names for consideration. Numeracy, an expression first used in the UK’s 1959 “Crowther Report” to include secondary school students’ ability to reason and solve sophisticated quantitative problems, their basic understanding of the scientific method, and their ability to communicate at a substantial level about quantitative issues in everyday life. Others call it Quantitative Literacy (QL), and describe this comfort, competency, and “habit of mind” in working with numerical data as being as important in today’s highly quantitative society as reading and writing were in previous generations. Still others refer to it as Quantitative Reasoning (QR), emphasizing the higher-order reasoning and critical thinking skills needed to understand and to create sophisticated arguments supported by quantitative data.

In 2006, the American Mathematical Association of Two-Year Colleges (AMATYC) highlighted the importance of quantitative literacy in its standards document, Beyond Crossroads with the recommendation that faculty integrate quantitative literacy outcomes into all mathematics courses and collaborate with faculty in other disciplines to integrate quantitative literacy into coursework across all disciplines. In addition, this document presented these QL outcomes for students in all college programs:

• exhibit perseverance, ability, and confidence to use mathematics to solve problems
• perform mental arithmetic and use proportional reasoning
• estimate and check answers to problems and determine the reasonableness of results
• use geometric concepts and representations in solving problems
• collect, organize, analyze data, and interpret various representations of data, including graphs and tables
• use a variety of problem-solving strategies and exhibit logical thinking
• use basic descriptive statistics
• utilize linear, exponential, and other nonlinear models as appropriate
• communicate findings both in writing and orally using appropriate mathematical language and symbolism with supporting data and graphs
• work effectively with others to solve problems
• demonstrate an understanding and an appreciation of the positive role of mathematics in their lives.

The above list is not comprehensive and represents only handful of important documents. Here is a list of other important publications about QL:

• Mathematics and Democracy: The Case for Quantitative Literacy, prepared by The National Council on Education and the Disciplines, Editor: Lynn Steen, 2001
• “Why Numeracy Matters?” formal papers of the National Forum on QL, December, 2001 (published proceedings of the National Forum: Achieving QL: Urgent Challenge for Higher Education (2004))
• Steen summary of the National Forum on QL, December, 2001 (MAA Focus)
• Calculation vs. Context: Quantitative Literacy and Its Implications for Teacher Education, Editor: Bernard Madison and Lynn Arthur Steen, published and distributed by the Mathematical Association of America, (2008)
• QL Survey Slides, MAA/JMM, 2010 and MAA Sigma on QL, Update on Activities October 2010 (87% FYC have college-wide QL req)
• Quantitative Reasoning for the Contemporary World (QRCW) course outlines from Central Washington U (Boersma); U of Arkansas (Madison); Hollings U (Diefenderfer)
• QRCW Project Summary and QRCW Grading Rubric (draft)
• QL Rubric from the Association of American Colleges and Universities (2009)
• Two articles by Madison and Dingman published in the National Numeracy Network (NNN) ejournal:
  • Quantitative Reasoning in the Contemporary World, 1: The Course and Its Challenges, Vol 3, Issue 2, Article 4
  • Quantitative Reasoning in the Contemporary World, 2: The Course and Its Challenges, Vol 3, Issue 2, Article 5
  • Third article is under development
• “Numeracy” article in the National Numeracy Network (NNN) ejournal, 2008
• P21 (Partnership for 21st Century Skills) Framework/Definitions
• OCED Education at a Glance 2010

What Community College Developmental Mathematics Students Understand about Mathematics

Carnegie Senior Partner Jim Stigler and colleagues Karen B. Givvin and Belinda J. Thompson from the University of California, Los Angeles were commissioned by Carnegie to investigate what community college developmental mathematics students understand about mathematics and how we might help turn around the alarming statistics that show that an enormous number of those students drop out of college because they aren’t successful in math courses. “An assumption we make in this report is that substantive improvements in mathematics learning will not occur unless we can succeed in transforming the way mathematics is taught,” the authors write. “In particular, we are interested in exploring the hypothesis that these students who have failed to learn mathematics in a deep and lasting way up to this point might be able to do so if we can convince them, first, that mathematics makes sense, and then provide them with the tools and opportunities to think and reason.”

Download report (PDF) »

See other reports in the series, Problem Solution Exploration Papers »

College Enrollment Hits All-Time High, Fueled by Community Colleges

18- to 24-year-olds attending college in the United States hit an all-time high in October 2008, driven by a recession-era surge in enrollments at community colleges, according to a Pew Research Center analysis of newly released data from the U.S. Census Bureau. Just under 11.5 million students, or 39.6% of all young adults ages 18 to 24, were enrolled in either a two- or four-year college in October 2008 (the most recent date for which comprehensive nationwide data are available). Both figures — the absolute number as well as the share — are at their highest level ever. Enrollments have been rising over many decades at both two- and four-year colleges, but the most recent annual spike has taken place entirely at two-year colleges. In October 2007, some 3.1 million young adults, or 10.9% of all 18- to 24-year-olds, were enrolled in a community college. A year later, that figure had risen to 3.4 million students, or 11.8% of all 18- to 24-year-olds. By contrast, enrollments at four-year colleges were essentially flat from 2007 to 2008.

Read more >>

Transfer from Two-year to Four-year College Complicated

President Obama wasn’t the first to recognize the importance of community colleges to the country’s education goals, but it seems lots of others are focusing on them as well. This month’s “Progress of Education Reform” report from the Education Commission of the States delves into a big issue for this sector education, transfer and articulation. Articles address research on the effectiveness of transfer and articulation policies, factors that negatively impact successful transfer from a two –year to four-year institution, and strategies that smooth the transition for community college students into bachelor’s degree programs. As the author’s note, “these studies illustrate that the transfer process is more complicated than simply creating articulation agreements…the seeds are planted in high school with students taking a rigorous college preparatory curriculum.” One of the ECS policy recommendations is especially relevant to Carnegie’s work in developmental math in community colleges: “a focus on student academic competencies rather than courses can better prepare students for transfer.”

More information on these research papers can be found at: http://www.ecs.org/

Addressing Student Success in Developmental Mathematics

During a July convening at Carnegie with representatives from several organizations working on increasing success for community college students in developmental mathematics, participants came up with some tensions that need to be acknowledged and/or addressed.

Those include:

• Increased pressure to increase graduation and completion rates and the desire to get students to learn more mathematics do not necessarily lead to the same actions.
• Instead of “tracking” students at community colleges (something done to students) can we instead provide options and ask students to decide what they need (as they do in choosing a major).
• We acknowledge that the low success rate in developmental math is not necessarily the fault of faculty or students or high schools, but instead is because developmental math needs to be done differently.