[prev in list] [next in list] [prev in thread] [next in thread]
List: lon-capa-cvs
Subject: [LON-CAPA-cvs] cvs: modules /gerd/correlpaper correlations.tex
From: www <lon-capa-cvs () mail ! lon-capa ! org>
Date: 2006-09-30 23:45:48
Message-ID: cvswww1159659948 () cvsserver
[Download RAW message or body]
This is a MIME encoded message
www Sat Sep 30 19:45:48 2006 EDT
Modified files:
/modules/gerd/correlpaper correlations.tex
Log:
Corrections.
["www-20060930194548.txt" (text/plain)]
Index: modules/gerd/correlpaper/correlations.tex
diff -u modules/gerd/correlpaper/correlations.tex:1.14 \
modules/gerd/correlpaper/correlations.tex:1.15
--- modules/gerd/correlpaper/correlations.tex:1.14 Sat Sep 30 16:35:10 2006
+++ modules/gerd/correlpaper/correlations.tex Sat Sep 30 19:45:46 2006
@@ -51,11 +51,11 @@
% but any date may be explicitly specified
\begin{abstract}
-An important result of Physics Education Research is that students' learning and \
success in a course is correlated with their beliefs, attitudes, and expectations \
regarding physics. However, it is hard to assess these beliefs, and traditional \
survey instruments such as the Maryland Physics Expectations Survey (MPEX) are \
intended to evaluate the impact of one or more semesters of instruction on an overall \
class and improve teaching. +An important result of Physics Education Research is \
that students' learning and success in a course is correlated with their beliefs, \
attitudes, and expectations regarding physics. However, it is hard to assess these \
beliefs for individual students, and traditional survey instruments such as the \
Maryland Physics Expectations Survey (MPEX) are intended to evaluate the impact of \
one or more semesters of instruction on an overall class and improve teaching.
In this study, we investigate the possibility of using the analysis of online \
student discussion behavior as an indicator of an individual student's approach to \
physics. These discussions are not tainted by the effects of self-reporting, and are \
gathered in authentic non-research settings, where students attempt to solve problems \
in the way that they belief is most efficient and appropriate.
-We investigate correlations with a traditional instrument, namely the MPEX, as well \
as correlations with the Force Concept Inventory (FCI), the final exam grade, and the \
overall course performance as a measure of the student's learning. To gauge the \
outcomes, we also investigate correlations between these measures.\end{abstract} +We \
calculate the correlation of both MPEX and student discussions with different \
measures of student learning, and find that on an individual base, student \
discussions are a stronger predictor of success than MPEX outcomes.\end{abstract}
\pacs{01.40.Fk}% PACS, the Physics and Astronomy
% Classification Scheme.
@@ -67,7 +67,7 @@
The MPEX makes the limitations of this approach very explicit in their ``Product \
Warning Label''~\cite{mpexwarning}: ``students often think that they function in one \
fashion and actually behave differently. For the diagnosis of the difficulties of \
individual students more detailed observation is required.'' Online student \
discussions associated with online physics problems are different in that they are \
generated within the real context of the course, and students have a vested interest \
in making these discussions as productive as possible, given their understanding of \
how physics is done and their approach to it. They could thus be a ``reality check'' \
of students' beliefs, attitudes, and expectations.
-The MPEX ``Product Warning Label'' continues, that ``this survey is primarily \
intended to evaluate the impact of one or more semesters of instruction on an overall \
class''~\cite{mpexwarning}, and recommends using the outcomes, in combination with \
evaluations of student learning of content, as a means to improve overall course \
instruction. In this paper, we are asking the question if the evaluation of student \
online discussion behavior can be used as a means to get to assess the attitudes and \
beliefs of an individual student, and if in turn, these can be used to predict the \
success of an individual student in the learning of physics content. An obvious \
application would be the early detection of students at risk. +The MPEX ``Product \
Warning Label'' continues, that ``this survey is primarily intended to evaluate the \
impact of one or more semesters of instruction on an overall \
class''~\cite{mpexwarning}, and recommends using the outcomes, in combination with \
evaluations of student learning of content, as a means to improve overall course \
instruction. In this paper, we are asking the question if the evaluation of student \
online discussion behavior can be used as a means to assess the attitudes and beliefs \
of an individual student, and if in turn, these can be used to predict the success of \
an individual student in the learning of physics content. An obvious application \
would be the early detection of students at risk.
In particular:
\begin{itemize}
@@ -79,7 +79,7 @@
\section{\label{background}Background}
Previous studies indicate that correlations between epistemological beliefs and \
academic performance exist, both directly and indirectly \cite{schommer93,may02}. The \
problem is how to measure these beliefs, and techniques include surveys, guided \
interviews, and observations.
-Many of these, though, take place in artificial research settings and outside the \
normal course activity over a relatively short time, and research results regarding \
their predictive power are not conclusive: Coletta and Philips~\cite{coletta05} found \
a strong correlation between the FCI Gain and the MPEX Score, while \
Dancy~\cite{dancy02} found low correlations between the MPEX and the the performance \
on homework, tests, and final exams. The discrepancies might all be traced back to \
the ``Product Warning Lab''~\cite{mpexwarning}, that the survey is best used to gain \
insights into the beliefs of the class as a whole, rather than on an individual \
level. +Research results regarding their predictive power of these instruments is not \
always conclusive: for example, Coletta and Philips~\cite{coletta05} found a strong \
correlation between the MPEX and FCI Gain, while Dancy~\cite{dancy02} found low \
correlations between the MPEX and the the performance on homework, tests, and final \
exams. The discrepancies might all be traced back to the ``Product Warning \
Lab''~\cite{mpexwarning}, that the survey is best used to gain insights into the \
beliefs of the class as a whole, rather than on an individual level.
Online discussions take place within the regular course context and over its \
complete duration. They are a rich source of feedback to the \
instructor~\cite{kortemeyer05feedback}, and their quality and character was found to \
be correlated with the type and difficulty of the associated \
problems~\cite{kortemeyer05ana}, i.e., data exists regarding the influence of {\it \
problem} characteristics on associated discussions. Unfortunately, less data exists \
on the correlation between {\it student} characteristics and discussion behavior, \
because usually only very few student characteristics are known, with the exception \
of the students' overall performance in the course. Thus, one of the few findings was \
the fact that certain discussion behavior, most prominently exhibited on \
``non-sanctioned'' discussion sites external to the course, is negatively correlated \
with performance in the course~\cite{kashy03,kortemeyer05ana}.
@@ -205,7 +205,8 @@
As already found in Ref.~\cite{kortemeyer05ana}, most students are quite prolific in \
their online discussions, but a few students only made a small number of \
contributions, leading to small statistics on their actual discussion behavior. For \
each of the discussion correlations, we thus also carried out a second calculation \
limited to students who contributed at least five entries over the course of the \
semester.
\subsection{\label{mpex}The MPEX}
-We deployed the Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the \
beginning and the end of the mechanics semester. Participation was voluntary. We \
calculated the score in comparison to the ``favorable" expert responses given in \
Ref.~\cite{mpex} on the final (post) deployment, as well as, for students who \
participated both times, the gain. The same analysis was done for each cluster of the \
MPEX (example statements are given, including the expert answer): +We deployed the \
Maryland Physics Expectations Survey (MPEX)\cite{mpex} both at the beginning and the \
end of the mechanics semester. Participation was voluntary. We calculated the \
``score'' in comparison to the ``favorable" expert responses given in \
Ref.~\cite{mpex} -- please note that the +word ``score'' in the context of the MPEX \
is thus not an absolute measure of correctness, but of agreement with the majority of \
an expert group, who does not even necessarily agree among each other. We calculated \
the final (post) deployment score, as well as, for students who participated both \
times, the gain. The same analysis was done for each cluster of the MPEX (example \
statements are given, including the expert answer): \begin{itemize}
\item{\it Independence:} student takes responsibility for constructing their own \
understanding, rather than takes what is given by authorities (teacher, materials) \
without evaluation \begin{quote}
@@ -232,7 +233,7 @@
Favorable: I go over my class notes carefully to prepare for tests in this course.
\end{quote}
\end{itemize}
-The overall scores of the students on the MPEX clusters were low (Independence 42\%; \
Coherence 46\%; Concepts 48\%; Reality Link 55\%; Math Link 40\%; Effort 47\%). +The \
overall scores (i.e., agreement with the expert group) of the students on the MPEX \
clusters were low (Independence 42\%; Coherence 46\%; Concepts 48\%; Reality Link \
55\%; Math Link 40\%; Effort 47\%). \subsection{\label{performance}Measures of \
Student Learning} As a measure of student conceptual understanding and learning, we \
deployed the revised Force Concept Inventory (FCI)\cite{fci} at the beginning and the \
end of the course, again with voluntary participation. As an additional measure of \
student performance, the performance on the final exam and the course grade for each \
student were taken into consideration. For the grade we used the raw percentage \
score, not the number grades, since it provides finer grained information about the \
overall student performance in the course. \section{\label{perception}Student \
Perception of the Online Discussions and Survey Instruments} @@ -251,41 +252,41 @@
\section{\label{results}Correlation Results}
\subsection{\label{MPEXDiscussion}Correlations between Discussion Behavior and MPEX}
-To directly compare the attitudes and beliefs measures, we calculated correlations \
between the prominence of discussion behavior classes and the MPEX clusters, and \
generally found them to be very low. As an example, the correlation between the score \
on the Concepts Cluster and the prominence of conceptual discussion contributions \
turned out to be $R=0.14 [-0.08 \to 0.34]; n=84$ when considering all students, and \
$R=0.15 [-0.13 \to 0.41]; n=51$ when only considering those who made at least five \
discussion contributions -- the 95\% confidence intervals (given in square brackets) \
include zero. Thus, we conclude that discussion behavior and the individual MPEX \
cluster scores are -- if at all -- only weakly correlated. +To directly compare the \
attitudes and beliefs measures, we calculated correlations between the prominence of \
discussion behavior classes and the MPEX clusters, and generally found them to be \
very low. As an example, the correlation between the score on the Concepts Cluster \
and the prominence of conceptual discussion contributions turned out to be $R=0.14\ \
[-0.08\to0.34] (n=84)$ when considering all students, and $R=0.15\ [-0.13\to0.41] \
(n=51)$ when only considering those who made at least five discussion contributions \
-- the 95\% confidence intervals (given in square brackets) include zero. Thus, we \
conclude that discussion behavior and the individual MPEX cluster scores are -- if at \
all -- only weakly correlated.
\subsection{\label{learningcorreldis}Correlations between Discussions and Learning}
Figure~\ref{physicsgrade} shows the correlation between the prominence of \
physics-related discussions and the course grade percentage (for better statistics, \
only students who contributed at least five discussion entries over the course of the \
semester were considered). \begin{figure}
\includegraphics[width=9cm]{physicsgrade}
-\caption{\label{physicsgrade}Correlation of percentage physics-related discussions \
with grade percentage ($R=0.33 [0.15 \to 0.49]$; $n=111$).} \
+\caption{\label{physicsgrade}Correlation of percentage physics-related discussions \
with grade percentage ($R=0.33\ [0.15\to0.49] (n=111)$).} \end{figure}
-Figure~\ref{fciphysics} shows how the percentage of a particular student's \
discussion contribution that was classified as "physics-related" correlates with \
their final FCI score ($R=0.34 [0.15 \to 0.51]$; $n= 95$). As already in \
Fig.~\ref{physicsgrade}, an additional analysis was carried out that was limited to \
students for which better statistics were available, which let to a stronger \
correlation ($R=0.51[0.29 \to 0.68]$; $n=57$). While physics-related discussions \
positively correlate with FCI scores and grades (Fig.~\ref{physicsgrade}), \
solution-oriented discussions negatively correlate (Fig.~\ref{solutionfci}; $R=-0.58 \
[-0.73 \to -0.38]$; $n=57$). +Figure~\ref{fciphysics} shows how the percentage of a \
particular student's discussion contribution that was classified as "physics-related" \
correlates with their final FCI score ($R=0.51\ [0.29\to0.68] (n=57)$). While \
physics-related discussions positively correlate with FCI scores and grades \
(Fig.~\ref{physicsgrade}), solution-oriented discussions negatively correlate \
(Fig.~\ref{solutionfci}; $R=-0.58\ [-0.73\to-0.38] (n=57)$).
-\begin{figure*}
-\includegraphics[width=9cm]{fcipostphysics}\includegraphics[width=9cm]{fcipostphysicsT}
-\caption{\label{fciphysics}Correlation between the FCI score and the percentage of \
that student's discussion that was classified as "physics" ($R=0.34 [0.15 \to 0.51]$; \
$n= 95$). The figure on the right only includes students who contributed more than \
five discussion entries over the course of the semester ($R=0.51 [0.29 \to 0.68]$; \
$n=57$).}
-\end{figure*}
+\begin{figure}
+\includegraphics[width=9cm]{fcipostphysicsT}
+\caption{\label{fciphysics}Correlation of percentage physics-related discussions \
with final FCI score ($R=0.51\ [0.29\to0.68] (n=57)$).} +\end{figure}
\begin{figure}
\includegraphics[width=9cm]{fcipostsolutionT}
-\caption{\label{solutionfci}Correlation of percentage solution-oriented discussions \
with final FCI score ($R=-0.58 [-0.73 \to -0.38]$; $n=57$).} \
+\caption{\label{solutionfci}Correlation of percentage solution-oriented discussions \
with final FCI score ($R=-0.58\ [-0.73\to-0.38] (n=57)$).} \end{figure}
\subsection{\label{learningcorrelmpex}Correlations between MPEX and Learning}
-Correlations between the MPEX and measures of student learning are generally weak. \
Considering final exam, FCI, and course grade, $R=0.36 [0.17 \to 0.52]$ ($n=97$) \
between the score on the Coherence cluster and the course grade percentage is the \
highest correlation found. +Correlations between the MPEX and measures of student \
learning are generally weak. Considering final exam, FCI, and course grade, $R=0.36\ \
[0.17\to0.52] (n=97)$ between the score on the Coherence cluster and the course grade \
percentage is the highest correlation found.
Dancy~\cite{dancy02} found similarly low correlations with the performance on \
homework, tests, and final exams: direct comparison with the performance on the final \
exams found $R=0.37$ for the correlation with the total MPEX score ($R=0.27$ here), \
$R=0.39$ with the Independence Cluster ($R=0.25$ here), $R=0.24$ with the Coherence \
Cluster ($R=0.36$ here), $R=0.29$ with the Concept Cluster ($R=0.25$ here), $R=-.02$ \
with the Reality Link cluster ($R=0.1$ here), $R=0.3$ with the Math Link cluster (no \
significant correlation found here), and no significant correlation with the Effort \
Cluster ($R=0.1$ here).
-Figure~\ref{mpexfci} shows how the final MPEX and FCI scores correlated with each \
other, i.e, $R=0.24 [0.04 \to 0.42]$ ($n=97$).
-Coletta and Philips~\cite{coletta05} found a strong correlation between the FCI Gain \
and the MPEX Score ($R=0.52 [0.24 \to 0.72]; n=37$), while the same correlation \
turned out much lower in this study ($R=0.17 [-0.05 \to 0.37]; n=84$ here). The \
correlations reported here are in the same range that +Figure~\ref{mpexfci} shows how \
the final MPEX and FCI scores correlated with each other, i.e, $R=0.24\ [0.04\to0.42] \
(n=97)$. +Coletta and Philips~\cite{coletta05} found a strong correlation between \
the FCI Gain and the MPEX Score ($R=0.52\ [0.24\to0.72] (n=37)$), while the same \
correlation turned out much lower in this study ($R=0.17\ [-0.05\to0.37] (n=84)$ \
here). The correlations reported here are in the same range that Perkins et \
al.~\cite{perkins04} found when investigating the influence of beliefs on conceptual \
learning, using the CLASS~\cite{adams04} and the Force and Motion Conceptual \
Evaluation (FMCE)~\cite{thornton98} instruments. \begin{figure}
\includegraphics[width=9cm]{fcipostmpexpost}
-\caption{\label{mpexfci}Correlation of the final FCI score with the MPEX score \
($R=0.24 [0.04 \to 0.42]$; $n=97$).} +\caption{\label{mpexfci}Correlation of the \
final FCI score with the MPEX score ($R=0.24 [0.04 \to 0.42] (n=97)$.} \end{figure}
@@ -300,15 +301,15 @@
\item FCI gain versus gain in prominence of solution-oriented and physics-related \
postings \end{itemize}
-As it turns out, the first correlations are significant, with $R=-0.44 [-0.65 \to \
-0.18] (n=47)$ for FCI gain versus solution-oriented discussions, and $R=0.4 [0.13 \
\to 0.62] (n=47)$ for FCI gain versus physics-related discussions. Such significant \
correlations do not occur for FCI gain versus any of the MPEX cluster scores. +As it \
turns out, the first correlations are significant, with $R=-0.44\ [-0.65\to-0.18] \
(n=47)$ for FCI gain versus solution-oriented discussions, and $R=0.4\ [0.13\to0.62] \
(n=47)$ for FCI gain versus physics-related discussions. Such significant \
correlations do not occur for FCI gain versus any of the MPEX cluster scores.
-On the other hand, the correlations with discussion-gain are not significant: $0.24 \
[-0.05 \to 0.49] (n=47)$ for FCI gain versus gain in solution-oriented discussions, \
and $-0.12 [-0.39 \to 0.17] (n=47)$ for FCI gain versus gain in physics-related \
discussions. Note that these correlations have the opposite sign than expected, \
however, the confidence intervals include zero in both cases. When looking at the \
absolute values, the average gain in solution-oriented discussions between the two \
halves of the semester is $2.4\%$, and the gain in physics-oriented discussions \
$-0.3\%$ --- in other words, the students did not really change their discussion \
behavior over the course of the semester, and their discussion behavior does not \
improve co-measured with their increasing understanding of physics. +On the other \
hand, the correlations with discussion-gain are not significant: $0.24\ \
[-0.05\to0.49] (n=47)$ for FCI gain versus gain in solution-oriented discussions, and \
$-0.12\ [-0.39\to0.17] (n=47)$ for FCI gain versus gain in physics-related \
discussions. Note that these correlations have the opposite sign than expected, \
however, the confidence intervals include zero in both cases. When looking at the \
absolute values, the average gain in solution-oriented discussions between the two \
halves of the semester is $2.4\%$, and the gain in physics-oriented discussions \
$-0.3\%$ --- in other words, the students did not really change their discussion \
behavior over the course of the semester, and their discussion behavior does not \
improve co-measured with their increasing understanding of physics.
Thus, the discussion behavior appears to be a property of the students that is \
almost constant over the course of the semester, just like Hammer~\cite{hammer94} \
already pointed out that it is unlikely that epistemological beliefs are changed \
implicitly by physics instruction.
-We also ran a linear regression analysis of the FCI scores versus discussion \
behavior. In the equations below, ``PostFCI'' is the predicted post (final) FCI \
score, ``PreFCI'' is the score on the pre FCI, and ``Solution'' and ``Physics'' are \
the percentage solution- and physics-oriented discussion over the course of the \
semester. For the physics-oriented discussion, we found +We also ran a linear \
regression analysis of the FCI scores versus discussion behavior. In the equations \
below, ``PostFCI'' is the predicted post (final) FCI score in points, ``PreFCI'' is \
the score on the pre-test FCI in points, and ``Solution'' and ``Physics'' are the \
percentage solution-oriented and physics-related discussion over the course of the \
semester. For the physics-oriented discussion, we found \begin{equation*}
-\mbox{Post FCI}=5.486+0.922\cdot\mbox{PreFCI}+0.24\cdot\mbox{Physics}
+\mbox{PostFCI}=5.486+0.922\cdot\mbox{PreFCI}+0.24\cdot\mbox{Physics}
\end{equation*}
with an explained variance of 45.6\% of the Post FCI score. The effect of the \
pre-test FCI is significant ($p<0.001$), the effect of the physics discussion is not \
($p=0.195$).
@@ -316,7 +317,7 @@
\begin{equation*}
\mbox{PostFCI}=7.606+0.857\cdot\mbox{PreFCI}+(-0.042)\cdot\mbox{Solution}
\end{equation*}
-with an explained variance of 47.9\% of the Post FCI score. Both coefficients are \
significant, the solution-oriented discussion has $p=0.019$. Thus, controlling for \
Pre FCI score, for each 10 percent increase in solution-oriented discussion, the \
predicted Post FCI score goes down by 0.42 points. +with an explained variance of \
47.9\% of the Post FCI score. Both coefficients are significant, the \
solution-oriented discussion has $p=0.019$. Thus, controlling for pre-test FCI score, \
for each 10 percent increase in solution-oriented discussion, the predicted post-test \
FCI score goes down by 0.42 points. Students who do not make any solution-oriented \
contributions would on the average gain 7.6 points on the 30 item FCI due to \
instruction, while at the other extreme, students who only make solution-oriented \
discussions would on the average only gain 3.4 points -- less than half. \
\section{Conclusions} Online student discussions have very little correlation with \
MPEX outcomes, but appear to be a good reflection of students' individual beliefs \
regarding the nature of problem solving in physics. Students who exhibit more \
expert-like views and strategies have higher learning success, even when controlling \
for prior physics knowledge. \begin{acknowledgments}
_______________________________________________
LON-CAPA-cvs mailing list
LON-CAPA-cvs@mail.lon-capa.org
http://mail.lon-capa.org/mailman/listinfo/lon-capa-cvs
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic