Abstract
1. Introduction
1.1 Collaborative Learning
2. Experiences With ALN's at NJIT
2.1. Virtual Classroom + Video: The Degree Programs
2.2. Access: Overcoming Logistical Problems
3. Hypotheses and Findings on Student Impacts: Interim Results
3.1. Problems reported by Students
3.2. Subjective Assessments: Relative Frequency Distributions for Different Modes
3.3. Outcomes: Grades and Completion Rates
4. Impacts on Faculty
5. Looking to the Future
Acknowledgments
References
New Jersey Institute of Technology has been delivering college
courses via an Asynchronous Learning Network (ALN) system called
the Virtual Classroom[TM] for a decade, using
various media mixes. Currently, two complete undergraduate degree
programs are available via a mix of video plus Virtual Classroom,
the B.A. in Information Systems and the B.S. in Computer Science.
This paper presents preliminary findings about impacts on students,
and touches on some issues and potential impacts on faculty, individual
universities, and the structure of higher education. Overall ratings
of courses by students who complete ALN based courses are equal
or superior to those for traditional courses. Dropout or Incomplete
outcomes are somewhat more prevalent, while grade distributions
for those who complete tend to be similar to those for traditional
courses. For both students and faculty, more startup time devoted
to solving the "logistics" of ALN delivery seems to
be required at the beginning of courses. ALN delivery is not just
a "different" way of doing the same thing, however;
it is likely to change the nature and structure of higher education.
The sophistication and flexibility of software structures for supporting distance education vary widely, from posting materials on the World Wide Web or simple electronic mail systems to conferencing systems that have been specially enhanced to support classroom-like experiences, particularly group discussions and joint projects. This paper focuses on what we have learned about teaching techniques that are effective in the networked environment as a result of a decade of experiences at New Jersey Institute of Technology (NJIT) using versions of a computer conferencing system enhanced with features to support collaborative learning, called the Virtual ClassroomTM. However, most of the themes emphasized are echoed in reports on similar projects[1,2,3]. The preliminary results reported here are based on the data for the first two years of a three year project, as of the fall of 1995; analysis of the full three-year project data was still underway as of the date of this report.
Generically, the Virtual Classroom is a teaching and learning environment located within a computer-mediated communication system. Rather than being built of steel and concrete, it consists of a set of group communication and work "spaces" and facilities that are constructed in software. Thus it is a "virtual" facility for interaction among the members of a class, rather than a physical space. Specifically, the Virtual ClassroomTM is NJIT's trademarked name for versions of its Electronic Information Exchange System (EIES2) with special software structures designed to support collaborative learning, including structures to order the transcript of discussions, to force active participation, to allocate unique assignment topics, and an electronic gradebook [4,5,6]. Participation is generally asynchronous or 'anytime/anywhere": that is, the participants may connect at any time around the clock, and from any location in the world accessible by the Internet or a reliable telephone system, rather than having to be online at the same time. The system stores the entries in a permanent, ordered transcript which keeps the equivalent of "bookmarks" to separate anything that is "new" for each individual from items that have already been seen.
The fact that the educational process is asynchronous means students may engage in more reflective thinking before having to answer or discuss issues, as compared to a synchronous or same-time interaction, and it also means that students can participate at their own convenience, and thus better fit the demands of a college degree program into busy lives.
It should be noted that a Virtual Classroom type of environment can be used successfully in many different media mixes:
Some ALN programs were designed to serve on-campus students, and
others to serve "distance" students. At NJIT, there
is a mix. If students who take most of their courses on campus
are permitted to, they will choose to take some portion of their
courses via an asynchronous mode such as Virtual Classroom, either
because of greater convenience, or because they prefer the mode
of interaction. In the fall of 1995, for instance, about half
of the students enrolled in the Virtual Classroom + Video undergraduate
courses were taking the majority of their degree courses on campus,
and half were "distance" students who took the majority
of their courses in distance modes. There is also a synergy; if
there are two sections, one face to face + VC and the other video
+ VC, they can be combined in the same online conferences and
activities, which seems to motivate students in both sections.
It does not "work" to simply make an ALN available and tell students that they can use it to ask questions about the readings or discuss aspects of the course at any time. If it is not a "required" and graded, integral part of the course, the majority of the students will never use it at all; and those who start to use it, will generally decide that "nothing is going on there" and will stop using it. The way in which ALN's can best enrich distance delivery, is to serve as the means to create the feeling of a true "class" or group of people who are learning together, and to structure and support a carefully planned series of collaborative learning activities which constitute the assignments for the course.
Collaborative learning is defined as a learning process that emphasizes group or cooperative efforts among faculty and students. It stresses active participation and interaction on the part of both students and instructors. Knowledge is viewed as a social construct, and therefore the educational process is facilitated by social interaction in an environment that facilitates peer interaction, evaluation and cooperation [7,8,9,10]. A variety of social-psychological mechanisms make collaborative learning effective, including "self-explanation," "internalization," and "appropriation" [11]:
The Virtual Classroom is an environment that was designed to facilitate collaborative learning -- among students, between students and instructors, among teachers, and between a class and wider academic and nonacademic communities. It also supports independent learning and generative, active learning techniques that are self-paced by each participant. For distance education students, the increased ability to be in constant communication with other learners is obvious. But even for campus-based courses, the technology provides a means for a rich, collaborative learning environment that exceeds the traditional classroom in its ability to "connect" students and course materials on a round-the-clock basis.
It is important to establish collaborative learning through substantive contributions by students to the class discussion from the very beginning of a course. However, with distance students, there are often problems in obtaining all of the books, videotapes, or other materials by the first week of the course. Therefore, the first week's assignment, in particular, should draw upon the students' own experiences, materials that are posted online, and general knowledge, rather than requiring the reading and synthesis of specific assigned materials.
An example of a collaborative learning strategy applied in the VC that is included in most courses is the "seminar" type of interchange in which the students become the teachers. Individuals or small groups of students are responsible for making a selection of a topic; reading material not assigned to the rest of the class; preparing a written summary for the class of the most important ideas in the material; and leading a discussion on the topic or material for which they are responsible.
Seminar-style presentations and discussions are thus an example
of a collaborative learning activity that is often difficult in
the Traditional Classroom (TC), but which tends to work very well
in the Virtual Classroom environment, even with fairly large classes
of undergraduates. Another example is to assign students to identify
key concepts or skills in each module of the course, make up a
question suitable for an exam to test mastery of this material,
and answer each other's questions. Exams then actually include
selections from the student-generated questions. Students are
thus made partners in deciding what it is that is important to
know related to course topics, and summarizing this key knowledge.
Other examples of collaborative learning strategy in the VC include
debates, group projects, case study discussions, simulation and
role-playing exercises, "ask an expert," sharing of
solutions to homework problems, and collaborative composition
of essays, stories, or research plans. (See Harasim et. al. 1995,
for descriptions and examples of these collaborative learning
strategies).
2. Experiences With ALN's at NJIT
Our first extensive experimentation with the Virtual Classroom began in 1986 and involved comparison to traditional face-to-face classrooms for a large number of courses over a period of two years in many different disciplines which used this medium of communication, including undergraduate courses in Sociology, Communication, English Composition, Management, Computer Science, Mathematics, and Statistics. For five of the courses, there was a "matched" section of the same course offered by the same instructor in a traditional classroom and using the Virtual Classroom (as the sole means of delivery, or in combination with a reduced number of face-to-face meetings). For other courses, there was no "match," and the comparison was subjectively made by the students and instructors to previous, traditional courses.
For the first study and the projects which have followed, we used
a "multi-method" approach to evaluation. This includes
pre and post-course questionnaires completed by students, direct
observation of online activities, interviews with selected students,
comparison of test or course grades or other "objective"
measures of performance, and regular reports by faculty, which
follow a common outline. The results presented here will rely
primarily on the post-course questionnaires measuring subjective
perceptions of learning. These were administered to all the VC
+ Video sections, plus sections of the same course taught by the
same teacher(s) in a different mode; either face to face plus
VC, completely face to face, or video only.
2.1. Virtual Classroom + Video: The Degree Programs
This project was designed to develop, offer, and assess the effectiveness of an undergraduate major in Information Systems delivered via Virtual Classroom plus video, to attain five objectives:
Some of the courses incorporating video for "lectures"
use standard public television courses, such as "Discovering
Psychology," produced by PBS. Most video segments are filmed
by NJIT in its "candid classroom" and then distributed
to remote students on videotape, or via broadcast on a cable channel
or satellite. In all video variations, the Virtual Classroom is
used for all assignments and additional discussions.
Altogether 26 courses were converted to VC + video format, and some were offered 6 to seven times thus far. Enrollments have shown a fairly steady upwards trend, aided by a substantial investment of personnel time and money in recruiting efforts (See Figure 1). The CIS department added the additional courses necessary to complete the more technical B.S. in Computer Science, which also increased total student enrollments.
2.2. Access: Overcoming Logistical Problems
The use of any medium for teaching interacts with characteristics of the students, the institutional context, and the subject matter. Thus, results for NJIT may not be generalizable to different contexts. The majority of NJIT's students are "first generation" college students, who must work while attending school and who commute rather than live on campus. They are faced with overcrowded classes with sections that fill up early and then become closed (due to budget cuts that reduced staff and raised class sizes). They are also faced with degree programs that because of their technological nature, have extensive sequences of courses that build on one another and must be taken in order. If a student is closed out of a course one semester or must withdraw, he or she might lose a whole semester waiting for another opportunity to take this course which is a pre-requisite for subsequent courses.
In addition, many of the courses are offered more slowly than would be possible for the better prepared or more motivated students to proceed. The "mass lectures" that have emerged as a result of budget cuts tend to slow down to the pace of the "lowest common denominator" and frustrate the better students. Both video and VC allow "fast forward" and "replay" self-pacing.
By using a combination of video (tape and/or broadcast) plus Virtual Classroom to deliver courses, students may more easily fit them into busy lives as employees and family members as well as students. Since the vast majority of NJIT students work 20 hours a week or more, and many work 40 hours a week or more (in order to pay their own educational expenses and/or to help support their families) it is very difficult for them to fit courses they need to graduate into their schedules. By offering distance education sections, scheduling difficulties can be alleviated. Students can graduate in fewer calendar years because they can more easily fit in all the courses they want.
In addition to the asynchronous nature of the VC helping to overcome course enrollment and completion difficulties, we are trying several innovations that may better facilitate self-pacing and improved rates of progress towards the degree, while maintaining or enhancing quality. These innovations include double courses offered during the same semester, independent study opportunities during the summer, and "late start" courses.
Many of the courses in the CIS majors form a series of prerequisites and must be taken in order. For example, the calculus perquisites must be taken before the first introductory course(113), which must be taken before 114 (the second part of Introduction to Computer Science); which serves as a pre-requisite to more advanced courses. Students can lose many semesters trying to get through this rigid sequence of courses, one at a time, meanwhile being closed out of some of the courses in the sequence some semesters.
We have redesigned a version of the introductory courses in this
series of pre-requisites as an integrated "double course"
in the Spring semester each year, for those students who were
making up pre-requisites in the fall and were unable to start
their major. It covers 113 at twice the normal pace, devoting
twice the normal number of hours to this task. By the middle of
the semester, some of these "fast track" students may
decide that they need a full semester after all; they drop into
a slower paced group online. Those who can learn at the accelerated
pace complete the first course in the first half of the semester,
and then go on to take the follow on course during the second
half of the semester. The participating faculty members coordinate
their efforts to provide a "seamless transition" between
the two courses. We have found that many students are able to
learn at the increased pace because of the intensive interaction
and the support available online. Accelerated (7 week) versions
of many of the ALN courses are also offered during the summer,
for students who need to "make up" a missing course.
Every time students successfully complete an accelerated course,
they can conceivably cut an entire semester off of the number
of semesters needed to finish their degree program.
3. Hypotheses and Findings on Student Impacts: Interim Results
The most basic hypotheses in our initial 1986- 1988 study about impacts of the Virtual Classroom on students are listed below; all were supported (see Hiltz, 1994, for complete data and analysis). They are also guiding some of the data analysis for the current Video + VC project; brief descriptions of the measures of variables used and some interim results will be presented below.
H1: Mastery of course material in the Virtual Classroom will be equal or superior to that in the traditional classroom.
H2: VC students will report higher subjective satisfaction with the VC than the TC on a number of dimensions, including improved overall quality, whereby the student assesses the experience as being "better" than the TC in some way, involving learning more on the whole.
H3: Those students who experience "group learning"
in the Virtual Classroom are most likely to judge the outcomes
of online courses to be superior to the outcomes of traditional
courses.
3.1. Problems reported by Students
On the post-course questionnaire, students in all modes were asked a series of questions about the course, the instructor, and their experiences. The relative distribution of responses allows us to compare different delivery modes indirectly. It should be noted that self-report data from questionnaires suffers from some validity issues. They can be biased (either favorably or unfavorably) by how much the student "likes" the professor, and by perceptions of risk that the answers may somehow affect one's grade. We attempted to deal with these issues in two ways. The questionnaires were mailed to all distance students by the assistant project director, and they were invited return them to her; confidentiality was guaranteed. If the questionnaires were handed out in class, a student volunteer was requested to place them all in an envelope and deliver them directly to the assistant project director.
The face to face and video only mode were combined into "no VC" for these analyses, because of the low number of video only responses. Most of the items showed no significant differences among modes; the differences in course and instructor ratings are evidently more affected by factors other than mode.
More students using the VC experienced more logistical problems than should occur in an "ideal" environment. These problems may seem "trivial" or local NJIT problems only; our observation after a decade of experience with ALN is that the specific nature of the technical and logistical problems changes, but that they are persistent.
For example, about 13% of the video + VC students indicated that access to a PC was a serious problem for them (using response 1 or 2 on a 7 point scale, where 1 was "serious problem" and 7 was "not a problem"; see Figure 2). These students should not have signed up for a course which required PC access. All registration materials, for instance, note that access to a "PC and modem are required." It seems that students are oblivious to information they do not want to see, or do not bother to look at the printed course descriptions before self-registering into any open section.
The most severe access problem is that 40% to 50% of students using VC at NJIT experienced serious problems with busy signals when they tried to dial in (Figure 3). The university does not seem to be able to provide enough modem access to meet the ever expanding demand; there were 48 dial-in ports to campus computing systems the first year of the project, and another 48 ports were added the second year, but within a week, they were saturated, too. One solution would seem to be for students to subscribe to an Internet Service Provider (ISP)for approximately $20 a month, and come in through the Internet, rather than direct dial. However, many students are resistant to spending this additional money, and even if they do, they encounter busy signals from their ISP. Our solution has been to create a "reserved" modem bank that allows only students enrolled in VC courses to use it; and to tell all students that they need to learn how to direct dial and quickly upload and download materials, as an alternative to ISP access.
In terms of disadvantages of learning networks as compared to
traditional courses, there are two items on the post-course questionnaire
that are worthy of note. Whereas the majority of students meeting
in face to face courses develop new friendships in class, only
a third of the VC + Video students report doing so (figure 4).
It is good to see that a substantial proportion can make new friends
via a written mode of communication, but it is obvious that close
personal relationships are more difficult. In addition, responding
to a different format of question in which VC students were asked
to compare their experiences to those they had in traditional
courses, VC students report that they are more likely to stop
"attending class" when they are busy with other things
(Figure 5). Since the class
does not meet at any particular time, it is easy to "postpone"
it, and this procrastination all too easily turns into falling
seriously behind.
3.2. Subjective Assessments: Relative Frequency Distributions for Different Modes
In looking at how interesting the course content was, the differences among delivery modes are not significant (p=.21), but those in the experimental Video + VC mode were most likely to strongly agree that the course content was interesting (Figure 6). Students in a mode using VC were slightly more likely to strongly agree that they "participated actively in the class discussion" (p=.06; Figure 7). This is a welcome result, since one of the objectives of the Virtual Classroom is to encourage/force active participation. On the overall course rating, the VC + video students are slightly more likely to rate it as either excellent or very good (p= .10; Figure 8). A few items yielded statistically significant differences. For example, for the question on whether the course was a "waste of time," the Video + VC mode students were more likely to strongly disagree ( p= .001; Figure 9).
A final section of the post-course questionnaire was administered only to those sections using the Virtual Classroom. The instructions to students were to compare their experiences in the course they had just completed, to their experiences in traditional, face to face college courses. The results of these items are presented in graphical form to show the distributions visually. First we will examine items related to access. The vast majority (71%) feel that ALN's provide better access to their professor (Figure 10). This is a very important outcome of ALN and provides one of the major ways in which it can improve overall access to learning opportunities. Of course, it is dependent upon the instructor's making himself or herself available online at least once a day. Despite the reported problems of some in reaching the computer facilities, over 2/3 (69%; see Figure 11) feel that the Virtual Classroom is, overall, "more convenient."
Only 15% did not "feel more involved in taking an active part" in the course (Figure 12). The results also indicate that "collaborative learning" did take place and did tend to have its intended motivational and learning consequences. For example, 55% felt more motivated to work hard on their assignments because other students would be reading them (Figure 13), and only 9% disagreed that reading the assignments of other students was useful (Figure 14).
Because of the convenient daily access to the class interaction, and the need and motivation to keep up with what their instructor and classmates are contributing, students (as well as faculty) tend to work harder in ALN based courses. For our interim results, the item on total amount of work is phrased as "I didn't have to work as hard for the online class." The majority (67%) disagree with this, while only 13% agree (Figure 15).
One overall measure of student satisfaction is whether students would choose to take another course using an ALN. Only 20% of the students agreed with a statement that they would not do so; 58% indicated that they would (Figure 16). This is a measure which undoubtedly would improve if logistical problems of reaching the network were lessened. In terms of the total amount that they learned, the modal response is that students are not sure whether they learned more or not than they would have in a traditional course; but 40% felt that they had learned more, where as 21% felt that they had not (Figure 17).
The final overall measures of subjective perceptions of the quality
of the educational experience are phrased in terms of one to seven
Likert-type scale items in which the polar points are 1="definitely
yes" and 7= "definitely not." The modal response
on whether the Virtual Classroom increased the efficiency of educational
delivery is that students are not sure; but 47% feel that it does,
as compared to 23% who feel that it does not (Figure 18).
In terms of whether the VC "increased the quality of education,"
the students are more sure; 58% say yes, while 20% say no (Figure 19).
3.3. Outcomes: Grades and Completion Rates
The results for the first two years support the hypothesis of equal or better performance in ALN as compared to traditional modes of course delivery. Two- year preliminary results for all courses combined showed no significant differences, though the withdrawal rate is generally higher than for traditional FtF sections. However, one really needs to look within each course in order to reach more definitive conclusions on this matter, since otherwise mode of delivery is confounded by differences in the level of difficulty and grading tendencies of different courses and instructors.
Figure 20 shows the results
for the first introductory course (CIS113) for the first two years
of the project. The various sections of this course were all taught
by the husband and wife team of Fadi and Maura Deek, who follow
exactly the same syllabus, grading policies, etc. As for the combined
results, there are results statistically significant at the .05
level. However, there is a strong suggestion that Video + VC is
much more effective than video alone (so much so that we do not
intend to offer any more video only sections, which may be educationally
sound but will prevent us getting a large enough "N"
to reach statistical significance on these conclusions!) There
is also a suggestion that a larger proportion of students in this
course do well (earning an A or B) in Video + VC sections than
in traditional sections (50% vs. 31%). The data thus do support
the conclusion that ALN is "as good or better" than
other modes of course delivery in terms of enabling students to
master course materials.
The following are a few preliminary impressions on impacts on faculty, gathered from completed course reports and discussions in our online "faculty lounge" where the issue of reporting and discussing perceived impacts has been raised. A full set of course reports by individual faculty members may be accessed on the author's Web home page.
The first consideration is impact on faculty workload:
There are also qualitative differences in the way a faculty member functions. For example, at the beginning of the semester, the instructor feels a bit like a "cybernetic cowboy," trying to herd all the students into the class corral. Willy Rossak, in the second week of a VC + Video course, says:
Well, at the moment I feel I am more busy to try to round up my
students instead of discussing the course topic. It seems the
instructor becomes system administrator for the conference, head-hunter,
and only then can be a lecturer.
Grading becomes a logistical problem Julian Scher says:
One unexpected impact was for the mentor to organize himself/herself for the grading process of homework/projects, much different than in the FtF mode. In the traditional class, I walk out of a class with a "batch" of projects/homework to grade, with an obvious batch grading. In video+vc, I gave students the option of submitting projects through e-mail, regular mail or in person. The completed homework arrived at different times and days. And I would feel guilty grading on the "first arrived-first graded" basis due to the desire to maintain consistency.
Impacts of ALN on the nature and size of the faculty teaching
load need to be considered by policy making bodies of the university.
For example, at NJIT we are trying to get agreement on a policy
whereby teaching assistants will be assigned to help with an ALN
course when class size goes over a certain limit, such as 30 and/or
that class sizes that require the splitting of a course into two
or more separate online groups will be counted as "double
sections" for course load purposes. There are also ongoing
issues related to intellectual property: who owns the rights to
the videotapes and the ALN materials created by a professor? Other
issues include whether and how faculty should be compensated for
their work in preparing course materials for distance delivery,
and how to provide adequate support to faculty to master the new
skills and techniques they need to be effective in an ALN environment.
When most students have multi-media work stations, the awkward logistics of tape distribution or student recording of broadcasts will probably be supplanted with CD-ROM based or World Wide Web-based digitized video modules. However, for the short term, the VC + video mix seems to work well for a wide variety of courses.
The future of this technology is tied to overcoming some of the difficulties related to the current situation of budget cutting and increased course loads for faculty in higher education. The first difficulty is the initial burden placed upon instructors to completely rethink the nature of their courses and adapt their teaching to a facilitative role. It is also necessary to provide some training for faculty on how to utilize collaborative learning approaches. Because there is an initial increase in workload in terms of creating materials in electronic form, adequate motivations or compensation must be provided, or faculty will feel that ALN will take away from the time devoted to research, which is clearly rewarded in terms of promotion and tenure. Such motivation might come in many forms, including reduced teaching load during the semester than a course is re-engineered for ALN delivery, or clear policies which reward such effort and innovation when promotion and tenure decisions are made.
Institutions may initially think that ALN based courses are a "cheap" way to deliver education. If done with attention to quality, which means that full time faculty should develop and at least be involved in conducting courses, it is actually more expensive. There is much to be done in discovering which tasks can be offloaded from faculty to lower priced teaching assistants, without substantially decreasing the quality of the product delivered. One of the central issues is to decrease the costs of higher education, and a differentiated staffing model may support this better than having faculty do everything, without substantially decreasing quality.
In long run, ALN's will increase the competition for students
and probably lead to a decrease in the total number of colleges
and universities in the U.S.. Large institutions and consortia
of smaller institutions will develop programs that students anywhere
can enroll in, and they will choose this rather than enrolling
in a "second rate" local college. ALN's will probably
accelerate the trend towards having larger proportions of courses
offered by adjunct faculty; already there are instances of faculty
members "advertising" their availability to teach Web-based
courses for colleges. ALN's will also contribute to the change
in the role of colleges and universities, from being primarily
residential institutions to take care of 18-21 year olds while
they finish growing up, to being centers for a variety of degree
programs designed to support students of all ages. Some of these
changes may seem desirable and others may seem undesirable. It
is up to faculty and to universities to put policies into place
that will encourage what are seen as desirable impacts, and discourage
the emergence of undesirable trends.
Development and research on the B.A.I.S. degree via a combination of video and VC was supported by the Alfred P. Sloan Foundation. Continuing research on appropriate software structures for collaborative learning via computer is partially supported by a grant from the National Science Foundation (NSF-IRI-9015236). A draft version of this paper was presented at the Sloan Conference on Asynchronous Learning Networks, Philadelphia, October 1995. A complete project evaluation report should be available by June 1997.
Many people have made significant contributions to the research
on this project, including Frank Mayadas, Murray Turoff, Ellen
Schreihoffer, and Beth Mardekian, as well as all the other faculty
members and student assistants in the Collaborative Systems Laboratory
at NJIT who participated. The author is also indebted to several
anonymous reviewers for the Journal of Asynchronous Learning Networks
for their excellent suggestions.
References
