Implementation Strategies

Implementation Strategies are a third component of the assessment matrix. After the development of the learning outcomes and the performance criteria, how would the 3-D virtual learning environment is implemented? Course developers and faculty members need to be able to answer the following questions as the implementation strategy is carefully crafted: “How will the outcomes be achieved?” and “Which course or program activities help to meet each outcome?” The course developers for the 3-D virtual learning environment tools should avoid broad goals for student achievement, e.g., students will be able to apply these concepts in subsequent courses in their educational program. The latter example would be too broad and not measurable. More important, course developers and faculty members need to allow students to meet the outcomes from the 3-D virtual learning environment and be able to apply in subsequent courses in their academic program. For example, if students are to learn the design process, or how to apply the concepts of supply and demand analysis to business applications, students must have an opportunity within their academic program to apply these concepts. The benefit of the latter is that the course developers and the faculty members will have a better idea how the learning tool(s) can be used in the curriculum and the impact it could have on subsequent outcomes of student learning.

Evaluation Methods

Evaluation methods, a fourth component of the assessment matrix, needs to be incorporated into the process of developing a 3-D learning environment. The purpose of developing effective evaluation methods is to determine if the program or a given course are providing the necessary outcomes for student learning. Valuation methods should attempt to answer the following questions: “What assessment methods will be used to collect data?” and “How will data be interpreted and evaluated?” Consequently, the evaluation methods should provide meaningful quantifiable guidelines that can be used in the evaluation process. Typically, the evaluation process would depend on the time and budgetary resources available. The development of the evaluation process should adhere to the following general guidelines:

• What type of evaluation methods will be used? Quantitative, qualitative, formative, summative? The evaluation approach will depend on the outcomes that are specified earlier in the development of the assessment matrix.

• It is not necessary to rely exclusively on one method but use a combination of methods, (e.g., quantitative and qualitative methods or mixed methods).

• It may be difficult to obtain purely objective assessment results. However, there are advanced methods that can be used to assess these types of outcomes.

Prus and Johnson (1994) delved into the evaluation tools that can be used in a traditional classroom that can be readily extended to the 3-D virtual learning environment formats. These classroom assessments that can be used to compile data for the evaluation process:

• Commercial, norm referenced standardized exams

• Locally developed exams

• Oral examinations

• Performance appraisals

• Simulations

• Written surveys and questionnaires

• Third party reports

• Portfolios

• Archival data

• Behavioral observations

• Exit interviews and other interviews

Timelines

The Timeline component of the assessment matrix asks when the evaluation will be conducted for the 3-D virtual learning environment tool. More specifically, following is the important question to be answered: “When will you measure?” The time to conduct the evaluation will depend on the types of outcomes that were created. For example, some outcomes require formative (midcourse) measurements; others are summative and can be put off until the completion of the course. In some cases it is best a combination of formative and summative assessments will work well. Some skills would take time to assess because the particular academic skill takes time to develop. Because of these skills, these evaluations may just take place at one point during the academic term. For developmental cognitive skills such as oral and written communication skills and critical thinking skills, these skills would be measured by formative assessments and would require assessment periodically as a means to remedy any deficiencies before it is too late. Technical knowledge from specific courses would be best assessed at certain points in the student’s academic program. Prus and Johnson (1994) stated there are no clear-cut guidelines to collect data; however, successful assessments often collect longitudinal data on students over the course of the student’s academic program. By doing the latter, faculty members can determine if the students are meeting the established learning outcomes. Typically, assessments should be periodic and one summative evaluation does not clearly provide any reasonable conclusions in regard to measuring changes in the learning outcomes of the students.

Feedback

The final part of the assessment matrix is feedback, and feedback is provided to the interested stakeholders, (e.g., faculty, students, other programs, accreditation bodies). The typical feedback questions that are asked include the following: “Who needs to know the results?”, “How can you convince stakeholders that the objectives were met?”, and “How can you improve your program and your assessment process?” Each of the stakeholders has different needs for the results of the evaluation and these specialized needs should be identified early so the appropriate evaluations can be conducted. To this end, the evaluation reports should be customized to meet the objectives of the appropriate audiences. Finally, if data collected does not provide value-added for any of the stakeholders, then the data should no longer be collected and disseminated.

FUTURE DIRECTIONS

A virtual world earning environment enables students to manipulate time and space, and students are able to learn content without risk and consequence. Despite these obvious advantages of learning in this virtual environment, how are learning outcomes effectively measured? What assessments are needed to ensure that learning actually occurred? Given that course developers must precisely know the learning objectives, course developers must develop the appropriate instructional strategies and learning activities to align with a virtual environment.

This discussion leads to the following two questions posed by Delgarno, Hedberg, and Harper (2002, p. 157).

• What are the important characteristics of learning tasks within a 3-D environment that will facilitate the development of a spatial cognitive model?

• What is the nature of the task support that is required for spatial knowledge development within a 3-D environment, and how should this support be provided within the environment?

This stand as questions to answer with future research. Once these questions are answered, course developers of 3-D virtual learning environments and faculty members will have evidence for best practice to maximize student learning. Finally, knowing precisely what and how to develop a 3-D virtual learning course, the course will do more than simply impress the students with technological “coolness”, but actually maximize student learning.

CONCLUSION

The delivery of education has changed dramatically in the past decade as the Internet usage soared exponentially and these changes in delivery has enabled students to purse an education that might have not been possible. In addition to the accessibility to education, 3-D virtual learning environments were introduced to help promote and enhance student learning. However, greater systematic research is needed to confirm that students are actually learning from the 3-D virtual learning environments. As with any form of evaluation, the research must be carefully crafted to assure that meaningful results are obtained. Until a more robust research design is developed for measuring 3-D virtual learning environments, course developers would need to rely mainly on qualitative feedback from students, faculty members, and other academicians that are using 3-D virtual learning environments to deliver an education.

REFERENCES

Abeles, T. P. (2007). Education unbound. On The Horizon , 15(4), 199–203. doi:10.1108/10748120710836219

Ai-Lim Lee, E., Wong, K. W., & Fung, C. C. (2010). How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education , 55(4), 1424–1442. doi:10.1016/j.compedu.2010.06.006

Appana, S. (2008). A Review of Benefits and Limitations of Online Learning in the Context of the Student, the Instructor and the Tenured Faculty. International Journal on E-Learning , 7(1), 5–22.

Atkinson, T. (2009). Second Life for educators: Teaching and learning. TechTrends , 53(3), 30–32. doi:10.1007/s11528-009-0285-7

Barker, J., & Gossman, P. (2013). The learning impact of a virtual learning environment: Students’ views. Tean Journal, 5(2). Retrieved from http://bit.ly/AtMwtr

Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction. Retrieved from http://www.hindawi.com/journals/ahci/2013/136864/cta/

Bower, M. (2008). Affordance analysis—matching learning tasks with learning technologies. Educational Media International , 45(1), 3–15. doi:10.1080/09523980701847115

Braman, J., Jinman, A., & Trajkovski, G. (2007). Exploring virtual worlds as an extension to classroom learning. In C. A. Shoniregun & A. Logvynovskiy (Eds.), Proceedings of the International Conference on Information Society (i-Society 2007) (pp. 82-86). London, UK: e-Centre for Infonomics. Retrieved from http://air.unimi.it/bitstream/2434/64612/1/i-SocietyProceedings.pdf

Britain, S., & Liber, O. (2000). A framework for the pedagogical evaluation of virtual learning environments. JTAP Report no. 041. Retrieved from http://www.leeds.ac.uk/educol/documents/00001237.htm

Calongne, C. (2008). Educational frontiers: Learning in a virtual world. EDUCAUSE Review , 43(5). Retrieved from http://www.educause.edu/ero/article/educational-frontiers-learning-virtual-world

Chapman, D. D., & Stone, S. (2010). Measurement of outcomes in virtual environments. Advances in Developing Human Resources , 12(6), 665–680. doi:10.1177/1523422310394792

Chen, C. J. (2010). Virtual reality (VR)-based learning environment: Design, develop, evaluate . Saarbrücken, Germany: Lambert Academic.

Chen, C. J., & Teh, C. S. (2000). Design issues of desktop virtual reality educational applications . In Khalid, H. M. (Ed.), Virtual reality: select issues and applications (pp. 63–78). London: Asean Academic.

Chen, C. J., Toh, S. C., & Ismail, W. M. F. W. (2005). Are learning styles relevant to virtual reality? Journal of Research on Technology in Education , 38(2), 123–141. doi:10.1080/15391523.2005.10782453

Colaric, S., & Taymans, J. (2004). Introducing Faculty to Online Learning: An Online Course as Professional Development. In G. Richards (Ed.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2004 (pp. 549-554). Chesapeake, VA: AACE.

Coppola, N. W., Hiltz, S., & Rotter, N. G. (2002). Becoming a virtual professor: Pedagogical roles and asynchronous learning networks. Journal of Management Information Systems , 18(4), 169–189.

Dawley, L. (2007). Persistent social learning: An emergent ID model for virtual world design [Video file]. Retrieved from https://admin.acrobat.com/_a768376479/p17101188/

De Bono, E. (1985). De Bono’s Thinking Course . New York, NY: Facts On File Publications.

Dede, C. (2005). Planning for neomillennial learning styles. EDUCAUSE Quarterly , 28(1), 1–10.

Delgarno, B., & Lee, M. (2010). What are the learning affordances of 3-D virtual environments? British Journal of Educational Technology , 41(1), 10–32. doi:10.1111/j.1467-8535.2009.01038.x

Dickey, M. D. (2005). Three-dimensional virtual worlds and distance learning: Two case studies of Active Worlds as a medium for distance education. British Journal of Educational Technology , 36(3), 439–451. doi:10.1111/j.1467-8535.2005.00477.x

Eccles, R. G. (1991). The performance measurement manifesto. Harvard Business Review , 69(1), 131–137.

Education Commission of the States. (1995). Making quality count in undergraduate education . Denver, CO: Education Commission of the States.

Fry, H., Ketteridge, S., & Marshall, S. (2003). A handbook for teaching and learning in higher education: Enhancing academic practice (2^nd ed.). London; Sterling, VA: Kogan Page.

Gardner, J. W. (1964). Self-Renewal . New York: HarperCollins.

Gee, J. P. (2003). What videogames have to teach us about learning and literacy . New York: Palgrave Macmillan.

Giroux, H. A. (2002). Neoliberalism, corporate culture, and the promise of higher education: The university as a democratic public sphere. Harvard Educational Review , 72(4), 425–464. doi:10.17763/haer.72.4.0515nr62324n71p1

Hedberg, J., Harper, B., & Dalgarno, B. (2002). The contribution of 3D environments to conceptual understanding. In O. J. McKerrow (Eds.), Winds of Change in the Sea of Learning: Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education(Vol 1, pp. 149-158). Auckland, New Zealand: UNITEC, Institute of Technology.

Hilbelink, A. J. (2009). A measure of the effectiveness of incorporating 3-D human anatomy into an online undergraduate laboratory. British Journal of Educational Technology , 40(4), 664–672. doi:10.1111/j.1467-8535.2008.00886.x

Hodge, E. M., & Collins, S. (2010). Collaborative efforts: Teaching and learning in virtual worlds. [Retrieved fromhttp://www.educause.edu/ero/article/collaborative-efforts-teaching-and-learning-virtual-worlds]. EDUCAUSE Review , 45(3), 62–63.

Holmberg, B. (2003). Distance education in essence: An overview of theory and practice in the early twenty-first century (2nd ed.). Germany: Oldenburg.

Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses: Shifting the focus from teaching to learning . Boston: Allyn & Bacon.

Jamlan, M. (2004). Faculty opinions towards introducing e-learning at the University of Bahrain. International Review of Research in Open and Distance Learning , 5(2).

Janicki, T., & Steinberg, G. (2003). Evaluation of a computer-supported learning system. Decision Sciences Journal of Innovative Education , 1(2), 203–223. doi:10.1111/j.1540-4609.2003.00018.x

Janson, B. (2013). Faculty Perceptions of Instruction in Collaborative Virtual Immersive Learning Environments in Higher Education [Dissertation & Theses Collection]. Paper AAI3558390.

Jarmon, L., & Sanchez, J. (2009). The educator’s coop: A model for collaboration and LSI communication research in the virtual world. The Electronic Journal of Communication , 19(1), 1–29.

Jarmon, L., Traphagan, T., Mayrath, M., & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life. Computers & Education , 53(1), 169–182. doi:10.1016/j.compedu.2009.01.010

Johnson, A. (2008). A nursing faculty's transition to teaching online. Nursing Education Perspectives , 29(1), 17–22.

Jonassen, D. H. (1991). Objectivism versus Constructivism: Do we need a new philosophical paradigm? Educational Technology Research and Development , 39(3), 5–14. doi:10.1007/BF02296434

Jonassen, D. H., Peck, K. L., & Wilson, B. G. (1999). Learning with technology: A constructivist perspective . Upper Saddle River, NJ: Prentice Hall, Inc.

Kelton, A. J. (2008). Virtual worlds? Outlook good. EDUCAUSE Review , 43(5). Retrieved from http://www.educause.edu/EDUCAUSE+Review/EDUCAUSE ReviewMagazineVolume43 /VirtualWorldsOutlookGood/163161

Kim, J. H., Park, S. T., Lee, H., Yuk, K. C., & Lee, H. (2001). Virtual reality simulations in physics education. Retrieved from http://imej.wfu.edu/articles/2001/2/02/index.asp

Kirkpatrick, D. (1979). Techniques for evaluating training programs. Training and Development Journal , 33(6), 78–92.

Kolb, A.Y., & Kolb, D. A. (2005). The Kolb Learning Style Inventory – Version 3.1: 2005 Technical Specifications. Haygroup: Experience Based Learning Systems Inc.

Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development . New Jersey: Prentice-Hall Inc.

Kotsilieris, T., & Dimopoulou, N. (2013). The evolution of e-learning in the context of 3-D virtual worlds. The Electronic Journal of e-Learning. 11(2), 147-167.

Kuhs, T. M., Johnson, R. L., Agruso, S. A., & Monrad, D. M. (2001). Put to the test – tools and techniques for classroom assessment . Portsmouth, NH: Heinemann.

Kurzweil, D., & Marcellas, B. (2008). The faculty - instructional designer dyad. Proceedings ofWorld Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 3445-3446). Chesapeake, VA: AACE.

Laurillard, D. (2002). Rethinking university teaching: a framework for the effective use of learning technologies (2nd ed.). London, New York: Routledge/Falmer. doi:10.4324/9780203304846

Lewis, C. C., & Abdul- Hamid, H. (2006). Implementing effective online teaching practices: Voices of exemplary faculty. Innovative Higher Education, 31(2), 83-98.

Lissitz, R. W., & Schafer, W. D. (2002). Assessment in educational reform: Both means and ends . Boston: Allyn & Bacon.

Luciano, C., Banerjee, P., & DeFanti, T. (2009). Haptics-based virtual reality periodontal training simulator. Virtual Reality (Waltham Cross) , 13(2), 69–85. doi:10.1007/s10055-009-0112-7

Martens, R., Bastiaens, T., & Kirschner, P. A. (2007). New learning design in distance education: The impact on student perception and motivation. Distance Education , 28(1), 81–93. doi:10.1080/01587910701305327

Mayes, R., Ku, H., Akarasriworn, C., Luebeck, J., & Korkmaz, O. (2011). Themes and strategies for transformative online instruction: A review of literature and practice. Quarterly Review of Distance Education , 12(3), 151–166.

Medawar, P. (1969). Induction and Intuition . Philadelphia: American Philosophical Society.

Mergel, B. (1998). Instructional design & learning theory. Retrieved from http://www.usask.ca/education/coursework/802papers/mergel/brenda.htm

Minocha, S., & Roberts, D. (2008). Laying the groundwork for socialisation and knowledge construction within 3-D virtual worlds. ALT-J . Research in Learning Technology , 16(3), 181–196. doi:10.1080/09687760802526699

Moore, A. H., Moore, J. F., & Fowler, S. B. (2005). Faculty development for the net generation. In D. G. Oblinger, & J. L. Oblinger (Eds.), Educating the Net Generation: Educause. Retrieved from http://www.educause.edu/educatingthenetgen/

Moore, M. G. (1999). Monitoring and evaluation [Editorial]. American Journal of Distance Education , 13(2), 1–5. doi:10.1080/08923649909527019

Nelson G. (2001). Do no harm: A first measure of effectiveness in small distance education programs.Proceedings of ED-MEDIA 2001: World Conference on Educational Multimedia, Hypermedia, and Telecommunications. Tampere, Finland.

Nettles, K., Dziuban, C., Cioffe, D., Moskal, P., & Moskal, P. (2000). Technology and Learning: The ‘no significant difference’ phenomenon: A structural analysis of research on technology enhanced instruction . In Dziuban, C. D., & Moskal, P. D. (Eds.), Distributed Learning Impact Evaluation . Orlando: University of Central Florida.

Olds, B. M., & Miller, R. L. (1998). An assessment matrix for evaluating engineering programs. The Journal of Engineering Education , 87(2), 173–178. doi:10.1002/j.2168-9830.1998.tb00338.x

Oravec, J. A. (2003). Some influences of online distance learning on US higher education. Journal of Further and Higher Education , 27(1), 89–103. doi:10.1080/03098770305628

Panda, S., & Mishra, S. (2007). E-learning in a mega open university: Faculty attitude, barriers and motivators. Educational Media International , 44(4), 323–338. doi:10.1080/09523980701680854

Petrina, S. (2007). Advanced teaching methods for the classroom . Hershey, PA: Information Science Publishing. doi:10.4018/978-1-59904-337-1

Phillips, J., & Berge, Z. (2009). Second Life for dental education. Journal of Dental Education , 73(11), 1260–1264.

Phillips, J. J. (1996). How much is the training worth? Training & Development , 50(4), 20–24.

Prus, J., & Johnson, R. (1994). A critical review of student assessment options . In Bers, T. H., & Mittler, M. L. (Eds.), Assessment & Testing, Myths & Realities. New Directions for Community Colleges, 88 (pp. 69–83). San Francisco: Jossey-Bass Publishers. doi:10.1002/cc.36819948811

Reigeluth, C. M. (1999). What is instructional-design theory and how is it changing? In Reigeluth, C. M. (Ed.), Instructional-design theories and models - a new paradigm of instructional theory (pp. 5–29). New Jersey: Lawrence Erlbaum.

Roblyer, M. D. (2003). Integrating educational technology into teaching (3rd ed.). NJ: Pearson Education.

Rock, D., & Schwartz, J. (2006). The neuroscience of leadership. Retrieved from http://www.strategy-business.com/article/06207?gko=6da0a

Rogers, E. M. (2003). Diffusion of innovations . New York, NY: Free Press.

Rokeach, M. (1960). The open and closed mind . New York: Basic Books.

Rovai, A. P. (2002). A preliminary look at the structural differences of higher education classroom communities in traditional and ALN courses. Journal of Asynchronous Learning Networks , 6(1), 41–56.

Rudestam, K. E., & Schoenholtz-Read, J. (2010). The flourishing of adult online education: An overview . In Rudestam, K. E., & Schoenholtz-Read, J. (Eds.), Handbook of online learning (pp. 1–18). Los Angeles, CA: Sage.

Rungtusanatham, M., Ellram, L. M., Siferd, S. P., & Salik, S. (2004). Toward a typology of business education in the Internet age. Decision Sciences Journal of Innovative Education , 2(2), 101–120. doi:10.1111/j.1540-4609.2004.00040.x

Salomon, G., & Perkins, D. (1998). Individual and social aspects of learning. Review of Research in Education , 23, 1–24.

Shieh, R., Gummer, E., & Niess, M. (2008). The quality of a web-based course: Perspectives of the instructor and the students. TechTrends: Linking Research & Practice to Improve Learning , 52(6), 61–68. doi:10.1007/s11528-008-0220-3

Sletten Sommerseth, K. (2014). Facilitating collaborative learning in 3-D virtual worlds. A Qualitative meta study [Unpublished thesis]. Reprosentralen, University of Oslo.

Sloan Consortium. (2006). Faculty satisfaction. Needham, MA: SloanCWiki. Retrieved from http://www.sloan-c-wiki.org/wiki/index.php?title= Faculty_Satisfaction.

Soto, V. (2013). Which instructional design models are educators using to design virtual world instruction? Merlot Journal of Online Learning and Teaching, 9(3). Retrieved from http://jolt.merlot.org/vol9no3/soto_0913.htm

Steinberg, A. D., Bashook, P. G., Drummond, J., Ashrafi, S., & Zefran, M. (2007). Assessment of faculty perception of content validity of PerioSim©, a haptic-3-D virtual reality dental training simulator . Journal of Dental Education , 71(12), 1574–1582.

Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In Y. B. Kafai, W. A. Sandoval, N. Enyedy, A. S. Nixon, & F. Herrera (Eds)., Embracing diversity in the learning sciences: Proceedings of the Sixth International Conference of the Learning Sciences (ICLS’04) (pp. 521–528). Mahwah, NJ: Lawrence Erlbaum.

Suebnukarn, S., Haddawy, P., Rhienmora, P., & Gajananan, K. (2010). Haptic Virtual Reality for Skill Acquisition in Endodontics. Journal of Endodontics , 36(1), 53–55. doi:10.1016/j.joen.2009.09.020

Vaill, P. (1996). Learning as a way of being . San Francisco: Jossey-Bass Publishers.

Vickerstaff, K. (2015). Implementation of technology enhanced learning pedagogy and impact on employability and learning within engineering education frameworks [Unpublished doctoral dissertation]. Plymouth University. Plymouth, UK.

Wasilik, O., & Bolliger, D. U. (2009). Faculty satisfaction in the online environment: An institutional study. The Internet and Higher Education , 12(3-4), 173–178. doi:10.1016/j.iheduc.2009.05.001

Wegner, S. B., Holloway, K. C., & Garton, E. M. (1999). The effects of internet-based instruction on student learning . Journal of Asynchronous Learning Networks , 3(2).

Wiggins, G. (1998). Educative assessments: Designing assessments to inform and improve student performance . San Francisco: Jossey-Bass Publishers.

Yılmaz, R. M., Topu, F. B., Goktas, Y., & Coban, M. (2013). Social presence and motivation in a three-dimensional virtual world: An explanatory study. Australasian Journal of Educational Technology , 29(6), 823–839.

KEY TERMS AND DEFINTIONS

E-Learning: Learning conducted via electronic media usually on the Internet.

Logic Bubbles: A term to describe the set of values, needs, beliefs and experiences that a person sees the world through as described by Edward de Bono.

Lateral Thinking: Applying an indirect and creative approach for solving problems and using reasoning that is not immediately obvious by involving ideas that may not be obtainable by using only traditional step-by-step logic as described by Edward de Bono.

Oculus Rift: The Oculus Rift is a set of virtual-reality goggles that will work with computers or mobile devices.

Second Life: A free 3-D virtual world where users can socialize, connect, and create using free voice and text or chat.

Three-Dimensional (3-D) Learning Environment: This type of environment allows interaction with other networked individuals as well as manipulation of objects (Loeffler & Andersen 1994).

Three-Dimensional (3-D) Virtual Learning: Web-based virtual learning provides an opportunity for teaching and learning without the necessity of buying expensive technology and equipment.

Two-Dimensional (2D) Learning Environment: 2D Interactive environments are a form of virtual environment with figurative appearance. Frequently, avatars communicate in bubbles and there can be interactive objects and multimedia imports.

Virtual Worlds: A virtual world is a computer-based online community environment that is designed and shared by individuals so that they can interact in a custom-built, simulated world (techopedia.com).

Chapter 4