Online Professional Development for Improving Teacher's STEM PCK Competence, Can It Be An Alternative? (An Evaluation Using the CIPP Model)

. During the pandemic, face-to-face training should not be carried out to reduce the spread of the covid outbreak. Therefore, online teacher professional development is an alternative method to replace the face-to-face training. In a program implementation, evaluation is an important component to determine whether the program has been implemented well or not. This evaluation is important to determine the effect of this online training on improving STEM PCK teachers. The evaluation used the CIPP model (Context, Input, Process, and Product), with the embedded mixed method research. This research was conducted on 60 science teachers of SMP alumni online training organized by PPPPTK IPA, from the “Sayangi Bumi” classroom. The results of the context evaluation show that 100% of respondents stated that the program is in accordance with the needs of teachers to strengthen their PCK., while the resources used in the program (Input evaluation) were good. The program implementation process ran 100% as planned and the product evaluation shows an increased teachers' understanding of the STEM approach. Teacher considered that the online professional development is very useful for teachers and can develop their skills in implementing STEM learning during distance learning. Online PD, apart from low-cost, can actually be an alternative way in improving teachers' STEM PCK competence and it also can reach all islands in Indonesia.


Introduction
Indonesia is the largest archipelago country in the world, with an area of approximately 7.81 million km 2 , and it consists of 34 provinces.The number of science teachers in Indonesia is around 133,908 people and are scattered in all these provinces.One of the efforts to improve science teachers' competence is through Professional Development (PD).An effective PD is a crucial component to enhance the quality of science education since a qualified science teacher is able to improve the quality of the science teaching and learning process.A large number of teachers causes very limited access to training for science teachers.To overcome this, online PD can be a solution.Online PD is a distance learning that involves information technology (Turcsányi-Szabó, 2008).The delivery of learning materials is provided through the Learning Management System (LMS).Communication during learning can be done asynchronous (indirect) or synchronous (direct).Another reason for the online PD is because of the pandemic, where the face-toface activities is reduced.Nevertheless, the teacher's competence development must continue.Hence, the Government issued a policy to organize online PD.Online PD can be an excellent alternative to use.Online training can overcome the drawbacks of the faceto-face PD, which are high cost while on a limited budget (Masters et al., 2010).
Many advantages possessed by online training, including that it can be accessed anywhere and anytime which makes it easier to be implemented (Bickmore, 2012); and can be tailored to teachers' needs in classroom learning (Lom and Sullenger, 2011).
One of the needs of teachers in classroom learning is to increase competence in pedagogical knowledge, including STEM learning.In Indonesia, science teachers' knowledge of STEM is still very low (Wahono and Chang, 2019).This needs to be improved, considering that one of the obstacles in implementing STEM learning is the lack of PD for teachers regarding STEM.To improve the implementation of the STEM approach in the classroom, teachers must be provided with training and equipped with teaching materials (Kanadlı, 2019).Research showed that some teachers admitted that they did not have sufficient knowledge of content and the standards of integrated STEM education because it focuses on big ideas connected and interrelated (Shernoff et al., 2017).The development of teacher professionalism is strongly influenced by teacher subjectivity, which includes perceptions, prior knowledge, and the influence of the surrounding environment (Fore et al., 2015).
There are five components which are indicators of the achievement of STEM PCK according to (Srikoom et al., 2018), namely (1) orientations toward teaching STEM; (2) knowledge of STEM Curriculum; (3) knowledge of students' understanding in STEM; (4) knowledge of instructional strategies and representations for teaching STEM; (5) knowledge of assessment of STEM Learning.STEM PCK can be used as a standard to what extent PD STEM effectively improves teacher competence.STEM PCK is a combination of several categories, namely: STEM content knowledge, STEM integration knowledge, pedagogical knowledge, 21st-century skills knowledge, and context knowledge (Chai, 2019;Eckman et al., 2016;Eilks and Markic, 2011;Epstein and Miller, 2011;Yildirim and Sahin Topalcengiz, 2019).
Teachers' knowledge of the subject matter (content knowledge) affects student achievement (Darling-Hammond, 2000).Variety outcomes on the analysis of the relationship between teacher content knowledge and student achievement are also linked to the teacher's ability in teaching the content effectively (Nadelson et al., 2012).Pedagogical knowledge refers to knowledge of teaching methods, evaluating learning, and the learning process.Teachers with good pedagogical knowledge understand how students construct knowledge, cognitive, social, and developmental learning theories to apply them in the classroom.Pedagogical knowledge, hence, is related to how students learn, class management, knowledge about evaluation, knowledge about student characteristics, and the development of learning planning and implementation (Yusof et al., 2012).
To equip students to face future challenges, teachers' knowledge of 21st-century skills is important.Teachers have to possess skills such as learning and innovation skills (including critical thinking and problem solving, creativity and innovation, communication and collaboration, visual literacy, scientific and numerical literacy, interdisciplinary thinking, and basic literacy); information, media and technology skills (including information literacy; media literacy; and information, communication, and technology literacy); and life and career skills (including flexibility and adaptability, self-initiative and direction, social and cross-cultural skills, productivity and accountability, and leadership and responsibility (Voogt, 2010).
Lots of online teacher training have been implemented in Indonesia.To find out how effective the training program is, it requires a thorough evaluation.Evaluation is beneficial in making decisions about a program, whether the program can be continued or not, or identifying factors that should be improved from a program (Kirkpatrick and Kirkpatrick, 2006).One of the program evaluation models is the CIPP (Context, Input, Process, Product).CIPP evaluation model is a model of evaluation that looked at a program as a system.If the evaluator has decided to use this model, the evaluator should evaluate the program in more depth and detail by components.Evaluation is a systematic and continuous process and includes three basic steps, namely delineating of the questions to be answered and the information to be obtained; the obtaining relevant information; and the providing information to decision makers so as to improve program quality (Stufflebeam, 1971).There are three kinds of decisions that can be taken, namely: 1) continuing and disseminating good programs; 2) the program is revised because there are things that are not following the desired benchmarks/criteria; 3) the program is terminated because it is not good enough or there are violations in its implementation (Warju, 2016).
The use of the CIPP evaluation model provides a detailed and comprehensive picture of program success.This is needed as consideration for what should be improved from a program.The components to be evaluated using the CIPP model are: 1) context evaluation (C), to evaluate needs, problems and opportunities as a basis for determining objectives and priorities and assessing the significance of the results; 2) input evaluation (I), to evaluate the approach used as a means of achieving program and resources that support a program; 3) process evaluation (P), to evaluate the implementation process as planned and explain the results; and 4) product evaluation (P), to identify the desired and undesirable outcomes of program implementation, and determine its effectiveness (Stufflebeam, 2000).This model views that the most important purpose of the evaluation is not to prove a program is good but to improve a program.
Based on the presentation, researchers are interested in evaluating online STEM training programs using the CIPP evaluation model.The research questions in this article are: 1) Are the program objectives in accordance with the needs of the teacher?2) Are the resources used to support the success of the program?3) Is the process of implementing the activities in accordance with the plan?4) Does the product of the training participants describe the participants' understanding of STEM learning?

Methods
This study used mixed method research, where researchers collected and analyzed data, integrated the findings and drew conclusions by using two approaches, namely qualitative and quantitative approaches.This research method can provide a complete picture when compared to using only one method (Creswell, 2012).The design used is the embedded mixed-method design, where qualitative research is a large part of this research, and quantitative data is used as supporting data for the research results (Figure 1).

Results and Discussion
Evaluation Context (C) According to (Stufflebeam, 2000), context evaluation evaluates needs, problems, and opportunities as a basis for determining goals and priorities, and assessing the significance of the results.To evaluate the needs, data were obtained through document analysis.This program was developed based on the results of a survey by the PPPPTK IPA with 5,531 teachers as respondents.One of the results of the survey was that teachers need to improve their pedagogical content knowledge (PCK) competence.
The training program aims to assist teachers to teach science using the STEM approach during the pandemic for the topics Interaksi Makhluk Hidup dengan Lingkungan (Interactions between living things with their environment) and Pencemaran Lingkungan (Environmental Pollution).This program also helped teachers have a deeper understanding of science content, mastering computer and internet technology in supporting teaching and learning, and understanding the 5E model and Project-Based Learning (PjBL) in online STEM learning.
The objectives of the program were in accordance with the results of TNA, because it contained the pedagogical knowledge (P), in example and the 5E learning model and PjBL with STEM approach and content knowledge (C).Based on the survey results, 100 % of participants stated that the contents of the program are in accordance with the needs of classroom learning.This indicates that the context of the program is in accordance with the requirements of the teacher (relevance value = 1).
The objectives of this program are also in accordance with the vision and mission of PPPPTK IPA.The vision is to realize a professional science teacher development and empowerment centre, through the mission of developing models for enhancing the competence of science teachers.
Based on the results of the above analysis, it can be concluded that the objectives of this program are good based on the evaluation of the context.Some research revealed that teacher professional development program would be more effective if it meets the needs of teachers (Shriki and Lavy, 2012).Therefore, it is very important to determine the context of the PD program that is appropriate to the needs of the teacher.Teacher learning needs are not only to increase competence, but also to improve professionalism (Clarke and Hollingsworth, 2002).

Input (I)
In this online training, the learning management system (LMS) used was Moodle.Based on the results of interviews and observations, participants have no difficulty in understanding and using the menus in the LMS.Although there was no specific introduction to the use of this LMS, almost all participants can follow it well.The obstacle was found by the participants from Papua and West Papua Provinces where the internet connection is a problem there.
The essence of online PD is to improve a culture of collaboration with peers accustomed to using technology so that teachers can simulate activities that use information technology in the classroom (Turcsányi-Szabó, 2008).This is also a means of preparing students to face the challenges of the 21st century.
For the video conference (Vicon), this study used Zoom application which is very popular and widely used.Facilitators and participants stated that Vicon was an indispensable method during online training because it was the most supportive online discussion tool.
According to the participants, the language of teaching materials used in the program was good, easy to understand, and easy to apply in implementing STEM learning during distance learning.During the training, participants were invited to implement the teaching material directly to clarify the stages of unit implementation in the classroom.
In STEM learning, the delivery of learning material in class often uses the Project Based Learning, Problem Based Learning, and Design Based Learning models (Lecorchick et al., 2020).The development of teaching materials using this learning model can be developed for further STEM training programs.Apart from the LMS and teaching materials, the facilitators were also evaluated by the participants at the end of the activity.Table 2 illustrates the results of the participant's assessment of the facilitator.

Figure 2. Result of Facilitator Assessment
The table above showed that the facilitator facilitated the participants well.During the training process, participants assessed the facilitator's skills to conduct a Vicon, respond to participants' questions in the discussion forum, and provide feedback on participants' assignments.

Process (P)
The results of LMS observations and Vicon recordings show that the training process's implementation is in accordance with the plan.

Figure 3. Result of Implementation Assessment
From the table, it can be seen that 99% of respondents stated that the program implementation process had been carried out well (with an effectiveness value = 0.99).The assessment for each session is illustrated in the table below.Overall, the respondents were satisfied with the learning process in each session.There were only a few dissatisfied participants, 16% in session 2; 10% at session 4; and 11% in session 5.They stated that the learning process was boring.From the observations, this boredom could be caused by the requirement that all participants were required to make presentations.To overcome this, it is necessary to think about ways to make the participant presentation session more interesting.
Online training can be very effective in improving teacher competence (Binmohsen and Abrahams, 2020).It can be said that this online PD program can be an alternative for providing training to overcome the limitations of not being able to carry out face-to-face training during the pandemic, limited implementation of training compared to the number of teachers in Indonesia; and to overcome the geographic situation in Indonesia which is very broad.

Product (P)
During the training, facilitators gave the participants assignments to find out their understanding of the material provided in each session.The mean score for the product assessment was 90, meaning that all assignments were very well completed by participants.
Almost all participants gained a high score on the post-test.The participants had two attempts to carry out the post-test.The scores obtained by participants at the pre and post-test are illustrated in table 2. There is a very significant change in the post-test scores.This can be interpreted that the participants' achievement for each indicator of the training is very good.Participants had been able to explain the relationship between STEM learning and 21st-century skills, the engineering design process (EDP) in STEM learning, understand 5E and PjBL in the STEM approach, and develop the assessment instrument in STEM learning.
The products produced during the training can be used to measure what level of teacher's STEM PCK skills are.According to (Srikoom et al., 2018), STEM teachers PCK consists of five components, namely: 1) orientation towards learning STEM teachers; 2) Knowledge about STEM curriculum; 3) knowledge of students' STEM literacy; 4) knowledge of how STEM learning strategies; and 5) knowledge about the evaluation of STEM learning.
In the teacher orientation component towards STEM learning, the participants already understood how to implement the STEM learning process with the 5E and PjBL models by following the unit's steps.Based on the interview results, they were very interested in implementing STEM learning in their classroom.Therefore, teacher orientation toward STEM learning can be classified as good.
The knowledge of the STEM curriculum, a component which is described as the teacher's skills to formulate learning objectives in STEM learning, have not yet been reflected in this training.This is because the products produced by the participant were only limited to activity sheets.The other components, likewise, have not been seen from this program.Therefore, it is necessary to improve the program structure that has been employed.
Regarding the advancement of the engineering field, PD on STEM learning underwent several pedagogical approaches to meet different needs.PD on STEM learning is often implemented primarily to develop engineering knowledge (Chai, 2019).To develop a program design that will improve teacher STEM PCK, it is necessary to develop a program with five components, namely teacher orientation towards STEM learning; teacher knowledge of the STEM approach; teacher knowledge about how to measure students' STEM literacy; teachers' understanding of how to develop STEM learning in the classroom; and teacher knowledge regarding the evaluation of STEM learning (Srikoom et al., 2018).These focuses can be used as a reference for further development of the STEM PCK program so that the program is more focused.This program needs to be developed further in view of many teachers in Indonesia who feel the needs for PD to continuously study the STEM approach (Hasanah & Permanasari, 2019).The results of this study prove that online training can be a very good alternative to improve the competence of science teachers.The effectiveness of online training is the same as the effectiveness of face-to-face training (Binmohsen & Abrahams, 2020).What can support the increase in the effectiveness of online training is a well-integrated pedagogical and technological approach, the programs offered are innovative and in accordance with the needs of teachers, and use the right technology (Turcsányi-Szabó, 2008).

Conclusion
Based on the analysis and discussion of the results of this study, it can be concluded that the context developed in this program are in accordance with the needs of teachers; the input used has been well appreciated by the participants; the process was carried out in accordance with the plan and was considered very good by the participants, and participant products can describe the indicators of achievement.
This program needs to be further developed to improve STEM PCK teachers and student STEM literacy.Online PD can be a good alternative for implementing PD.Program development can be designed in Online PD, which is followed up with an evaluation of STEM learning implementation in the classroom.

Figure 1 .
Figure 1.Embedded Mixed Method Design

Figure 4 .
Figure 4. Results of Assessment in Each Session

Table 1 .
Components and Data Collection Methods Respondents in this study were 60 secondary school science teachers, alumni of an online STEM training program with the theme of Sayangi Bumi, which was organized by P4TK IPA.

Table 2 .
Value of Pre and Posttest