d) Evaluation

MOOC – Virtual Hackathon

Module – Evaluation

Prof. Adela García-Aracil

This lesson proposes a set of interventions aimed at evaluating a hackathon approach. We propose to use a hackathon format as a way to raise interest among potential future professionals and spread knowledge to the larger population. Hackathons are time-bounded events during which participants from diverse backgrounds form teams and work on projects that are of interest to them. Hackathons are more and more used as a tool for education and learning, and in fact, learning has been cited as one of the key motivations for individuals to participate. However, there is a need for a hackathon approach in improving the level of knowledge among participants, and the way to control it is throughout the evaluation process. While learning can be considered an essential part of a hackathon, it is relevant to measure what organizers want participants to learn and the difference of what participants usually have learnt at a hackathon.

Section 1. Presentation of a hackathon as a tool for learning and research

In this lesson, hackathon is presented as a tool for learning and research. It can be synthesis from the literature that hackathon is an invention development method in which experts from interdisciplinary fields attempt to solve a challenge or a group of challenges in a specific amount of time (Rys, 2021 – Invention Development. The Hackathon Method. Knowledge Management Research & Practice, 579). The hackathon may be used in various environments and on different occasions, as it is very flexible in its nature and development. Because hackathons are focused on a challenge, they always require invention on the part of the involved people. For example:

• If the participants are required to solve a challenge, they are expected to come up with a solution
• If participants are asked to create a new business, it is necessary for them to invent a way to do so

Section 2. Hackathon as a tool for learning

Hackathons emerged in 1999 from technology and non-technology-based companies to provide intellectual and creative ideas. After that, hackathons spread across various domains ranging from large corporations, small-medium size enterprises, student events and civic engagement. This adoption has broadened the focus of hackathons from creating innovative ideas or software products to informal and collaborative learning, expanding or creating communities, supporting open civic innovation, tackling social and environmental issues and more.

Hackathons involve newly formed teams or existing communities. According to literature, corporates and institutes organize hackathons for the internal or external audience. Internal hackathons are conducted within the company’s premises to identify novel business processes, generate ideas, promote healthy competition or also help identifying new talent. The hackathons conducted at educational institutes for recruitment purposes and for generating new ideas from students are the external hackathons.

The studies on student hackathons indicate that hackathons are considered an innovative pedagogical tool. In this sense, hackathons provide engaging and fun learning opportunities. Educational institutes can adopt student hackathons as pedagogy, assessment, evaluation and a non-traditional form of learning. It can further help to identify concepts and theories from course content, provide great ideas, extend productivity and apply learning to practice.

Anyway, external or internal hackathons, if the purpose is used it as a learning tool, the designed of the learning approach requires to consider

• Complex problems
• A real-world environment
• Opportunities for realistic interactions
• Student ownership of the process
• Student reflection, facilitation and instruction

This approach encourages

• Active participation by the learners, and
• Collaboration with their peers
• The role of the mentors is crucial in helping participants consider the challenge in a holistic way rather than simply jumping straight into design

Section 3. Hackathon as a tool for research

Literature review shows different types of hackathons. One of the typologies categorizes them as either towards community nurturing, issue-oriented or towards the search for innovation. Other authors distinguish hackathons depending on their focus, which could be on innovation, education, or communication. These categorizations are useful to orientate the design of a hackathon towards one focus. Later, the good connection between the design aspects and the hackathon’s outcomes is also relevant. Literature shows that organizations often run hackathons to enhance innovation processes, both directly through ideation or prototyping of new products or indirectly by increasing employees’ motivation for innovation. The close connection between hackathons and the production of innovative ideas has been supported in the academic literature. Under this perspective, hackathons are considered as a tool for research based on their focus (i.e. tech vs. issue), design characteristics (i.e selective vs. open, competitive vs. collaborative, physical vs. virtual vs. hybrid) and outcomes. After the hackathon, organizers should follow up with participants using questionnaires, interviews, and even team meetings to understand their attitudes and feedback.

• Hackathons pose a major challenge when it comes to research
• It is relevant to highlight that the event itself is very intense and requires a lot of focus and dedication all the time
• It is extremely difficult to observe, as there are many things happening at the same time

Participants, after interaction with peers or the public, have to recognized the relevance of the engagement with users and realise that they are dealing with a human problem.

Section 4. Relevance of Hackathon Evaluation Approaches

It is important to pay attention to the quality of every step of the evaluation process in a hackathon, not just the accuracy of data collection. Some dimensions of quality for evaluation:

Utility: far better an approximate answer to the right question.

Validity: how data are collected, analyzed and reported with validity

Feasibility: thinking about the ethical issues, keeping in mind the wider impacts of the process of doing evaluation, as well as the issues involved in data collecting and reporting.

In general, the hackathon challenges ask participants to innovate. However, innovation does not always mean invention of something new. It can involve translation from another setting, bricolage (combining) of existing elements, or systematization of some existing good practices.

Therefore, evaluation approaches for a hackathon should be applied at the different stages, step-by-step, before, during and after. The dominant method for evaluation is observation. However, there are other methods that can be used such as traditional methods (quantitative measures), processes, designs and approaches. However, whether existing evaluation tools, processes, methods and systems are not enough to meet current challenges, it is suggested to draw on innovation in evaluation, considering:

• Attribution issues: Did the intervention cause the impact(s)?
• Apportioning: The what extent can a specific impact be attribute to the intervention?
• Contribution: Did the intervention make a difference?
• Explanation: How has the intervention made a difference?

Transportability: Is the intervention likely to work elsewhere? What is needed to make it work elsewhere?

Prof. Rosa Isusi-Fagoaga

This lesson proposes the use of rubrics to assess the competencies (kind of knowledge, skills and attitudes) acquired or developed during the implementation of a hackathon. Hackathons are intensive, competitive events during which team activity is focused on solving a specific problem or creating a viable business idea or business model. Two main competencies are associated to hackathons: digital and entrepreneurial ones. However, other competencies are promoted such as: troubleshooting, critical thinking, teamwork, collaboration, negotiation, communication, time management, project management, among others.

Section 5. Evaluating competencies

In this lesson, we explain that there are different methods to evaluate competencies. One of them is through the use of a rubric.

The rubric is defined as a set of criteria and standards, generally related to learning objective, which are used to evaluate a level of performance of a task or behavior. Because they are designed according to specific criteria and standards, they allow impartial assessment.

The rubric can be used as training tool based on their potential to help students develop a vision of success, provide ongoing feedback on progress and what matters ranked from excellent to poor. Literature suggests that deep engagement is required to use rubrics as self- or peer-assessment tools. Involving the student in the co-creation of a rubric helps to demonstrate the competencies they need to master, and their progress in relation to expectations.

The rubric is usually presented as potential assessment tool based on a matrix of learning outcomes along a continuum on the horizontal axis, with the elements (or the criteria) for evaluating learning outcomes shown on the vertical axis.

Section 6. The use of rubrics at Hackathons

The quality of any educational program, in educational institutions, has traditionally been assessed by using input, process and output measures. Input measures review resources such as students, faculty, infrastructure and financial stability as these are considered essential for conduct of academic activities. Process measures assess the delivery and use of institutional services in terms of class sizes, student to faculty rations, teaching methods, etc. Along with these, output measures such as salary levels of graduates, placement and graduation rates, intellectual contributions by faculty and the like, reflect what an institution has achieved as a result of inputs applied. However, today, there is an emphasis of the perspective of the measurable advances that educational programs make in developing competencies in students. In other words, outcomes measures reveal what individual students learn. In this context, rubrics are guidelines that enable the assessment process of expectations, provides ongoing feedback and grading.

A rubric is defined as a document that articulates the expectations for an assignment by listing the criteria, or what counts, and describing levels of quality from excellent to poor. Apart from being considered as an effective tool for measuring, evaluating and reporting student achievement, rubrics are also designed to guide students’ learning, teachers’ instruction, course development and administrators’ program observations. Therefore, rubrics are held as a being direct assessment measures which help to answer the key questions driving outcomes assessment, i.e. how students learn; what students learn; how is student learning assessed; and how are assessment results used.

In a hackathon, rubrics can be used in every stage, for the evaluation of: the process, the team, the mentors, the organizers, the results, the competencies acquired, and so on.

***************************************************************************

Additional training material – READ and ANSWER QUESTIONS

Please, access to this article about the quality and effectiveness of descriptive rubrics. Here you have the link. http://dx.doi.org/10.1080/00131911.2014.929565. After reading it, answer the following questions:

Reference of the article: Susan M. Brookhart & Fei Chen (2015) The quality and effectiveness of descriptive rubrics, Educational Review, 67:3, 343-368, DOI: 10.1080/00131911.2014.929565

A1. A rubric is a coherent set of criteria for…

1. Politics, employers and representatives of NGOs about students’ performance level
2. Students envision what to do to improve their work
3. Comparing rating scales with traditional students’ performance scores

Correct answer: b

A2. Rubrics are used…

1. Only in higher education level
2. Only in primary education level and in post-secondary education
3. On formative assessment in education at all levels

Correct answer: c

A3. According to the authors of the paper, rubric is

1. Complementary to student learning or self-regulation
2. Synonymous of achievement
3. Synonymous of validity

Correct answer: a

A4. According to the authors of the paper, what are appropriate measures of reliability for rubrics?

1. Measures of absolute agreement among raters
2. Measures of relative agreement among raters
3. Both measures, absolute and relative agreements among raters

Correct answer: c

A5. Validity evidence for rubrics means that

1. Only one argument is needed for validate rubrics’ content
2. Sometimes is useful to distinguish various types of evidence to bear on a rubric’s validity argument
3. With a few exceptions, validity does not support the rubrics scores

Correct answer: b

A6. According to literature shown for authors,

1. Teachers and students have the same perspective on the reliability and validity of rubrics
2. Teachers and students have the same perspective on the reliability of rubrics
3. Teachers and students have different perspectives on the reliability and validity of rubrics

Correct answer: c

A7. According to the authors of the paper

1. Utility is the same as validity
2. Utility is not the same as validity
3. Validity is referenced to purpose “useful for what?”

Correct answer: b

A8. According to the results of the paper, rubric use is associated with

1. Always an increase of student’s achievement in writing and mathematics
2. Sometimes no significant effects on student’s performance
3. Always a decrease of student’s achievement in physics and social studies

Correct answer: b

A9. According to the literature review of the paper, using rubrics is associated with

1. Gains in self-regulation of learning or in self-efficacy
2. Non-effect in self-regulation of learning
3. Decreased student self-efficacy

Correct answer: a

A10. According to the literature review of the paper, using rubrics is associated with

1. Negative effects on students’ motivation in all cases shown in the paper
2. Sometimes positive effects on students’ motivation
3. Negative effects on teachers’ motivation

Correct answer: b