Paperclickers: Affordable Solution for
Classroom Response Systems

Several studies [8, 9] suggest that CRSs make the students more attentive: knowing that they will be polled make them better prepared for classes. Other studies [10, 9] mention frequent and positive interactions when using CRSs during discussion sessions, reporting better articulated thinking [11], since students have to commit to an answer, and, if required, defend it (when CRS are used in combination with PI).

There are also potential benefits of CRSs for instructors, since they can analyze learning in real time [8] and adapt the lecture accordingly [9, 12]. Some [8, 9] suggest that such immediate feedback increases learning performance.

Hunsu et al.’s meta-analysis [13] investigated the cognitive and non-cognitive effects of CRSs, compared to conventional lectures, comprising over 50 selected articles, with a total of 26,085 participants. They considered 111 independent learning outcomes, coded from all the variables present in the original studies, split in 86 cognitive and 25 non-cognitive outcomes. The meta-analysis shows that CRSs improve students’ participation in large lecture halls (a common, and particularly challenging, environment for STEM courses). It also reveals small, but positive, cognitive outcomes of those technologies, specially on higher-level learning goals, like knowledge transfer, or knowledge application.

Despite those positive results, CRS implementation is challenging. Technological problems may occur (devices not working properly, communication problems) [14]. Applying CRSs without pedagogical preparation may not benefit the learning process, and can even harm the positive perception about those devices within classes. Preparing proper questions — those which identify misconceptions — is hard; if a standard set of questions is not available for using the CRS, the instructor will have the burden of choosing them for each lecture [9].

Furthermore, CRS benefits only appear if classroom culture supports student engagement and interaction. Students must feel their answers are important, in order to engage, and to take the procedure seriously. Otherwise, adverse situations may arise, like giving wrong answers on purpose [14].

In their most usual form, CRSs use ‘clickers’, one small infra-red or radio-frequency transmitter per student, and one receiver per classroom[9]. When the instructor poses a question, the students use the transmitters to answer it, and the receiver instantly records and tabulates those answers. The instructor can display the results for immediate action, save the data for later analysis, or both [11].

‘Clicker’ solutions count with wide commercial availability and professional support. If correctly implemented and well-maintained, they allow CRSs to run smoothly and non-intrusively [9]. The commonest devices allow only multiple-choice or yes/no answers, but recent (and more expensive) versions allow numbers, words, or short phrases.

Those solutions, however, are expensive. Ideally, each student should own their transmitter (to avoid the time-consuming hassle of distributing and collecting the devices at each class), and receivers must be available for each classroom or, if they are portable, teacher. Teachers have to be trained to use the transmitters [11], and support must be readily available to solve problems [9], minimizing class disruption. The system needs continual maintenance to work optimally; in particular, it is critical to establish who will be responsible for the devices’ batteries (students, teachers, or technicians), and ensure that responsibility is taken seriously.

Due to those costs and inconveniences, CRSs based on ‘clickers’ are very challenging to implement in developing countries.

“Bring Your Own Device” (BYOD) — a model in which students use their own smartphones, tablets, or even laptop computers — replace hardware ‘clickers’ by software applications that transmit the answers through WiFi or mobile data [15]. There is a good offer of commercial BYOD software, sometimes from the same vendors of hardware ‘clickers’, even allowing hybrid solutions where both are supported at once.

CRS software usually has modest hardware requirements, allowing to apply BYOD to a wide range of devices, including obsolete/donated hardware. BYOD also allows very flexible forms of polling, that no other solution provides, including mini-essays, points and graphs in Cartesian coordinates, or even freehand drawings.

Teachers and students still need to be trained to use the system, and there still needs to be technical support, especially to solve connectivity issues. Indeed, network quickly becomes a bottleneck — both when using public (mobile data) or local (WiFi) infrastructure — as it is challenging to support many simultaneous connections. Ensuring the usage of the devices remains on purpose and productive is an additional challenge, as students may turn to leisure texting or web browsing.

BYOD assumes there is good network infrastructure, and that each student owns at least a smartphone, and that they feel safe to bring it to school [15]. Those assumptions are far from obvious in developing countries, especially in schools serving disfavored communities. For example, the Brazilian Internet Steering Committee shows [16] that as recently as 2014, even urban schools in Brazil had connectivity issues: although 93% of those schools had some access to the Internet, only 41% of them granted access to students. In addition, connectivity was often low-speed and unreliable.

Image-processing solutions minimize costs by giving the students passive devices, usually cards with especial colors or codes, and keeping all active processing into a single device, which remains with the teacher. Most often, the students rotate use a card printed with a special 2D barcode, which serves both as a location and orientation marker, and as a unique ID for each student. The students can answer multiple-choice questions by rotating the cards.

Amy and Amy patented a low-cost optical polling framework [17] with a generic computing element that recognizes the orientation of fiducial marks on printed cards. The proposed solution is available as a smartphone app --- Plickers¹¹ 1 https://plickers.com/, which currently accommodates up to 63 students, which must enroll on a web-based system. Fiducial markers on printed cards had already appeared on previous works, e.g., the augmented reality system ARTag [18]. Amy and Amy innovate by exploiting them for low-cost CRSs.

Cross et al. [2] also proposed a system that recognized the orientation of printed cards with unique IDs for each student, which they called ‘qCards’. The teacher captures the answers with an off-the-shelf webcam mounted on a laptop, with software to recognize, tabulate, and display the results. The authors ran initial trials on secondary schools in Bangalore, India, with 99.8% recognition accuracy, and 97% captured responses in a 25-student classroom.

Miura and Nakada presented similar work [3]: they used printed cards with fiducial markers as codes, and a similar setup of camera, and PC containing the software. Their system recognizes three rotational parameters for each cards (roll, pitch and yaw), allowing students to select one of many possible multiple-choice answers in a screen. A preliminary experiment, with 19 students succeeded in tracking 18 markers.

The solution proposed by Gain [19] approaches CRSs by colored blocks printed on cardboards and a camera-phone to capture images. Students select answers by picking different colors. They report 85% recognition accuracy in a medium-size class (up to 125 students). Although the system forgoes peer anonymity, and the possibility to track responses to individual students for later analysis, it is the only image-processing system tested in classes that big.

Finally, Ito and Miura [20] experimented a portable version of that previous system, recognizing the same fiducial marks in an tablet computer, including the capability of detecting the response printed card bending amount as an additional input mechanism, which could encode the student mood.

As far as we known, paperclickers is the only existing image processing CRS solution at the intersection of being an academic work, having its entire source code publicly released, and being available for download on user’s devices for actual, practical use. Being on that intersection allows future contributors to easily test hypotheses and add improvements to paperclickers, with real-world impact to users.

Paperclickers’ first design was similar to Cross et al.’s [2], but exchanging the PC+webcam setup by a mobile device with embedded camera. In both designs, the instructor uses the camera to film the class, as the students hold up cards with their answers — using four different orientations to pick among four possible answer choices. The system tabulates the answers in detail, or summarizes them in a graph.

Development started with brainstorming sessions to storyboard the application use cases, its interface, and its behavior. The storyboards were the main planning tool for the development — they provided a good compromise between our desire for an informal, lean process, and the need to design the application, and document and communicate the decisions among team members. The implementation was guided by the decisions made during storyboarding (figure 1).

The storyboards described the following workflow:

1. 1.

   Opening the class: the instructor selects the class identifier to open a session;

2. 2.

   Starting a question: to start polling the students, the teacher select the question number, for later identification (those numbers autoincrement, but can also be changed manually);

3. 3.

   Image capture: the camera activates and the instructor scans the entire class, capturing the answers;

4. 4.

   Results screen: the application shows each student answer; if there remain unidentified answers, the instructor can go back to the image capture stage, or input the answers manually;

5. 5.

   Chart screen: the application summarizes the result in a pie chart.

Initial design envisioned an offline module to manage classroom definitions: class IDs, and their enrolled students. During prototyping we managed that information manually, directly on the application internal data files.

During initial tests, we added two new features: a separate flow for roll-call — letting the teacher to do an initial roll-coll in the classroom to later better identify absent students from unrecognized codes; and an augmented reality feedback on the answers capture screen, to let the user identifying recognized cards (Figure 6).

At first, for the students cards, we chose QR Codes (ISO/IEC 18004) [21], which can store many characters, have error correction, and have location patterns that easily establish their orienation [22]. We developed a prototype mobile app in Android²² 2 http://www.android.com/, using the ZXing³³ 3 https://github.com/zxing/ open source library to generate and decode the QR Codes.

QR Codes would allow unique codes for each student across an institution, or even across an entire education system, since it can easily store dozens of digits. However, we found next to impossible to adapt them to our use case. QR Codes are optimized for recognition and decoding of a single code at close range, while we needed to find and decode dozens of codes across different distances from the camera, in a single photo. Although we obtained moderate success after several modifications in ZXing code, the decoding was still too slow to be used on a video stream, and had to employ still pictures. Worse yet: recognition accuracy — even on ideal situations of illumination and occlusion — was unacceptable.

Giving up on storage capability, a natural solution presented itself from the literature of motion capture and augmented/virtual reality: fiducial marks, which are engineered to be recognized in real-time in a video stream. We selected TopCodes[4], originally designed to create virtual objects for the Tern tangible programming environment [23]. We still had to modify the recognition/decoding code for our purposes (Section IV-D), but after such fine-tuning they proved extremely robust and very fast. On the other hand, TopCodes support only 99 unique identifiers, making possible to give unique ids for each student only in small to moderate-large classes. Using on large institutions is not impossible, but requires the additional burden of keeping tables to translate class IDs into institution IDs for each class.

We applied usability testing on a small subject sample (N = 11), employing direct observation, user and interaction recording, semi-structured interviews and a questionnaire. Although we have found relevant usability issues, we concluded that the proposed prototype was adequate as CRS. Usability tests are effective when compared to the field testing [24], and can be cost effective with a small number of subjects: Nielsen Normal Group⁴⁴ 4 How Many Test Users in a Usability Study - https://www.nngroup.com/articles/how-many-test-users/ points out 5 samples can provide an important usability insight. We detail the user-interaction/interview findings in Table II, and summarize the questionnaire findings in Table III.

We tested two use cases, covering typical usage:

First use case — starting the class: launching the app and performing a roll call. The teacher starts the paperclickers application on her mobile device, and then chooses the class to start. After the students arrive, she touches the roll call icon. She asks the students to raise their cards (in any orientation) and films the entire class. She touches the screen to advance to the result screen. The application shows the roll call result, if some student is present but has not been detected, the teacher can correct the status by tapping the corresponding icon on the device screen. The teacher touches the ”done” icon to finish the roll call procedure.

Second use case — asking a question: during class the teacher polls the students. The teacher wants to verify the students’ understanding of a given topic. He opens the paperclickers application and touches the ”Start” button. He chooses a number for the question. He enunciates a multiple-choice question, and waits for the students to commit to an answer; then he asks the students to lift their cardboards to indicate the chosen answer, using the letters printed on the back of the card as cue (the chosen letter must be upright). The teacher touches the ”Collect answer” button to advance to the scan screen. After capturing the answers, he touches the screen to advance to the detailed results screen. If many answers are missing, he can choose to go back to the scan screen, and then the application would merge the results from both scans. He can also touch the icons to manually change the answers. If the answers are correct, the teacher can touch the graph icon to advance to the summary results screen, that shows a pie chart. At this point, the teacher can touch the ”Try again” button to return to the scan screen or the ”New question” button to pick the number of the next question.

The prompts we gave to the test subjects were much less directive than the detailed scripts shown above. For example, we would ask the users ”please, start a roll-call” not ”touch the roll-call icon”, and would observe if the users were able to figure the interface by themselves. Only when the users were blocked for several minutes we would give a direct prompt.

Our experiments ended up reflecting the DECIDE framework, as described by Preece et al. [25], which establishes a checklist comprised of 6 steps to guide the evaluation preparation: “Determine the overall goals that the evaluation addresses; Explore the specific questions to be answered; Choose the evaluation paradigm and techniques to answer the questions; Identify the practical issues that must be addressed, such as selecting participants; Decide how to deal with the ethical issues; Evaluate, interpret, and present the data”.

Determine the goals:

• •

  The solution offers a convenient and consistent user interaction, satisfying the user;

• •

  The overall design allows the user to perform easily and efficiently the required tasks, suitable to the users’ real problem;

• •

  It is easy to learn how to use the solution.

Explore the questions:

• •

  Is the answers capturing screen, with the augmented reality codes visualization, legible and readily explainable to the user?

• •

  Are the detailed answers visualization and answers chart screens legible and readily explainable to the user?

• •

  Is the screen navigation throughout the application consistent and does it match user expectations?

Choose evaluation paradigm and techniques:

• •

  Direct observation and monitoring;

• •

  Recording of the user with a video camera;

• •

  Recording of the user interface with screencap;

• •

  Semi-structured interview after the experiments, asking about the overall experience and main difficulties;

• •

  Questionnaire after the experiments, exploring quantitative information on the user satisfaction and understanding about specific visual elements.

Identify practical issues:

• •

  The tests would be applied on volunteer participants older than 18-year old. We did not put any other restrictions, but since we announced the test in the University, we knew we would be mainly sampling graduate and undergraduate students. Although a broader audience would be desirable, we considered that a sample of students was aligned with at least part of the target users, providing thus already valuable feedback. Broader audiences could be addressed later, on a refined product.

• •

  We defined test scripts, to be used as instructions during the experiment, for both the applier and the test subjects.

• •

  The experiment procedure considered only one participant using the application; we fixed the cardboards on the backrests of the class’ seats, simulating the students answering performance during the class. A research team member would follow for support.

• •

  We defined a controlled environment, using two rooms at the University: the first for subjects using the app, and the second room for the semi-structured interview and filling the questionnaire.

Decide how to deal with the ethical issues:

Although the proposed tests corresponded to the simple indoor usage of a smartphone application, without repetitive actions or sensitive data collection, we identified and planned to mitigate any ethical concerns.

• •

  The privacy of user information gathered throughout the usability tests, the interview and questionnaire, also including the recorded personal image; we created an Informed Consent Form to clarify our privacy agreement.

• •

  Any discomfort the users might few performing the tests, related to shyness of begin recorded or interviewed, for example; we provided clear and formal recommendations about the absolute freedom the participants had to leave the experiment during any phase.

The experiment was approved by the Brazilian research ethics committee.

Evaluate, interpret and present the data:

• •

  The overall feedback was positive regarding the application usage and suitability to the proposed tasks: its main features were easily recognized and all the intended tasks successfully and easily performed by the majority of the testers;

• •

  All the testers were able to successfully scan the class for results — understanding the overall usability, including the augmented reality feedback — as well as to read and manipulate the detection results in the detailed answers screen;

• •

  We found two major usability issues about application convenience (roll call feature identification and initialization) and consistency (backward navigation throughout the application’s screens);

• •

  The user interaction recording provided the most valuable data, identifying the users’ actions along with users’ impressions, through touches or audible comments (also recorded); the post-test semi-structured interview provided important data to confirm the findings, adding overall impressions on the application proposal.

The recordings made clear the second screen was confusing and inconvenient: users were requested to perform a roll call — reachable by an icon on the upper left corner — but the main graphic elements were the ”question number definition” box and its controls, and the ”Collect answer” button (refer to figure 4). We can infer this confusion through the users reactions: some of them navigated backwards — either through the device operational system ”back” key or using the ”home” icon — and some of them even expressed audible surprise, giving the impression they were expecting something else.

We found consistency issues related to backwards navigation. Depending on the screen, backward navigation lead the user one, two or even three screens back. That was not only internally inconsistent, but also mismatched Android’s expected behavior. One egregious example was the navigation from the chart screen, which represents the end of the scanning process: as visible in figure 10, there were 4 different UI elements to navigate backwards, each one with a different behavior. Although those options gave users flexibility, they were very confusing.

During the semi-structured interview all the testers gave positive feedback on the application and usage scenario, but expressed a few concerns, like whether its usage might distract the students, might reduce intimacy between teachers and students distance, or might be unreliable on large classrooms.

The questionnaire data analysis (Table III) was important mainly to confirm the same issues detected through the user interaction and interview, providing quantitative data regarding users’ preferences on other aspects, like the fact most of the users disliked the forced landscape orientation of the app.

The user experiments prompted us to perform an additional development cycle before the public release: not only there were important usability issues to work, but the speed and reliability of TopCodes detection could be improved. On that new phase we again relied on storyboards to redesign and guide the implementation changes. The second cycle concluded with the release of paperclickers on the Google Play Store⁵⁵ 5 https://play.google.com/store/apps/details?id=com.paperclickers, and the publication of its source code under the GNU Public Licence v2⁶⁶ 6 https://github.com/learningtitans/paperclickers.

The usability tests taught us that many extra features were more confusing than helpful, thus we turned to the Minimum Viable Product approach [26], focusing on the essential: collecting and summarizing students’ responses. Keeping poll data for posterior analysis is still possible, but no longer encumbers the main workflow, if the teacher is interested in those extra features, she has to use the settings screen.

We removed the second screen entirely, along with its two features, the preliminary definitions (class and question) and roll call feature initialization. Although the users found those features interesting, they had trouble understanding and accessing them. We kept the main workflow as simple as possible for basic usage, and added a separate settings screen for advanced users, with options to add tags to the questions and to share the answers log, allowing those expert teachers to analyze individual students’ answers. Figures 2 until 12 contrast the main screens from the prototype versus from the released versions.

We also worked to improve some issues found on the TopCodes detection and decoding. TopCodes original use case is the tracking of a small number of fiducial points to create virtual objects for virtual/augmented reality. They were not optimized for our context of tracking dozens of codes in very uncontrolled and dynamic environments, keeping careful track of each code orientation. In the second version, we found and corrected the problems explained below.

Errors in detection and decoding: In TopCodes’ reference implementation, partially occluded codes are often decoded incorrectly (Figure 13). In the original contexts, this creates a short transient detection that has no consequence other than quickly flashing an object out of place, but in paperclickers, errors of detection or decoding may set the wrong answer for a student. Worse: partial occlusions are very common as the teacher scans a classroom full of students. To overcome that issue, we added a time-consistency constraint to the detection: only codes which are detected across several contiguous video frames are considered valid; the threshold of frames is set dynamically according to the duration of the take.

Too many code candidates: TopCodes’ reference implementation binarizes the image and then scans it horizontally, looking for the right transitions between black and white pixels, to find potential candidates for code locations. However, when the background has many vertical lines (e.g., curtains, blinds…), that results in a huge number of candidates, slowing the detection. To reduce that sensitiveness to the background, we added a vertical scan step, and kept only the candidate points found both by the horizontal and the vertical scans.

Sensitivity to hairline code defects: Although TopCodes are extremely robust to affine transformations (scale, rotations, moderate camera baseline changes, etc.), we found them very sensitive to hairline defects, i.e., situations where a single row or column of the code becomes entirely white or black after binarization. We found those defects would be very common if the codes were printed in less-than-perfect printers, or if the students ignored the admonition to not fold the cards. After considering several complex solutions, we attempted using morphological operations to seal those small gaps. We tested many alternatives, but a binary closing followed by a binary opening using a 3 by 3 pixels square element offered the best compromise between eliminating defects and preserving details. However, further tests showed that the best solution — both in terms of precision and speed — was simply to instruct the user to not film from too close!

We also improved the overall detection speed --- one of the user complains about the prototype --- fine-tuning the grayscale conversion and using the Android’s native image processing multi-core CPUs and GPUs usage⁷⁷ 7 Android Renderscript computation engine framework - https://developer.android.com/guide/topics/renderscript. After those changes, the TopCodes detection and decoding functionality reached the scan cycle performance of about 2 frames per second, including all the changes described above, running on 2017 mid-tier Android devices⁸⁸ 8 1.5 GHz Cortex-A53 CPU, 1920x1080 pixels image.