Log in / Register
Home arrow Computer Science arrow Augmented and Virtual Reality
< Prev   CONTENTS   Next >

4.2 Experimental Method

To test the whole pipeline of the system, comprehensive experiments were performed. In these experiments, subjects, not expert in TSL (blindfolded and with ears covered with headphones), are required to recognize the signs performed by the robotic hand, while a proficient in SL performs the signs in front of the input device in another room of the same building.

Each test subject is visually trained for five minutes on the subset of chosen signs with the proficient in LIS (the robotic hand is not used in this phase). After this first phase, the subject is blindfolded and his/her ears are covered; then, the subject is introduced in the room where there is the robotic hand. Note that the subject does not see the robot hand when training.

The message is sent to the robotic hand through a net (local network in the experiments). The results collected so far show that most of the times signs are correctly sent over the network and successively recognized in few seconds touching the robotic hand, even by non expert people.

In these experiments, we use only a subset of the LIS alphabet, consisting of characters S, U, V, W, F, because these are the signs recognized with more

Fig. 7. Average percentage classification accuracy for different hand gestures, the sign is on the x-axes; please note that signs P and S are repeated two times because LIS admits two ways of performing those signs

accuracy and are also reproducible by the robotic hand, which is a first prototype with 6 DoF and cannot reproduce the whole possible static signs from LIS.

In particular, the experiments show that the system is able to work without errors for hours, and pose the basis for a more intensive session with deaf-blind subjects.

A test application collects data about the experiments. An example of recorded data is provided in Table 2. The test application randomly produce a List of 40 Signs (LoS) and the signer is asked to perform these signs, one by one, in front of the acquisition device. Moreover, it records the list of Signs Recognized by the Input module (SRI).

Recognized signs are transmitted to the controller of the hand by the transmission module. The robotic hand performs the sign and holds it for 5 seconds, and then it comes back to a rest position (open hand) and waits for the next sign. The subject has to recognize the sign by touching the robotic hand (using one or both hands), and then pronouncing the sign he/she understands. An experimenter records the Signs Performed by the Hand (SPH) and the relative Signs Recognized by the Subject (SRS). Each experiment lasts about 20 minutes

(a) Ground truth (b) Labelling (c) Ground truth (d) Labelling

Fig. 8. Classification and ground-truth for two poses of the LIS alphabet. (a,b) V letter; (c,d) T letter. The classification accuracy is, respectively, 81% and 46%.

Fig. 9. Subject (blindfolded and with headphones) interacting with the robotic hand during the test session (V letter is performed in the picture)

per subject. Fig. 9 shows a subject during a test, while Fig. 10 illustrates the pipeline of the described experimental apparatus.

The experiments consist in repeating the procedure previously described for 10 subjects, for total amount of 400 signs produced. At the end of the experiments, four list of signs (LoS, SRI, SPH and SRS) are available. Hence, the

recorded lists have been compared among each other and the results summarized in Table 3.

LoS VS SRI refers to the percentage of signs correctly recognized by the input

module. This comparison to evaluate the effectiveness of recognition module. Here errors are due to finger occlusions, that sometimes deceive the recognition algorithm, but mainly to mistakes of signer. An example of erratic recognition is shown in column 3 of Table 2.

SRI VS SPH refers to the percentage of signs correctly sent to the hand. This

comparison to evaluate the effectiveness of transmission module and the robotic hand module. No errors happened in this stage during the experiments.

Table 2. Example of recorded data during real experiments. Here are reported the List of Sign (LoS) to be produced, the list of Signs Recognized by the Input module (SRI), the list of Signs Performed by the Hand (SPH) and the list of Signs Recognized by the Subject (SRS).

Fig. 10. Pipeline of the experimental apparatus

SPH VS SRS refers to the percentage of signs correctly recognized by the subjects. More of the errors in this phase can be ascribed to the subjects' lack of experience in Tactile LIS. In particular, the sign W is often confused with the sign F, since in both cases three fingers are opened.

Finally, LoS VS SRS measures the efficiency of the whole experimental appa-

ratus. Here the percentage of success synthesizes the other percentages.

As shown in Table 3, overall success rate is 91%. Such result proves the general robustness of our system. In addition, we are confident that success rate will be higher in communication with real deaf-blind persons, that would surely make less mistakes in both performing and recognizing signs and letters from their SL alphabet.

Table 3. Reliability performances of the proposed system

Found a mistake? Please highlight the word and press Shift + Enter  
< Prev   CONTENTS   Next >
Business & Finance
Computer Science
Language & Literature
Political science