A Preliminary User Experience Evaluation Of MISP Work In Progress Borce Stojkovski

morning for my side my name is takovski and it's a pleasure to present to you the preliminary work that we are working on mess with respect to disability and user experience this is Joint work with my supervisor Gabriela lancini and we are from the center for security reliability trust at the University of Luxembourg so just to give you a brief outline I'm going to introduce a bit the motivation about these projects the methodology and preliminary results that we have and what are the next steps that we do in this direction so I guess I do not need to explain much to this community how Mist Works what type of use it has or how the information is shared or could be shared among the instances and communities but it is important to know that there are different types of users that come from different types of organizations and they have different purposes for which they use the Miss platform so we're dealing with a very technically Advanced information sharing platform here and as I said it has a diverse set of stakeholders that are involved in this platform they all have distinctive needs and objectives and probably some of you have experienced but as communicated by misp often there are sharing difficulties so basically that are not so much on the technical level but on the social level in terms of certain restrictions why people share information or do not share information how do they consume the information do they trust the information or not so there are a lot of issues that hover let's say the technical layers of this platform and as a research suggests as well the importance of user experience and usability is let's say these aspects are Paramount for the successful operation of an information sharing platform and the amount of information that is consumed and contributed towards it so we set out to investigate basically how do misp users evaluate the usability and user experience aspects of the platform and more specifically how do they perceive both the pragmatic and hedonic qualities of the system and I will explain a bit more in detail now so we set out to investigate this as a first step via a quantitative method called the user experience questionnaire which is basically a combination of six scales with 26 items that asks people to rate specific aspects from two imposing views such as for example do you find the system annoying or enjoyable so they have to pick a value between these two values so there are 26 questions of this form that try to answer or get some information about the attractiveness perspicuity or basically how difficult it is to learn the platform or the system the product how it is efficient what is the dependability the stimulation and Novelty of this platform so just to give you an interdependence of these scales basically the attractiveness is a pure balanced dimension whereas the efficiency dependability and persecuity are more on the goal directed aspects of the platform or the pragmatic qualities and then on the hedonic sides we have the stimulation and Novelty basically how do people find these aspects as well which usually in the past have not been considered via usability studies so the second part was to investigate a bit more who are the users that use this platform so basically the demographics the background profile education Etc and some of the questions were what is the company that you come from how long have you been using Miss Etc and the last part is one more quantitative scale which is called the system usability scale and I presume maybe some of you might have come across it because it is a more known or standardized scale that has existed for more years than the ueq however with any like with all of the other scales there are certain limitations or things that we need to take into account when deploying them and specifically in the sus it does not look into any of the user experience or the hedonic aspects so so this is for example what it what questions from the sus look like so at the training session that was carried out in March early this year we had discussed with the team of circle to to try to get the preliminary feedback from the participants of this training session on a voluntary basis to tell us how do they perceive the the Mis platform Etc so there were 50 people that were the training session um 25 respondents 25 did not respond to the questionnaire of those 25 24 were male one was a female um they had 20 had a technical or engineering background five were non-technical in terms of Education 19 or more than 75 percent no mistaken uh had they um bachelor's degree or or higher um when it comes to the age span we can see that the largest group of people were in the range of 26 to 35. It's relatively a young age group and when it comes to the prior experience with misp the majority of them had used the system for less than six months out of which from one to six months was the largest group were basically eight people when we ask them how often they use misp the people that attended this training session said that most of them said that it is between once and three times per week only a small number used it on a daily basis and the next smaller group 17 was even less than one week when it comes to the industry sector the participants came from these let's say Industries the largest was banking I don't know if it has to do the fact that we did the training here in Luxembourg but that was the case in this training session and when it comes to user profiles now this was interesting because before the server we thought that people have a clear let's say role but many of them actually see themselves as software analysts in addition to another role so this is how each of the people have basically how many have indicated and see themselves as software analyst in addition to maybe some some of the other roles so it is interesting to see that there are certain users that use the platform for multiple purposes actually okay so just to summarize so 48 were in this pin from 26 to 35 the majority identify themselves as security analysts they came from banking the majority of them and while it was the first training session for all of them over 50 of them had used the training materials or the virtual machine before coming to the training session which is one of the let's say materials that uh cert Circle provides okay so these are the results of the questionnaires and I presume this might not necessarily mean much so I will try to explain it within a larger context so we'll start with the user user experience questionnaire and basically if we look at the if we compare those means against Benchmark values we can see that the Mist platform is positively evaluated on all aspects except for perspicuity or basically those users found it quite complex and difficult to use the platform on all the others it is very good and especially on the hedonic side which is actually excellent or above average so these Benchmark values are obtained from a data set that has been carried out or that compromised comprises 401 studies with 18 more than 18 000 people so basically the average estimates of different types of platforms using this scale give us certain Benchmark values against which we can compare other systems and of course we need to take this with a bit of a consideration but I will come to the let's see the limitations in a minute so when it comes to the hedonic aspects so basically how people find the platform in terms of novelty motivation Etc it is above average it is in the best Android then percent best results when it comes to how easy it is to learn the system that is let's say the aspect that misp is worst on 25 and when it comes to the sus so basically the third and the second scale that I uh showed you before the average score is 64.5 again this is a benchmark value that has been obtained by analyzing a large number of different products Etc where 68 is considered to be the average value so basically 64.5 is considered to be below average which again probably might be due to the fact of the difficulty to to learn the platform Etc so what we need to mention here is that these findings are based on a very small sample so basically based on 25 responses so we cannot really conclude anything strongly however it can give us an early indication or direction in which we can try to explore further and that is definitely what we would like to do so using mixed methods and here I again mentioned these quantitative surveys such as the euq but also using interviews and other methods we would like to understand better what are the biggest usability user experience issues because of course these surveys can only point us to Direction but it does not they do not really explain why people answer these questions like that or why do they find the platform difficult to learn Etc so we would like to complement within the next period quantitative methods using some interviews so I'll be here throughout the day if any of you have any feedback or comments I would be more than happy to hear them but of course it is on a voluntary basis so you're more than welcome to also provide maybe some other feedback via other ways through the Gip or Etc so just to mention as a the final step I invite you all to take part in the user experience survey which we have ported into an online form and this will be basically provided I think also with the latest training materials and the link will be provided from from the circle website but from my site that is all for now and I thank you for the intention and if you have any questions feel free to let me know either in person or during this Podium thank you thank you yes so in relation to the survey how long do you plan to keep it open and wait until you get enough results just to have a rough ID so I think we will perform analysis on a continuous basis but I would say we would have it at least for six months but even more uh hopefully it will be something that we'll continue on and we will get continuous feedback in terms of also to be able to compare how the introduction of new features might impact the overall experience Etc so it sounds like some stuff you've done here are also like related to the threat Intel community so maybe it'd be interesting to compare with how it is try and tell wise or maybe other similar software have you considered doing that with similar software thank you that's an interesting question um so basically if we have more data we will also be able to compare how different people evaluate the platform depending on their background depending on their backgrounds uh their role Etc but as you say it would also be good to compare different types of products using these methodologies now we would probably need to find a large enough sample that uses both of these systems so that we can compare and see how they evaluate those systems or if it's if it's an even bigger sample then maybe not the same people need to necessarily evaluate both systems but we need to make sure that the sample of the users that answer these questions is representative so that we do not come to some false conclusions but I think that's something definitely that could be done thanks very much for this uh was your instrument clearly focused on what I call the UI of the tool because it occurs to me that there may be particularly in some of the categories of respondent some conceptual issues like what's a Galaxy and that might be a separate set of you know user experience issues then why is this button disappearing at this time kind of thing I'm just wondering was this both either I wonder if there's a connection between I don't understand what you mean by matrix and I'm not having a good experience with the tool so the questions that are in the scales they're independent from any platform they can be applied as such across different so there is no ambiguity when it comes to the specifics of the platforms such as as you mentioned specific features etc those are probably things that could be investigated furthermore in an interview but I would not say that any bias was introduced via the terminology that is used in the platform and what the questions ask because these are basically yeah applicable throughout without any consideration for the specific features okay thank you [Applause]