Sunday 8 June 2014


Web Impact Metrics for Research Assessment



University of Wolverhampton, Statistical Cybermetrics




Overview: Web metrics are being increasingly explored in the assessment research impact. Hyperlinks, web citations, and URL citations can today be systematically compared with conventional measures (e.g., Web of Science citation counts). Formal citations are also being extracted from web databases and digital libraries by CiteSeer, Google Scholar, and from the huge digitized database of Google Books. These may prove informative as alternative and supplementary citation impact metrics, especially in the social sciences, arts and humanities, where traditional citation indexes are not available or have insufficient coverage. New web impact metrics come from citations in online syllabi and course reading lists, which reflect the educational impact of research, and from download counts of academic publications, which reflect reading and usage. Social impact metrics or Altmetrics — including social bookmarks, tweets, online reading of scientific publications, and viewings of online academic videos — are also emerging. Web impact metrics need to be used cautiously in research evaluation, however, because they still suffer from a generic lack of quality control compared with traditional citation metrics.
READINGS:
    Kousha, K. & Thelwall, M. (2014). Web Impact Metrics for Research Assessment. In: B. Cronin & C.R. Sugimoto, (Eds), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, MIT Press.
    Thelwall, M., Vaughan, L., & Bjorneborn, L. (2005). 
WebometricsARIST39(1), 81-135. 
    Kousha, K., & Thelwall, M. (2007). 
Google Scholar citations and Google Web/URL citations: A multidiscipline exploratory analysisJournal of the American Society for Information Science and Technology58(7), 1055-1065.


28 comments:

  1. What sentiment analysis service/API did you use for the webometrics comment analysis you showed near the beginning of your talk?

    ReplyDelete
    Replies
    1. Thanks, SentiStrength please see http://sentistrength.wlv.ac.uk/ - see also
      Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detection for the social Web, Journal of the American Society for Information Science and Technology, 63(1), 163-173.
      Thelwall, M., Buckley, K., Paltoglou, G. Cai, D., & Kappas, A. (2010).Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12), 2544–2558.

      Delete
  2. My question for Professor Kousha is: What do you think are the most significant difficulties in implementing web impact metrics with regard to scholarly citations?

    ReplyDelete
    Replies
    1. I think the challenge is the definition of impact of a scholarly work. When you use the term "impact" most people think about citation impact. Ok that's very valuable metric for assessing research impact and we know this for many years. However, one scholarly work may have other values which can not be measured by citation counting. For artistic work such as fine arts, music and dance we do not usually think about citation impact, but cultural or social impact. I think web impact can potentially help assessing these types of values. So, we should change this views. Please see also ACUMEN project http://research-acumen.eu/ as an effort to do so in EC.

      Delete
    2. Thank you for the response and for the reference!

      Delete
  3. Is manual Google search really the best way to discover a source's impact (like you showed, searching Google Scholar/ Google Books/ Mendeley)? It seems to me you could write a computer program to search a list of resources/search engines and scrape whatever data you wanted.

    What do you do with this impact data once you have gathered it? Is your interest just in capturing the reach of a source or do you look into the content, (i.e. what makes one source more impactful than another)?

    ReplyDelete
    Replies
    1. Sounds like a project for you, Rachel!

      Delete
    2. Thanks. No, I don't think so. However, for a small scale project and when API is not available manual searching seems to be only solution. For Google Books you can use Webometric Analyst (http://lexiurl.wlv.ac.uk, see its “Books” tab), to locate GB citations automatically and to remove false matches (e.g., advertisements, book reviews and bibliographies) by feeding structured queries for publications (e.g., papers or books). I didn't have time to explain how to do it.
      Some databases like Mendeley support API so we can go for it, but this is not a case for many databases.
      We usually compare them against other indicators such as citation counts or peer review to see if there is a significant correlations between two or not. A high degree of association may indicate that one well-known metric (e.g., citations) influences the other (e.g., number of Mendeley bookmarks) and at least statistically they have similarities, although this does not imply causation. Content analysis and qualitative research should be done to validate any significant correlation in the next step.

      Delete
  4. Is there anybody doing webometrics based on implicit citations (reuse of of concepts as defined by lexical patterns)?

    ReplyDelete
    Replies
    1. I am not quiet sure I get you question, Sorry! Could you please explain more about your idea?

      Delete
    2. It's all right – it's my fault.

      You can track concepts lexically in texts—sometimes, it's as simple as a single word (e.g. "stigmergy"), sometimes it's patterns of word apparitions. Anyways, it's something we do in our lab, and one of my colleague uses it to track concept diffusion in journalistic corpuses and make (simplistic) graphs of citations.

      So, I was wondering if it was being applied in scientometrics/webometrics.

      Delete
    3. I haven't seen such a study in webometrics, However, I am not sure in scientometrics at all, You may search Scopus or WoS and then limit your keywords to "Scientomerics" or "Journal of the American Society for Information Science and Technology" (name of journals) as source title or publication name. . .

      Delete
  5. It pleases me to see that ratings of an article's impact is beginning to extend beyond the impact factor of the journal it is published in. Now some academic articles include metrics such as downloads, views, citations, and shares.

    However, these metrics don't inform the potential reader about the quality of the research. This responsibility is given to a few peer-reviewers and is then finished with. Perhaps a more crowd-sourced opinion on the quality of the work would be helpful. A system that builds of the like/dislike option on youTube and facebook could prove useful to researchers.

    ReplyDelete
    Replies
    1. http://j.mp/peersupplement

      Harnad, S. (1995) Interactive Cognition: Exploring the Potential of Electronic Quote/Commenting. In: B. Gorayska & J.L. Mey (Eds.) Cognitive Technology: In Search of a Humane Interface. Elsevier. Pp. 397-414. http://cogprints.org/1599/

      Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. http://cogprints.org/1694/

      Delete
    2. Sometimes, even the quality of research can not be measured through citation analysis, especially in arts and humanities. Please see also http://books.google.ca/books?id=D9SaJ6awy4gC&printsec=frontcover&dq=mode+citation+analysis+in+research+evaluation&hl=en&sa=X&ei=LY2-U8TYE8HisASfj4C4Bg&redir_esc=y#v=onepage&q&f=false
      See chapter 15, what do references and citations measure?

      Delete
  6. The use of different metrics to evaluate a scholar works show us more information than only the citations, but using twitter or amazon user evaluation ( for me) is sort of subjective appreciation, because we don’t know the background of who made this appreciation. In this case do we need to put a ‘Tags’ in order to evaluate objectively those metrics !

    ReplyDelete
    Replies
    1. Thanks very much for very good question. We have less evidence about both Amazon and twitter. We need to investigate more. For twitter for instance see Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4).

      Holmberg, K. & Thelwall, M. (in press). Disciplinary differences in Twitter scholarly communication, Scientometrics.[

      Thelwall, M., Haustein, S., Larivière, V. & Sugimoto, C. (2013). Do altmetrics work? Twitter and ten other candidates. PLOS ONE, 8(5), e64841. doi:10.1371/journal.pone.0064841

      For Amazon we have done a study (not published yet), in the next step we need to separate out reviews by those who have bought books (Amazon signifies this), those reviews with affiliation, size of reviews, and perhaps put more weights to reviews with many helpful comments than other comments.

      Delete
  7. It would be interesting to look whether citation relations tend to form clusters, which are aligned with different schools of thought/intellectual traditions.

    ReplyDelete
    Replies
    1. http://j.mp/citationclusters

      Delete
    2. Yeah, it occurred to me that scholarly articles that have more everyday practicality might show up more often in social media or summarized in news media (as awful as that usually is). Conversely, scholarly articles that have profound implications for researchers of a specific discipline, like methodological papers, may show up frequently within academic sources from a particular discipline. It could be beneficial to somehow weight these measures of impact based on the size of the readership such that, papers coming out of niche fields of study, while of high quality and extremely relevant to certain groups of people, do not score low in impact simply as a function of their small readership.

      Delete
  8. According to me. We must take be careful to measure some sentences. Because some people use some negative words into the sentences in order to reinforce positive comment. My questions for Professor Kousha are: 1.- If he takes into account those phrases in his metrics or measures? 2.- And if it is positive for the question 1 how he evaluate these types of positive phrases when people use negative words in the comments positive direction.

    ReplyDelete
    Replies
    1. For short-medium social text we may be able to detect negative /positive views on an even or a wok.
      Thelwall, M., Buckley, K., & Paltoglou, G. (2012). Sentiment strength detection for the social Web, Journal of the American Society for Information Science and Technology, 63(1), 163-173.
      Thelwall, M., Buckley, K., Paltoglou, G. Cai, D., & Kappas, A. (2010).Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12), 2544–2558.

      Delete
  9. Quelles sont les limites de la webometric? Peut-elle analyser tous les médias sociaux? A quoi sert-elle? Je comprends aisément son utilité au sujet de l’érudition, toutefois j’ai de la difficulté à concevoir son but au sujet des médias sociaux, tels que les discussions sur youtube. Est-ce que l’objectif serait celui de comprendre l’intérêt des gens qui sont sur le web? Si c’est le cas n’est ce pas se limiter à un type particulier de personne? Pourquoi faire une webometric des médias sociaux étant donné qu’une analyse de ceux-ci n’est pas utile à la participation à la vie sociale, ce que je pense être un de leurs objectifs.

    ReplyDelete
    Replies
    1. Translation:

      What's the purpose of web metrics for social media? Their purpose is clear in science.

      Delete
    2. I think we may use web metrics (like/dislike, sentiments, download, views, comments) to explore trends of using them or possibly to predict future popularity of a web content. We may also compare two similar social media such as Mendeley vs. CiteUlike for the same web object (e.g., paper).

      Delete
  10. Dear Kayvan, it was a very useful presentation ! I have one "simple" question :) : How do you think that it’s possible to predict the probability of a scientific article to be cited or not in Wikipedia?

    ReplyDelete
  11. Thanks, it is not so simple question, I'm afraid. For many years this has been challenging issue in scientometric/webometric. However, several studies suggest that some web indicators can potentially estimate future citation. For instance, download counts of papers, Mendeley bookmarks (online reference manger), and tweets may all help to capture earlier citations (since formal citations need more time).
    Here are some readings
    Bollen, J., & Van De Sompel, H. (2008). Usage impact factor: The effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136-149. DOI: 10.1002/asi.20746

    Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060-1072. DOI: 10.1002/asi.20373

    Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4). DOI: 10.2196/jmir.2012

    Mohammadi, E. & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, 65(8), 1627-1638.

    This recently published book is also very useful I think:
    Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact , MIT Press. ISBN: 978-0262525510
    http://www.amazon.com/Beyond-Bibliometrics-Harnessing-Multidimensional-Indicators/dp/0262525518/

    ReplyDelete
  12. Using metrics to rate the merit of academic articles will modify what authors write, regardless of the metric used. Awarding number of publications can cause authors to publish more short articles with the same total content. Awarding the impact factor of the journal cause authors to write on timely topics that will impress reviewers and promise a wide readership. Awarding citation count, downloads, or 'likes' will also have implications.

    One big issue I see is the lack of merit awarded for attempting to replicate a study. Is there any metric that might encourage researchers to not overstate their findings while also propelling replication attempts?

    ReplyDelete