Archive for the ‘E-learning research’ Category

Quality discourse: flourish or whither?

Sunday, November 25th, 2007

George Siemens post ‘blog malaise’ raises some really interesting questions about the ‘worth of knowledge’ in today’s context. 

The fact is, today’s information currency finds its value in connections. And Google is the banker. If you want society to know you exist, you need to be found by search engines.    

If you want something chances are the first thing you will do is google it. He goes on:

What happens when brilliant researchers conduct brilliant research but publish it in closed journals? The researcher or the research may likely not reach the awareness of individuals who find information through search engines (though, with Google Scholar, this will likely change somewhat). Voices of authority (as seen from the perspective of an average information seeker) are determined by how accessible and how prominently connected they are.          

Which suggests that there is a shift in what counts in terms of authority and how that’s determined. I think this raises profound issues for us as academics. I am very aware of this from my own perspective through a number of things recently. Firstly as we finalise our RAE submission what counts there is the weighty peer reviewed journal article and each individual’s indicators of esteem. But this doesn’t capture the rich ways in which many academics are now communicating - through blogging and by sharing presentations on slideshare. Which is the more valid representation of academic knowledge? Secondly I’ve written up some of our learning design research in a couple of book chapters recently and now feel slightly frustrated that I can’t make that material available electronically because it goes against the publisher’s copyright. Thirdly I am aware that using social software (such as writing blogs, sharing references via social bookmarking, or making presentations available on slideshare) exposes an individual’s work to a far wider audience than would be possible via traditional means. All this seems well and good, providing a vibrant environment for sharing of ideas. However it seems that we are currently in a transition period. A lot of the more educationally focussed researchers I admire and follow are currently not in the blogosphere, which means there is a divide between 1) traditional academic communication and 2) social networking academic communication. Will we eventually reach a situation where the rest of academia see the benefit and importance of social networking tools and switch to using them as a standard means of communicating their ideas or will there always be those who favour the traditional methods? If the later happens will a lot of good intellectual thinking tied up in subscription-based journals or books get lost in the large noise of easily accessible “blogspeak”?  

The academic perspective

Saturday, November 10th, 2007

The student experience? What about the academic perspective? Following on from Michael Wesch’s ‘A vision of students today’, now read ‘A Vision of professors today’. Alarmingly realistic!!!

In an RAE quantum fix

Friday, November 9th, 2007

Schodinger's catAs I sit here fielding last minute panic enquiries from folk leading our CREET RAE submission about my stuff, I really aligned with Alan Cann’s post Schrodinger’s RAE. Partly because I was a fan of Schodinger’s work in the days when I was a Chemist and understood all that stufff, and partly because it hits the nail on the head, see my post for what its worth. Heisenberg’s uncertainty principle comes to mind come to think of it… Image from flickr

I think it is also worth saying a collective thanks to all those who are working around the clock to get all our RAE submissions in on time - a huge, huge and usually thankless job.

The paper vs. blog argument…

Monday, October 29th, 2007

My post ‘the nature of academic discourse’ seems to have hit something of a nerve, viagra which I thought was worth summarising here. Not surprisingly the issue of traditional academic output (such as published papers or books) vs. blogging comes to the fore. Sarah Stewart and Leigh Blackall challenge the current predominant norm of not counting blogging, arguing that it has an in-built quality assurance mechanism – through peer commenting and referencing and through ratings like technorati. Karyn Romeis points out that blogs report on ‘what’s happening now’, whereas peer reviewed papers are ‘old news’ because of the lead in time for publication. She thinks this time lag is particularly difficult for those working at the cutting edge. She also agrees with Sarah and Karyn:

Considering the readership of the blogs I would be likely to cite, I would contend that the level of expertise represented and the willingness to refute, rebut, challenge, defend, revisit, question etc. etc is far more rigorous than the peer review process. Sadly the assessing organisation does not agree.

David McQuillan celebrates the ‘stream of consciousness’ nature of blogging, suggesting that it is a valuable ‘route to publication’ and that by making thoughts public as they happen they are there and accessible for others to review and provide their perspective on. However Martin Oliver (a friend and someone I totally respect academically) suggests that there is potentially an information overload issue with blogs:

Please, don’t condemn me to having to wade through pages of peoples’ blogs in order to find the one or two good ideas in there!

He argues in favour of the traditional method, which he describes as ‘slow publication’ (reminds me of the ‘slow food’ movement and also Peter Goodyear’s call for more ‘slow learning’). Martin says:

It [blogging] has its place, but so does the discipline of shaping ideas in a format that can take a year or more to come to fruition. Distance brings its own perspective, and can help discern what’s of lasting value, rather than momentary excitement.

Which I do think is an important point.

My own view? Actually I think I agree pretty much with all these comments. However we need to be careful – I don’t think a direct comparison of journal papers and blogs is appropriate; people blog for a whole range of reasons not just for academic recognition and institutional ‘performance ticking’. I think what we are seeing is a confused transition, whilst we try and work out the co-evolution of tool use and our own working practice (both as individuals and as a society). Similar arguments are being raised of course about email (I haven’t dared open my email yet this morning having not looked at it since Friday…) but I think Patrick McAndrew pretty much sums it up and comes up with some nice lateral thinking on the issue of email overload!!

The nature of academic discourse

Saturday, October 20th, 2007

Doing this blog has really got me thinking about the nature of academic discourse. It particularly hit me yesterday when I was checking the proofs of a paper that will be coming out in Computers and Education soon on our LXP learning experience work. Whilst reading through the paper a number of things came to mind. The paper took a particular stance on the work – weaved a specific narrative – which is of course what papers are supposed to do. It grounded the work in the wider literature and fore grounded certain aspects of the data to support the chosen narrative. It presented a coherent story (well at least I hope it did!). This was in stark contrast to the reality of the research process. In my mind I could remember the messiness of the real research process, the various blind alleys we went down as a team, the panic at different points (will the technology work? will we be able to recruit any students to the study? will the data generate anything useful in terms of our research questions?). Another ‘view’ on the LXP project is through various conference presentations – these also weave a particular story, for example for a lifelong learning conference I considered the implications for lifelong learning, whereas at EUROCALL 2007 I concentrated on the findings from the Language learners. So although presentations, like papers, weave a narrative, because the medium is different (i.e. a slide presentation with pictures, audio and video alongside a verbal presentation as opposed to a textual paper) the message is different too. I have also blogged about aspects of the work and blogging seems to offer an alternative style of voice, one which is much more reflective and ‘of the moment’. So the function and nature of the three media seems to be:

  • Academic paper: reporting of findings against a particular narrative, grounded in the literature and related work; style – formal, academic-speak
  • Conference presentation: awareness raising of the work, posing questions and issues about the work, style – entertaining, visual, informal
  • Blogging – snippets of the work, reflecting on particular issues, style – short, informal, reflective
  • Which of these is a true representation of the research? Which is real academic discourse? In the old simple world, academics were recognised and rated primarily by their textual output – ‘the seminal text’, ‘the paper published in the best journal in the field’. And indeed that of course is still the primary mechanism for the Research Assessment Exercise (don’t get me started on that one again). But now increasingly the blogosphere is offering an alternative style of academic discourse, which you could argue is potentially a counter-culture to mainstream academic publishing. It has its own federated peer-reviewing mechanisms, such as cross-referencing between blogs and indicators of esteem such as the technorati authority. Increasingly academics are taking note of this new communication space – however one could argue that the uptake is sadly slower than it should be, I can’t believe it has taken me this long to recognise the value of blogging. In welcoming me to the blogosphere George Siemens wrote

    It’s great to see research-focused academics entering the blog space. We need a edublog ecology which runs the full breadth from practical application to theory and research.

    I wholeheartedly agree with him and hope that in time, more researchers take the plunge and communicate their findings and thinkings through blogs as well as via papers and presentations.

    Coming back to the question of which represents academic discourse – to my mind it’s all three – in different ways writing a paper, giving a presentation and blogging all help me to formulate and take forward my thinking on a particular topic, a means of meaning making and transformation of the raw ‘data’ to new understandings – surely that’s one of the cornerstones of what being an academic means?

    For what it’s worth

    Saturday, October 6th, 2007

    Eric Duval raises some interesting issues about how work is cited, and asks us to consider the following set of questions:
    • which of your papers has been cited most often?
    • who has cited you most often?
    • which papers cited a particular publication of yours?
    • whether more and more or less and less people are citing you over the years?
    • whose citing behavior is close to yours?
    • which conference or journal contains most citations of your papers?
    • which conference or journal contains you cite most often?

    Steven Harnard asks similar questions; How much is my research read, used, cited, and built-upon, in further research and applications? How much of a difference does it make that I did it at all? He goes on to argue for open and digitally accessible research outputs: “what is needed is that all university research output should be accessible and hence assessable online — and not only the references cited but the full text.

    Eric’s post got me thinking about the broader question of ‘what does it mean to be an academic nowadays?’, ‘What counts?’, or more controversially ‘Who’s good and who isn’t?’ Of course in the UK at the moment these kinds of questions are uppermost in the minds of those involved in research as the next Research Assessment Exercise (RAE) is nearly upon us. In a recent editorial for ALT-J I reflected on these and other issues and in particular looked at what the RAE offers us in terms of positioning our own research discipline and how it might help us to answer the question – what constitutes good e-learning research?

    Extract from ALT-J… An overview of the RAE

    The RAE is intended to provide a periodic review of the quality of research in Higher Education institutions and is important not only because it essentially results in a ‘research league table’ but also because significant funding is attached to the process for those areas deemed to be research excellent. A newspaper article last year (Guardian Unlimited 2006) highlights the central role that the RAE has played in the direction and focus of UK research activities and associated culture:

    “Over two decades the RAE has become an obsession for British academics and the ratings - from one to five-star - have made or broken the reputations of university departments. On the basis of the RAE judgments made at roughly five-year intervals depend billions of pounds of funding from the Higher Education Funding Council for England and its equivalents in Scotland, Wales and Northern Ireland.”

    The exercise is designed to be peer reviewed by a panel of experts drawn from the relevant research community. There are 67 units of assessment (UoE). Each academic submitted is judged on a number of criteria. The most important of these is the four publications (journal articles, books, etc.) they choose to submit as evidence of their academic standing published during the review period. Gradings in the new system range from 0 - 4 (table one). In addition each person submits an ‘indicator of esteem’ which includes information on their success in research funding, involvement as referees or journal editors, invited talks and keynotes, or other recognised evidence of academic worth and ‘international standing’.

  • Four star Quality that is world-leading in terms of originality, significance and rigour.
  • Three star Quality that is internationally excellent in terms of originality, significance and rigour but which nonetheless falls short of the highest standards of excellence.
  • Two star Quality that is recognised internationally in terms of originality, significance and rigour.
  • One star Quality that is recognised nationally in terms of originality, significance and rigour.
  • Unclassified Quality that falls below the standard of nationally recognized work. Or work which does not meet the published definition of research for the purposes of this assessment.
  • Taken from ‘RAE 2008: Guidelines on submissions’, June 2005

    Criticisms of the exercise

    Despite the fact that the exercise is peer reviewed there have been significant concerns voiced over its validity. See for example a recent article in the Independent. Firstly, many argue that the quality across the panels varies significantly, with wide disputes and differences in what is deemed ‘high-quality research’. Others feel aggrieved that money-rich disciplines in their opinion fair much better; the new post-92 universities make similar arguments and it is true that they have faired less well overall. And it is widely known that one of the purposes of the introduction of the exercise was to make research funding more selective, with money biased towards the university elite. Many also argue that the over concentration on the RAE and its importance has further deepened the divide between research and teaching, with teaching now more than ever suffering as the poor sister. For individuals in research-intensive institutions deciding where to concentrate their efforts is simple; research wins every time… if you want to get promoted that is. The Guardian ran an interesting piece recently, discussing a publication by the British Academy defending the value of peer view

    Lessons from the RAE

    This is the third RAE exercise I will have been involved with. The first was in my original academic area, Chemistry. In a sense submission then was fairly straightforward for me – I published in the standard recognised journals for my area, I was lucky enough to be collaborating with a good range of internationally recognised researchers. When I moved into the area of e-learning things became less clear; as a relatively young research area should I be publishing in new e-learning focused journals or more main-stream, well established education journals? What should the balance be in terms of standard empirically based submissions and more risking, but perhaps more innovative approaches?

    I have been involved in helping with the RAE mock exercises at my current institution (the Open University) and my previous institution, Southampton University, both in terms of developing research group narratives and peer reviewing exercises. It has been interesting and educational but reiterates for me the extremely subjective nature of the process. Here are a list of some of my thoughts when reading and re-reading articles. I know from talking to other colleagues involved in the process that many others echo similar concerns.

    1. Below is a list of some of potential ‘indicators of worth’ for a paper:

  • It covers an important and topical area
  • It is something which is likely to be cited a lot by others
  • It is a key positional paper or review which gives a definition of an area
  • It provides a critically reflective piece which provides new insight/ways of thinking
  • It is something which will have impact – on policy makers or practitioners
  • It provides the development of a new theory, framework or model
  • It includes good solid empirical studies which provide interesting results and add to the area
  • It has a good grounding in the literature and evidence of knowledge of key issues
  • There is evidence of novel, new thinking, new approaches
  • There is evidence that the findings are having impact beyond one institution
  • There is a clear articulation of the methodology and a critique of the approach adopted
  • The submission demonstrates evidence of linking to higher agendas of the day – policy directives or evidence of ideas being embedded in or aligned with funding council programmes
  • A retrospective piece showing how the work builds on or provides a foundation for other work that followed.
  • The paper provides clarity and insight into a well recognised problem.
  • 2. On reading the papers a raft of unresolved issues came to mind:

  • In such a fast moving area as e-learning, is there an issue with papers becoming dated; what papers ‘age’ well and what are their characteristics?
  • Academics submit four artefacts, these are supposed to be reviewed separately, but how much importance will the panel actual put in the coherence of the set in terms of the breadth of expertise demonstrated by researcher across four papers chosen? Will overlapping submissions be viewed unfavourably?
  • What issues are there around individual contributions and multiple authorships? Is collaboration good or bad? Do international co-authors add benefit?
  • How will the panels really view other outputs of research – for example a web site of resources or a technological artefact?
  • How is the issue of comparability going to be addressed? One submission may represent five years of work, another a small-scale study?
  • One could argue, if you take a very strict reading of the criteria in table one, that very rarely is work ‘seminal’ i.e. 4* - but what does this mean in terms of the message it sends on the perceived worth of UK research?
  • How a paper is viewed inevitably depends on who is reading the paper – if it is an expert in the field they either may be more critical (because they are very aware of the subtleties of the work and/or may have a different take on the area) or interested (because it aligns with their own work) compare with a researcher from another area who might rate a review paper highly because if provides an overview/summary of the area or might be disinclined to a paper because it adopts a radically different epistemological position than their own.
  • The RAE is a mammoth exercise, which has absorbed enormous amounts of time and resources, for questionable benefit. RAE 2008 will be the last exercised carried out in this painstaking and complex process. The Government has announced that the exercise will be replaced by a much lighter weight metrics-based system. However it is not clear what this will involve and no doubt a barrage of complaints will surround its introduction as well. It seems it is impossible to get the balance right. Whilst I don’t think many in academia would argue that it is not appropriate for some form of peer-assessment and recognition of worth (after all surely this is the core of how we work anyway through the peer-review funding and publication mechanisms) somehow how it has been done to date does not seem to have been thought through enough. I would advocate for a more interactive, formative and constructive approach with feedback on a timely basis to academics in terms of helping them maximise their research potential whilst balancing this in terms of ensuring high-quality teaching, with strong synergies between the two.

    Handing over ALT-J…

    Monday, October 1st, 2007

    I am just recently submitted the copy for ALT-J issue 15.3 - my final issue as editor! Being an editor has been an interesting experience. These are some thoughts - taken from my editorial for the issue…

    I remember what a huge learning curve it was taking over the first issue I was involved with, as deputy editor for ALT-J 6.3 back in 1998. And if I am honest production of each issue always seems to be an agonising and painstaking experience which didn’t get any easier with time!!! However, I learnt an enormous amount in the process and being editor enabled me to get a perspective on the way e-learning as a research area has developed. I was lucky to have the privilege to work alongside Professor David Squires, until his untimely death in August 2001. Issue 10.3 is dedicated to him and contains research articles from colleagues from around the world who worked with him. In the last nine years the nature of the content in ALT-J has changed, reflecting the general changes in e-learning more broadly. Martin Oliver, Jane Seale and myself as the editorial team worked hard to improve the quality of the articles published in the journal and instigated a more rigorous refereeing process and better feedback to authors on how their papers might be improved. We attempted to introduce a more discursive note to the journal through the inclusion of discussion pieces. I look forward to seeing the journal continue to improve and to being a relaxed, reader, rather than a hassled editor!!!