[Technology] Paying to publish research findings and a row about the fees

[Green Cross Code Man]

William Gibson said that the future is already here, just not evenly distributed, and that seems true of AI. ChatGPT is already a bit "old hat" and limited compared to some of the AI that's already in use (not speculative, but in use already). That said, I use ChatGPT a few times a week to simplify tasks and a software engineer mate reckons his team are 75% more productive using it. They have it generate baseline code which they then edit, or they paste in their code and have it debug what they've done. Both much quicker than humans alone.

I mostly use it to summarise documents - yesterday for example I was reviewing a number of case studies where huge long narrative blocks of text essentially hid what people were trying to say. ChatGPT pulled out the key points in no time and gave me enough to work out if I wanted to look deeper. I use it on my own writing too. I was asked to submit 200 words on a topic for a trade magazine. Bashed out a first draft of just under 300, Asked ChatGPT to edit it down to 200 and correct grammar and make recommendations for clarity...and I only made a handful of amends myself after from the output so it retained my "voice".

In my use-case that's the strength of ChatGPT - it's like a very fast, very capable assistant augmenting what I do not replacing it. I wouldn't use it for searching for things, but AI is being integrated into searching. At the moment you have to go to the site to use it but it's being seamlessly integrated into our day-to-day...Bing is about to be heavily changed with AI features, Google too...in a couple of years we'll look at internet search as it is now as something very quaint and basic.

We're barely scratching the surface though, and some elements of what I do for a living will 100% be replaced by AI soon. IBM announced last week a recruitment freeze on 30% of roles because they think AI will replace them in the next 5 years - while most analysts think new jobs will be created (many jobs people do now didn't exist pre-Internet or smart phone for example) there's still a prediction of a net job loss overall globally. Interesting times.

That is an incredible stat. 75% increase in productivity is revolutionary.

I read some William Gibson books back in the 80's and his ideas have certainly become more apparent in our daily lives these past 20 years or so and more so now where we will start to see greater dependency on AI and perhaps more technological integration within our bodies. Maybe just depends how far we are prepared to go although someone somewhere will go further than law or convention allows. Musk is experimenting in the 'borg' type direction. Thanks for your use case examples - interesting.

 

[Guinness Boy]

Niche, Hazza, but I like it. NSC never fails to surprise.

A wee bit off topic but I agree with the use of ChatGPT as a code review tool. Between that and improvements in automated test tools and better low code / no code solutions you’d think software projects will finally get cheaper and quicker.

“Fortunately”, at least in my industry you can always count on Infosec taking two weeks to agree who creates an SSH key or whitelist an IP, Compliance to spend months signing off the product and literature and at least three mandatory go / no go CAB meetings

 

[Happy Exile]

Brilliant question. The simply answer is we cannot tell, a priori, what is the best (most correct and relevant) research and how this maps to the JIF (which journal to trust, in other words) any more than we can for a 'newspaper' or TV channel (with some obvious egregious exceptions). We know what is the most complicated and expensive data when we see it - and that's the stuff published in the high JIF journals. Journals advertise their JIF (or 'impact factor) like a badge of honour, but you need an expert in the field to interpret it for you because it is based on citations. In medicine, JIFs can be very high - some Lancet journals (there are many) have JIFs over 70. In my area, 10 is considered very good. In engineering 2 is good, plus engineers don't really care because their sort of work is objective and easily testable. Unequivocal and meaningful biomed research is harder to generate, and to verify.

You have also guessed the existence publication bias, how stuff in high JIF journals is read by more people, trusted by more people and cited by more people (having more impact), but has no guarantee of being more likely to be correct. In fact Nature has one of the highest retraction rates (where authors 'retract' their paper because either they or (more likely) someone else has tried to repeat the work or build directly upon it and it turns out to be bolleaux). This is because Nature published work that 'if true would be of incredible importance'. Because it is important people will try to build on it. But it may not be true and if so this will soon be discovered (one hopes). Me and two others wrote a book on all this stuff a couple of years ago. PM me if you're interest. I warn you it is dull.

So in my game (biomed/drug research) the only test of whether a publication is correct is whether it is reproducible and leads on to something. Ironically, journals with low JIFs attract duller research that is less likely to trigger follow-up, and the work is instantly forgotten. The retraction rate in the journal will be low but that doesn't mean the work published is correct. And don't start me on Chinese 'paper mills'.

Paradoxically peer review at high JIF journals can be bloody onerous. Moon on a stick stuff. 'Maintaining standards of quality and depth' as they put it. And yet the one thing that is repeatedly overlooked is a exploration of whether the work is a fair test (done using randomized and blinded methods). A 'peer' may ask me to recapitulate my animal model data in a second species with trangenic knock out of the putative drug target (techno-bollocks), but won't ask me whether we cheated by doing the study unblinded. Madness. Anyway....

The business of generating and testing and publishing scientific information is, alas, undermined by the usual - it is run and done by human beings.

Really, really interesting, thank you. Will PM you about that book - one of the things I try and emphasise in my own work is evidence-based practice. If that evidence is potentially compromised or displaying bias (or later retracted) that's good to be aware of! I appreciate the time taken to answer and the detail, thanks!