I know MediaBiasFactCheck is not a be-all-end-all to truth/bias in media, but I find it to be a useful resource.
It makes sense to downvote it in posts that have great discussion – let the content rise up so people can have discussions with humans, sure.
But sometimes I see it getting downvoted when it’s the only comment there. Which does nothing, unless a reader has rules that automatically hide downvoted comments (but a reader would be able to expand the comment anyways…so really no difference).
What’s the point of downvoting? My only guess is that there’s people who are salty about something it said about some source they like. Yet I don’t see anyone providing an alternative to MediaBiasFactCheck…
Because their methodology is nothing but buzzwords:
Despite apparently having “rigorously defined criteria”, they don’t actually say what they are.
They literally publish their methodology and scoring system.
https://mediabiasfactcheck.com/methodology/
So they do say exactly what their criteria is, and how it is scored. None of that is buzz words, it’s just a summary that fit in a few sentences. You can look at the full methodology if you want more than just that small bullet description.
I’m not saying that you have to agree with their scoring, or that it is necessarily accurate. I just think if you’re going to critique a thing, you should at least know what you’re critiquing.
Oh look. You copied my link!
Sorry. No they don’t.
That’s not “rigorously defined”. It’s a bunch of weasel words and vagaries.
For example. In “factual reporting”, to get a “very high” score:
What does “consistently factual” mean? What qualifies as “a credible source”? What does “prompt” mean?
Those are all nice sounding words, but they don’t really tell you anything. Prompt could be anything from seconds to weeks. (And let’s be honest, probably varies from researcher to researcher.)
Oh they go into more detail….
A questionable source, for example:
Who defines their extreme bias? What is propaganda?
Voice of America is literally a government ran propaganda service yet they assign it high factual, least-biased and high credibility.
Sorry, but their methodology isn’t a methodology, and the only thing that’s inherently reproducible is their fact check rating. Everything else relies on what their subjective analysis.
Consistently factual is exactly that. Both of those words mean actual things. And they go on to say that they can’t fail fact checks. And prompt corrections likely means that as a story develops, that if there were incorrect things reported, they are corrected as soon as the new information is available.
As for who defines extreme bias, it’s literally them. That is what they are saying they are doing. And they spell out what their left vs right criteria are. And how they judge it. Of course this is subjective. There isn’t really a way to judge the political spectrum without subjectivity. They do include examples in their reports about what biased language, sources, or reporting they found. Which allows you to easily judge whether you agree with it.
As for VOA, they say in the ownership portion that it is funded by the US government and that some view it as a propaganda source. They also discuss the history and purpose of it being founded. And then continue on with the factual accuracy and language analysis. You may not agree with it, but it is following their own methodology, and fully explained in the report.
Again, there isn’t anything saying you have to agree with them. It is a subjective rating. I’m not sure how much more transparent they can be though. They have spelled out how they grade, and each report provides explanations and examples that allow you to make your own judgments. Or a starting point for your own research.
If you can define a completely objective methodology to judge political bias on whatever spectrum you choose, then please do. It’s inherently subjective. And there isn’t really a way around that.
So what constitutes “consistently factual”, then? if the ‘consistently factual’ means ‘always factual’, then the explanation of allowing ‘prompt corrections’ is unnecessary. A “correction” is different than an “update”, after all. so what rate of error is “rigorously” defined here?
Further, how do they deal with (the vast majority) of fact checkers, using qualified language like “mostly factual” or “misleading” or “out of context”. or “distorted”?
“likely…” They don’t say that. I wonder why they don’t just say that?
You’re assuming that’s what “prompt” means, but that’s… an assumption. as I said, it could be anything from seconds to weeks. I assume- i don’t know, lets just be honest here- that their language is intentional. which means it’s probably not that.
Seems like it would be a super easy thing to actually define. Like. ‘Consistently Factual’ could be “No more than X percentage of articles requiring corrections or otherwise failing a 3rd party fact check”.
So glad we agree on that.
Compare, VOA’s to Al Jazeera’s. Which, Al Jazeera is Qatar-owned. even so, It’s widely considered a reliable news source; where as, VOA was literally forbidden from being served within the US borders precisely because it was propaganda, until 2013- when it decided to open up drops to the internet specifically to “counter” Al Qaeda messaging. (aka. propaganda.)
VOA:
it should be noted that A), its so nice to know that their journalists are held to a standard. (I’m sure Al Jazeera journalists aren’t…) and b) that there’s a survey saying 84% of people that actually look at VOA is reliable. A survey conducted by… their board of governors… and the linked source is the appropriations PDF…
Compared to Al Jazeera:
now, I’m not saying Al Jazeera isn’t Qatari propaganda, it more or less is. but you see the the totally different tone here?
Now lets move onto the bias/analysis section. VOA:
Emphasis mine (also the italics just to make the headlines clear.) Now the emphasised bits is straight up bullshit. it’s government funded. It’s entire purpose- even today- is to disseminate pro-US propaganda everywhere outside the US. it’s forbiden from radio broadcasts that might reach US soil, and it’s only allowed to drop things on the interent because of a special provision specifically to counter messaging by terrorists.
Factual or not, it’s a propaganda outlet.
Al Jazeera:
now, VOA’s review is easily seen as pure spin. MBFC goes out of their way to assauge any doubt what so ever that they’re factual and not biased. nop. no sir. Now, it would be fair to say that because they literally define bias using the US discourse as the meter stick… that there is no bias. Sort of chicken and the egg, right? any how… there’s no mention of Al Jazeera’s code of ethics… and the cited failed fact checks? date to 2018, one of which falls outside the 5 year window since it was last updated- the fact check was published august of 2018, when it was updated in October of 2023. Pedantic, I know, but the 5 year window is their rule.
all it takes is a five minute scroll through VOA to see that they have the same misleading bias towards the US/US government as Al Jazeera has towards Qatar.
VOA’s was last updated in… Nov 2022.
you don’t need to define something that’s not subjective, exactly. But they need to explain what the methodology is. they’re looking for loaded words? then we need examples of what are loaded words that they’re looking for. that shouldn’t be too hard. it doesn’t even need to be exhaustive. just exhaustive enough.
Putting it on the individual articles makes it arbitrary. ask yourself… is “deadly” a loaded word? Or is it qualitative leading to understand that people actually died from the “deadly attack” rather than were just sent to the hospital in “an attack”. or that people died in a wildfire, hurricane or something else. Nobody can check every article to get a sense for their own criteria, and what they posted as a methodology is far from sufficient to the task of repeating their process. Ideally, I should be able to take their methodology article, follow it more or less step by step, and produce at least similar results. Can’t come even close.
With your own reply you show that they have given you most of the information needed to make your own assessment. Like I’ve said other places in this thread, you don’t have to agree with them. I have never claimed they are correct. I’m saying that they provide information about how they arrived at their conclusion, you can assess that information and decide whether you agree.
It still stands that it is at least a reasonable place to look to gather basic information about a media source. And provides you with a solid starting point to research and make an assessment about a news source.
I agree that using the US political spectrum pretty significantly skews things since US politics is almost all center to right if you compare it to the wider spectrum globally. But since they gave their information, and what spectrum they are using it makes it pretty simple to get a baseline for most media outlets at a glance if it’s not one I’m familiar with.
And with the number of outright insane news sources people like to share, it’s useful to have a way to get at least a decent snapshot of what to expect.
There is a lot of good stuff there but it’s still opaque when it comes to bias specifically. I mean, am I missing somether here? I genuinely feel like there must be a whole section I’ve missed or something based on some of the other commenters. The bias methodology is no more a methodology than “grind up some wheat, mix some water and yeast before chucking it in the oven for a bit” is a recipe for bread. You rate 4 categories from 0 - 10 and average it, but the ratings themselves are totally subjective.
What does this even mean? If a site runs stories covering the IPCC recommendations for climate action but doesn’t run some right wing conspiracy version of how climate change is a hoax, is that biased story selection?
What did I miss here?
Bravo for bringing the notes. On a first glance, some of these feel like they require subjectivity (like, do we really believe the political spectrum is 1d?), but I agree I could run the computation myself from this.
There is definitely some subjectivity. Language isn’t something that is easily parsed and scored. That is why they give examples on the actual report about the kind of biased language they saw, or whatever other issues led to the score given.
I don’t think they mean for their website to be the end all bias resource. More of a stepping off point for you to make your own judgments.
On each page, they describe, in detail, exactly how they come to their conclusions.
While you may disagree with what they have to say, to claim they’re hiding anything or that they aren’t being transparent or arbitrary is just untrue.
here’s their definition of what’s a left or right bias
It’s pretty fucking arbitrary.
Additionally, their methodology is a bunch of gibberish and buzz words. that they explain their justification on each article is inadequate. For example, Al jazeera is dinged for using “negative emotion” words like “Deadly”.
Deadly might invoke a certain kind of emotion. but it’s also the simplest way to describe an attack in which some one dies. Literally every news service will use “deadly attack” if people are dying, regardless if it’s an attack by terrorists, or by cackling baboons. (or indeed not even an attack. for example ‘Deadly wildfire’ or ‘deadly hurricane’.) the application of using that as an example is extremely arbitrary, on a case by case basis.
Now you’re just repeating yourself. That doesn’t make it any more true.
And as far as your claims of methodology being arbitrary, just because you use words in an arbitrary manner does not make their methodology arbitrary.
Like I said, just because you don’t agree with them doesn’t make them wrong or you right. Feel free to block them if you don’t like it. But other users here have clearly demonstrated how your argument does not hold water.
Okay.
Take their methodology.
Work through it.
You can’t because most of the “rigorous definitions “ aren’t shared.
You still haven’t explained what “factually consistent” means in a method that’s repeated and able to be applied regularly.
Their methodology as posted is far too vague to adequately consider their ability to provide consistent neutral ratings.
How are “loaded” words evaluated? Is there a table of words that are considered “loaded”? Personal feeling? We don’t know. We know what some of them are, since they’re mentioned on specific articles.
But that isn’t a consistent or “rigorously defined” criteria. So what is the “rigorously defined criteria”- and why is that not published?
Do you not see how that’s ripe for abuse?
I have used their methodology and worked through it. I find no fault with it.
And finally, you’re the one who makes claims that there is some problem with their methodology, yet you have not demonstrated that at all. All you demonstrated is that you happen to disagree with it and that you don’t like it. If you wish to prove your point, you’re gonna need evidence for that, and all of your carrying on here I have not seen the shred of that.
Just block it and move on already. Your disagreement is hardly worth this crusade.
So then, It should be simple for you to tell us what rate of error is acceptable to still qualify as “factually consistent”.
This is like giving. Recipe without measures, or a “how to build a shed” guide without describing how to build the pad it sits on.
I haven’t? Huh. Interesting. So all those “rigorously defined criteria”, those are public? We know how they’re actually evaluated?
We know what error rate is “Factually Consistent”, we know how they treat “misleading” tags or “misrepresentation” tags in their factual rating?
I mean in my looking for an example where they clearly do not have a consistent methodology, I found it the first place I looked. (Okay, so I knew VOA news and Al Jazeera are both state owned propaganda outlets.)
They’re both inherently biased. Yet one is “least biased” just because its owners happen to the us gov? Oh look. Here’s a third gov-owned propaganda outlet. Gee, what makes VOa special?
No but the open discourse here and in similar communities is. Me blocking it just hides it from me. MBFC is being used, in part, to evaluate sources for articles.
It’s a third party, private-interest group whose methods aren’t clear and self-evidently inconsistently applied.
Even if they were demonstrably always right… that’s a problem, because sometimes the best source/news agency to talk about a given issue sucks.
Sometimes the discussion is about awareness of how shitty “that rag” is.
i’m not here to waste time trying to convince of you of something about which you’ve clearly made up your mind, since others have shared plenty of facts, made great arguments, and all you do is keep shifting the goalposts.
not to mention: it’s not for me to prove your claims-- that’s on you, and you haven’t. all i have claimed is that i’m satisfied, and the only proof you need of that is my word ont he matter.
so, once again, since you haven’t proven anything other than you disagree with it, i suggest you simply block it and move on with your life. you have no greater authority to decide what is or is not a “reliable source” than MBFC, but at least they show their work.
I shift the goalposts but am just repeating myself? interesting.
In any case… as for my “claims” perhaps I’ve missed something. Again. From their own methodology page:
Okay. so that’s the highlevel sales pitch. emphasis mine.
Perhaps. just perhaps, I’ve missed where they dropped what those defined criteria are. lets keep reading.
Objective indicators? what indicators? Where? for you or me to understand how they’re arriving at their analysis, I need to understand what “objective indicators” they’re using. they’re not listed anywhere I can find. Perhaps I’ve missed it. I don’t think I have. but. perhaps I have.
Now, Skipping down to the specific categories…
Alright. now we’re getting to the stuff I’m asking for! maybe. uh. shit. The just “Biased Wording/Headlines” at that. So they have no list of common loaded words, (For example, is “Deadly Wildfire” okay but “Deadly Attack” not? both are describing events in which people presumably died. What you, I or anyone else perceives as “loaded” is going to be entirely different. You want to rigorously define criteria for bias? you’re gonna have to at least provide examples. And not on the individual ratings. Protip. the lack of strong or emotional language is also an indication of bias- for examples of that, watch reports surrounding any cops that killed a subject. you’re almost certainly going to be seeing the pro-cop news agencies shy away from language that evokes anger.
Then then get into their “comprehensive” analysis:
yeah. uhm. that’s not “comprehensive”. at all. MPR news, just from today, just the ones that get highlighted, Minnesota Public Radio news has 28 articles. from today. and that’s not even bothering to look at all of the massive amounts of MPR/NPR affiliated podcasts and such being pumped out sometimes 3 times a day.
Further, there’s no information on which articles are selected. Which can have a profound impact on whether or not they get a passing grade for factualness. If you’re only checking ten out of literal thousands of articles a year. or, even a hundred articles, out of thousands a year, how you select articles to review are going to have a profound impact. Is it random? is it by top rating? are they cherry picked? top headlines from random dates?
And lets draw attention to that last line. “This process can be time consuming or very simple, depending on the source”. meaning… it varies based on the source. Even if there’s more to work with for a given source… the process should probably not be any more or less simple- the process should be the process. that’s the purpose of a methodology.
Skipping the descriptions of their fact check ratings… all I’m going to say here is that there’s no objective standard for what “consistent” or “often” or any sort of miss-rate on being factual. I will submit that, for example VOA news probably should be given a low factual score based on this statement: >A “Low” rating indicates the source is often unreliable and should be fact-checked for fake news, conspiracy theories, and propaganda.
you know, considering VOA is literally a state media outlet. whose entire purpose is to pump out propaganda; yet it’s given a ‘high’ rating. but what do I know, they certainly weren’t forbidden from broadcasting inside US boarders because of their propagandist nature.
their critera for who they use as a factcheck service is useful:
IFCN is good. the date restriction is good. explaining how correct fact checks affect things… is good. I would like to see a comment about which fact checkers they always use, or always use when it’s relevant (for example, reviewing a french news service using, idunno, a taiwanese fact checker seems kinda sketchy.) Do they search all 115 current signatories and the other 54 that are in the renewal process? do they search only those from the source’s home country? when do they elect to expand beyond that? do they only use one service at all?
I’d assume they use some sort of aggregator service to look for fact checks across all of them at once. Personally, my preferred choice would be an aggregation service combining all of them, and searching for articles tagged as fact checking the specific source, rather than for each of the articles being reviewed. Then organize those by some sort of pass/mostly-pass/fail/epically-fail sort of metric. but that’s just me.
TL:DR? my goal post has always been that their methodology is opaque and not useful to determine that their method reasonably eliminates their bias. that has never changed. they don’t describe what acceptable error rates are for factualness (never mind severity of the error. reporting a person wore a green shirt when they wore a blue shirt might be factually incorrect, but does it really matter if the story isn’t about what shirt they wore?). they don’t describe even in brief detail what ‘loaded’ or ‘biased’ headlines actually look like. They describe a literal propaganda service as being “Least Biased”.
They cite newsguard as a competitor (i’m not sure about that, but they’re in the same space. from what I see on their website… they’re selling their service to different audiences. Like brands looking to advertise on a specific site, etc.) Lets look at their methodology page. I’m not going to go into detail. but you see how it’s broken down? how specific. each criterion is specifically listed, with reasons for it passing or failing a given criterion listed, as well as express explanations of what things mean. When you’re looking through it. not ‘we judge on bias… which means that we look for biased words…’. Like a phrase you see is ‘that a regular use would not likely see it on a daily basis’.
Check their scoring process. They have a researcher (described as a trained journalist), research the website, make a report, then they write the article. that article is then put on pause for comment from the company in question … then it is reviewed by a people (“at least one senior ediitor and Co-CEO”…) to check for factual accuracy and what have you. Only then is it published. I assume that MBFC has something similar, but that’s an assumption. no where does it describe the editorial process. for all we know, it really is just one guy in a cat suit working the one article, doing it his way while the lady in the dog suit is doing it her way and the editorial staff are in a two-person horse suit searching for organic oats. I’d rather assume not, but again. that is an assumption on my part.
That’s not sufficient.
A private trust assessing company shouldn’t be given free space in an open public forum as though it’s assessments we’re something the general public should be aware of. If you trust it you can go seek it’s assessment off site. But this company shouldn’t be allowed to spam the fediverse of all places.
By that logic, no privately owned media company would be able to post links here at all. Because your description pretty much describes all of them too, from the AP to CNN to Fox News.
And why should you get to set the standards for what everyone else sees? If that’s what you want, start your own instance and ban this bot. But this bot was put in place by the instance admins, and they get to do what they want on their own server. You not liking it or happening to disagree with it gives you no right to tell them what to do.