⚓ T393237 Some en.wikipedia pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
Page Menu
Phabricator
Create Task
Maniphest
T393237
Some en.wikipedia pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
Closed, Resolved
Public
PRODUCTION ERROR
Actions
Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Assigned To
Zabe
Authored By
Helpful_Raccoon
May 3 2025, 8:14 AM
2025-05-03 08:14:29 (UTC+0)
Tags
Wikimedia-production-error
(Apr-Jun 2025)
WMF-General-or-Unknown
DBA
(Done)
Wikimedia-database-issue (Bad data)
Referenced Files
F59834418: T393237.sql
May 9 2025, 10:44 PM
2025-05-09 22:44:33 (UTC+0)
F59651417: enwiki_text_table_rows.txt
May 3 2025, 6:32 PM
2025-05-03 18:32:14 (UTC+0)
F59650882: image.png
May 3 2025, 5:14 PM
2025-05-03 17:14:47 (UTC+0)
F59645451: image.png
May 3 2025, 8:14 AM
2025-05-03 08:14:29 (UTC+0)
Subscribers
Aklapper
Anomie
Cremastra
dancy
Ladsgroup
matmarex
Peachey88
View All 13 Subscribers
Description
Consistent error loading specific enwiki pages
If I navigate to
, I get an internal error page: Fatal exception of type "MediaWiki\Revision\RevisionAccessException". This error happens consistently.
It occurs on multiple devices (Safari on iPhone, Firefox on Windows 11), both logged in and logged out. I am not getting this error on any other page, including other archives of
Error
service.version: 1.45.0-wmf.1
trace.id:
6a3f577c-d577-4615-8995-bffab74d53ef
Find trace.id in Logstash
labels.normalized_message
[{reqId}] {exception_url} MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
Frame
Location
Call
from
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionStore.php(1208)
#0
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionStore.php(1514)
MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#1
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/SlotRecord.php(324)
MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#2
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRecord.php(221)
MediaWiki\Revision\SlotRecord->getContent()
#3
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RenderedRevision.php(230)
MediaWiki\Revision\RevisionRecord->getContentOrThrow(string, int, null)
#4
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRenderer.php(236)
MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#5
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRenderer.php(169)
MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, MediaWiki\Parser\ParserOptions, array)
#6
/srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RenderedRevision.php(196)
MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#7
/srv/mediawiki/php-1.45.0-wmf.1/includes/Storage/DerivedPageDataUpdater.php(1422)
MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#8
/srv/mediawiki/php-1.45.0-wmf.1/includes/Storage/DerivedPageDataUpdater.php(1862)
MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#9
/srv/mediawiki/php-1.45.0-wmf.1/includes/page/WikiPage.php(1852)
MediaWiki\Storage\DerivedPageDataUpdater->doParserCacheUpdate()
#10
/srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiPurge.php(116)
MediaWiki\Page\WikiPage->updateParserCache(array)
#11
/srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(2010)
MediaWiki\Api\ApiPurge->execute()
#12
/srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(948)
MediaWiki\Api\ApiMain->executeAction()
#13
/srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(919)
MediaWiki\Api\ApiMain->executeActionWithErrorHandling()
#14
/srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiEntryPoint.php(152)
MediaWiki\Api\ApiMain->execute()
#15
/srv/mediawiki/php-1.45.0-wmf.1/includes/MediaWikiEntryPoint.php(198)
MediaWiki\Api\ApiEntryPoint->execute()
#16
/srv/mediawiki/php-1.45.0-wmf.1/api.php(44)
MediaWiki\MediaWikiEntryPoint->run()
#17
/srv/mediawiki/w/api.php(3)
require(string)
#18
{main}
error.stack.previous_trace
from /srv/mediawiki/php-1.45.0-wmf.1/includes/Storage/SqlBlobStore.php(279)
#0 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionStore.php(1204): MediaWiki\Storage\SqlBlobStore->getBlob(string, int)
#1 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionStore.php(1514): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, null, null, null, int)
#2 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/SlotRecord.php(324): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#3 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRecord.php(221): MediaWiki\Revision\SlotRecord->getContent()
#4 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RenderedRevision.php(230): MediaWiki\Revision\RevisionRecord->getContentOrThrow(string, int, null)
#5 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRenderer.php(236): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#6 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RevisionRenderer.php(169): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, MediaWiki\Parser\ParserOptions, array)
#7 /srv/mediawiki/php-1.45.0-wmf.1/includes/Revision/RenderedRevision.php(196): MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#8 /srv/mediawiki/php-1.45.0-wmf.1/includes/Storage/DerivedPageDataUpdater.php(1422): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#9 /srv/mediawiki/php-1.45.0-wmf.1/includes/Storage/DerivedPageDataUpdater.php(1862): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#10 /srv/mediawiki/php-1.45.0-wmf.1/includes/page/WikiPage.php(1852): MediaWiki\Storage\DerivedPageDataUpdater->doParserCacheUpdate()
#11 /srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiPurge.php(116): MediaWiki\Page\WikiPage->updateParserCache(array)
#12 /srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(2010): MediaWiki\Api\ApiPurge->execute()
#13 /srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(948): MediaWiki\Api\ApiMain->executeAction()
#14 /srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiMain.php(919): MediaWiki\Api\ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.45.0-wmf.1/includes/api/ApiEntryPoint.php(152): MediaWiki\Api\ApiMain->execute()
#16 /srv/mediawiki/php-1.45.0-wmf.1/includes/MediaWikiEntryPoint.php(198): MediaWiki\Api\ApiEntryPoint->execute()
#17 /srv/mediawiki/php-1.45.0-wmf.1/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#18 /srv/mediawiki/w/api.php(3): require(string)
#19 {main}
Impact
Notes
See also:
T387683: Some de.wikipedia.org pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
T386162: Some dewiktionary pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
Details
MediaWiki Version
1.45.0-wmf.1
Related Objects
Mentions
Duplicates
Mentioned In
T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production)
Mentioned Here
T373668: MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}. If this problem persist, use the findBadBlobs maintenance script to investigate the issue and mark bad blobs.
T392834: mwmaint1002 is out of disk space
T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production)
T386162: Some dewiktionary pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
T387683: Some de.wikipedia.org pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
Duplicates Merged Here
T417904: MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}
T406518: MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
Event Timeline
Helpful_Raccoon
created this task.
May 3 2025, 8:14 AM
2025-05-03 08:14:29 (UTC+0)
Restricted Application
added a subscriber:
Aklapper
View Herald Transcript
May 3 2025, 8:14 AM
2025-05-03 08:14:30 (UTC+0)
Peachey88
changed the subtype of this task from "Bug Report" to "Production Error".
May 3 2025, 8:22 AM
2025-05-03 08:22:21 (UTC+0)
Restricted Application
added a project:
Wikimedia-production-error
View Herald Transcript
May 3 2025, 8:22 AM
2025-05-03 08:22:21 (UTC+0)
Peachey88
subscribed.
May 3 2025, 8:23 AM
2025-05-03 08:23:27 (UTC+0)
Comment Actions
@Helpful_Raccoon
: In future, please copy and paste the full error messages, as it allows the text to be selected/searched.
Helpful_Raccoon
added a comment.
May 3 2025, 8:24 AM
2025-05-03 08:24:24 (UTC+0)
Comment Actions
I am getting this error on at least one other archive page:
[dca5c09c-1703-4a35-b1f2-db3cdce4d179] 2025-05-03 08:24:07: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"
Zabe
subscribed.
May 3 2025, 9:29 AM
2025-05-03 09:29:04 (UTC+0)
Anomie
subscribed.
May 3 2025, 3:31 PM
2025-05-03 15:31:10 (UTC+0)
Comment Actions
I was seeing this on
. At least for that case, I tracked it down to something being broken with
Anomie
added a comment.
May 3 2025, 4:01 PM
2025-05-03 16:01:11 (UTC+0)
Comment Actions
Checking various other pages transcluding Template:Purple, some are showing the error currently, while others seem to work from cache for the moment but attempting to preview an edit breaks. Attempting to edit or delete Template:Purple itself also fails with the same message about a RevisionAccessException.
matmarex
subscribed.
May 3 2025, 5:00 PM
2025-05-03 17:00:22 (UTC+0)
Comment Actions
[13a24de6-3fd1-4757-ad2d-d04126119d13] /wiki/Template:Purple MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274359757. Use findBadBlobs.php to remedy. for revision 488705891. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
matmarex
added a project:
WMF-General-or-Unknown
May 3 2025, 5:01 PM
2025-05-03 17:01:00 (UTC+0)
matmarex
added a project:
DBA
May 3 2025, 5:11 PM
2025-05-03 17:11:30 (UTC+0)
Comment Actions
See also
T387683
, which may or may not be the same error.
Tagging
DBA
for help, based on the last time this happened (
T386162
).
matmarex
added a comment.
May 3 2025, 5:14 PM
2025-05-03 17:14:47 (UTC+0)
Comment Actions
Relevant Logstash query:
There is a very slow trickle of 1-10 errors per day (affecting different revisions on different wiki), then a big spike of the errors from Template:Purple on enwiki, with the first one on May 2, 2025 @ 23:26:41.784.
Zabe
added a comment.
May 3 2025, 6:32 PM
2025-05-03 18:32:14 (UTC+0)
Comment Actions
This was caused by
T183490
. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.
But for some reason I cannot explain, 431 of those rows are actually still in use.
enwiki_text_table_rows.txt
5 KB
[1] -
Zabe
mentioned this in
T183490: MCR schema migration stage 4: Migrate External Store URLs (wmf production)
May 3 2025, 6:55 PM
2025-05-03 18:55:29 (UTC+0)
Xaosflux
subscribed.
May 3 2025, 7:11 PM
2025-05-03 19:11:04 (UTC+0)
Cremastra
subscribed.
May 3 2025, 7:51 PM
2025-05-03 19:51:37 (UTC+0)
Ladsgroup
claimed this task.
May 3 2025, 9:10 PM
2025-05-03 21:10:07 (UTC+0)
Ladsgroup
subscribed.
Comment Actions
The same old thing as
T386162: Some dewiktionary pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
fixing it.
Ladsgroup
moved this task from
Triage
to
In progress
on the
DBA
board.
May 3 2025, 9:10 PM
2025-05-03 21:10:21 (UTC+0)
Zabe
added a comment.
May 3 2025, 9:57 PM
2025-05-03 21:57:45 (UTC+0)
Comment Actions
We looked at the binlogs and it appears that the current revision of the purple template is corrupted, so I went ahead marked that as bad and created a new revision with the correct content.
zabe@mwmaint1002:~$ mwscript findBadBlobs.php enwiki --revisions 488705891 --mark "T393237"
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Scanning 1 ids
! Found bad blob on revision 488705891 from 20120422202619 (main slot): content_id=444499441, address=
Changed address to
- Scanned a batch of 1 revisions
Marked 1 bad revisions.
zabe@mwmaint1002:~$ mwscript shell.php enwiki
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Psy Shell v0.12.7 (PHP 7.4.33 — cli) by Justin Hileman
> $cache = \MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
= Wikimedia\ObjectCache\WANObjectCache {#902}
> $key = $cache->makeKey('revision-slots', '', 14350025, 488705891);
= "enwiki:revision-slots::14350025:488705891"
> $cache->delete( $key );
= true
> ^D
INFO Ctrl+D.
zabe@mwmaint1002:~$
Xaosflux
added a comment.
May 3 2025, 10:37 PM
2025-05-03 22:37:52 (UTC+0)
Comment Actions
Note, viewing the old version still results in a mediawiki error
Zabe
added a comment.
May 3 2025, 10:42 PM
2025-05-03 22:42:20 (UTC+0)
Comment Actions
In
T393237#10789204
@Xaosflux
wrote:
Note, viewing the old version still results in a mediawiki error
Also marked the two previous revisions as bad. Please note that there are multiple caching layers, so it will take a bit for that to be visible.
zabe@mwmaint1002:~$ mwscript findBadBlobs.php enwiki --revisions 488567353,276113208 --mark "T393237"
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Scanning 2 ids
! Found bad blob on revision 276113208 from 20090309211539 (main slot): content_id=247612691, address=
Changed address to
! Found bad blob on revision 488567353 from 20120421230203 (main slot): content_id=444370439, address=
Changed address to
- Scanned a batch of 2 revisions
Marked 2 bad revisions.
zabe@mwmaint1002:~$
Xaosflux
added a comment.
May 4 2025, 4:43 AM
2025-05-04 04:43:20 (UTC+0)
Comment Actions
One of the bad revisions appears to be cleared (giving badrevision result), however this is still failing with error:
Xaosflux
added a comment.
May 5 2025, 8:01 PM
2025-05-05 20:01:34 (UTC+0)
Comment Actions
In
T393237#10789308
@Xaosflux
wrote:
One of the bad revisions appears to be cleared (giving badrevision result), however this is still failing with error:
Note, this is now marked as a bad version
Zabe
added a comment.
May 6 2025, 3:40 PM
2025-05-06 15:40:15 (UTC+0)
Comment Actions
Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above.
@Ladsgroup
could you look if the binlogs do contain something for this one?
service.version: 1.44.0-wmf.27
trace.id:
f52ca013-0dbc-4d9c-ad84-b2868a676b4c
Find trace.id in Logstash
Failed to load data blob from Unable to fetch blob at tt:274392442. Use findBadBlobs.php to remedy. for revision 819689534. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
Zabe
added a project:
Wikimedia-database-issue (Bad data)
May 6 2025, 9:58 PM
2025-05-06 21:58:02 (UTC+0)
Ladsgroup
added a comment.
May 7 2025, 10:23 AM
2025-05-07 10:23:50 (UTC+0)
Comment Actions
The same issue:
### DELETE FROM `enwiki`.`text`
### WHERE
### @1=792737651 /* INT meta=0 nullable=0 is_null=0 */
### @2='DB://cluster25/140977019' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
### @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--
### DELETE FROM `enwiki`.`text`
### WHERE
### @1=274392442 /* INT meta=0 nullable=0 is_null=0 */
--
### DELETE FROM `enwiki`.`text`
### WHERE
### @1=724987058 /* INT meta=0 nullable=0 is_null=0 */
### @2='DB://cluster25/106413391' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
### @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--
It's also clear that it's not a log rotation at middle of transaction issue.
Stashbot
added a comment.
May 7 2025, 3:49 PM
2025-05-07 15:49:22 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2025-05-07T15:49:21Z]
T393237
Ladsgroup
added a comment.
May 9 2025, 12:33 PM
2025-05-09 12:33:46 (UTC+0)
Comment Actions
I suggest running findBadBlobs on all of 400 revs and call it a day.
Pppery
subscribed.
Edited
May 9 2025, 3:51 PM
2025-05-09 15:51:44 (UTC+0)
Comment Actions
How was it possible that 400 revisions were lost here? Why were backups of the text table not taken that could be used to recover from this mistake? Just accepting that the content of 400 revisions is now gone forever seems remarkably cavalier to me.
Ladsgroup
added a comment.
May 9 2025, 3:55 PM
2025-05-09 15:55:00 (UTC+0)
Comment Actions
In
T393237#10807988
@Pppery
wrote:
How was it possible that 400 revisions were lost here? Why were backups of the text table not taken that could be used to recover from this mistake? Just accepting that the content of 400 revisions is now gone forever seems remarkably cavalier to me.
1- 400 in 1,280,000,000 edits is nothing.
2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.
matmarex
added a comment.
Edited
May 9 2025, 10:44 PM
2025-05-09 22:44:33 (UTC+0)
Comment Actions
In
T393237#10807998
@Ladsgroup
wrote:
In
T393237#10807988
@Pppery
wrote:
2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.
The revisions were old, but they certainly existed until recently. Template:Purple (whose latest revision was affected) was accessible until recently (
), and so were pages using the template (e.g.
). This looks to me like we lost the data recently, not so long ago that we can't do anything about it.
In
T393237#10796606
@Zabe
wrote:
Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above.
Presumably this error was masked at first due to the caching layers mentioned in
T393237#10789205
In
T393237#10788958
@Zabe
wrote:
This was caused by
T183490
. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.
But for some reason I cannot explain, 431 of those rows are actually still in use.
The two revisions that caused widespread issues, and – I just checked – the majority of the affected revisions, are revisions that are followed by page protections or moves, which create dummy revisions with identical text. This is probably a hint. Maybe the text rows could be associated with multiple revision rows, but the script assumes a one-to-one relationship and deleted too much?
T393237.sql
7 KB
Sample:
rev_id
content_address
comment_text
505952912
tt:274350127
Anaxial moved page [[Talk:Mexican Cottontail]] to [[Talk:Mexican cottontail]]: Per [[WP:FNAME]]
282162613
tt:274350152
moved [[Talk:The Rockettes]] to [[Talk:The Rockettes (dance company)]]: with only the definite article making the difference, there are two similarly-named groups, a dance company and a synchro skating team
316302510
tt:274350153
moved [[Talk:Amphitryon (1935 film)]] to [[Talk:Amphitryon (film)]]
276186194
tt:274350364
Protected Tennessee: [[WP:MOVP|Page-move vandalism]]: target ([move=sysop] (indefinite))
651596623
tt:274350442
Corkythehornetfan moved page [[Template talk:Truman State Bulldogs football coach navbox]] to [[Template talk:Truman Bulldogs football coach navbox]]: Truman does not use 'State' in their name
matmarex
added a comment.
May 9 2025, 11:16 PM
2025-05-09 23:16:25 (UTC+0)
Comment Actions
They also all were originally created in a narrow time range, between 9 and 10 March 2009. According to Server Admin Log archives (
2008 Oct - 2009 Jun
March 9-10
), there was some trouble with the revision external storage on enwiki on those exact days:
March 10
00:52 brion: domas makes it all better. yay domas!
00:50 brion: cluster21 master (srv161) in read-only; we're having write failure problems on multiple wikis
00:48 brion: putting ES back into service on enwiki; srv160/cluster20 master has been fixed. slaves still running it, but this is safe for read/write since reads will fall back to master
March 9
20:15ish-20:30ish Brion:
some (unspecified) blob tables in ES broken due to MAX_ROWS=1m. writes broken on enwiki. domas is rebuilding tables
disabling wgDefaultExternalStore on enwiki temporarily as hack measure. last text_id was 274347580 before
now actually disabling read-only too
So presumably these revisions were written in a funny way, or maybe already processed with some other maintenance scripts that modified them in a funny way.
(Reminder to please log your one-off database-altering scripts… You never know when this will answer someone's question 15 years later)
Zabe
added a comment.
May 10 2025, 12:30 AM
2025-05-10 00:30:58 (UTC+0)
Comment Actions
In
T393237#10808846
@matmarex
wrote:
In
T393237#10807998
@Ladsgroup
wrote:
In
T393237#10807988
@Pppery
wrote:
2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.
The revisions were old, but they certainly existed until recently. Template:Purple (whose latest revision was affected) was accessible until recently (
), and so were pages using the template (e.g.
). This looks to me like we lost the data recently, not so long ago that we can't do anything about it.
In
T393237#10796606
@Zabe
wrote:
Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above.
Presumably this error was masked at first due to the caching layers mentioned in
T393237#10789205
For the second occurrence mentioned above the content of the revision was indeed cached in the
SqlBlobStore cache
In
T393237#10788958
@Zabe
wrote:
This was caused by
T183490
. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.
But for some reason I cannot explain, 431 of those rows are actually still in use.
The two revisions that caused widespread issues, and – I just checked – the majority of the affected revisions, are revisions that are followed by page protections or moves, which create dummy revisions with identical text. This is probably a hint. Maybe the text rows could be associated with multiple revision rows, but the script assumes a one-to-one relationship and deleted too much?
T393237.sql
7 KB
Sample:
rev_id
content_address
comment_text
505952912
tt:274350127
Anaxial moved page [[Talk:Mexican Cottontail]] to [[Talk:Mexican cottontail]]: Per [[WP:FNAME]]
282162613
tt:274350152
moved [[Talk:The Rockettes]] to [[Talk:The Rockettes (dance company)]]: with only the definite article making the difference, there are two similarly-named groups, a dance company and a synchro skating team
316302510
tt:274350153
moved [[Talk:Amphitryon (1935 film)]] to [[Talk:Amphitryon (film)]]
276186194
tt:274350364
Protected Tennessee: [[WP:MOVP|Page-move vandalism]]: target ([move=sysop] (indefinite))
651596623
tt:274350442
Corkythehornetfan moved page [[Template talk:Truman State Bulldogs football coach navbox]] to [[Template talk:Truman Bulldogs football coach navbox]]: Truman does not use 'State' in their name
Yes indeed all of these former text table rows do have multiple content rows references to them. This is something that should not happen (dummy revisions should reuse the former content row), but we did realise this at some point (
T373668
). So we used a bloom filter to gather a list of text table rows which are referenced multiple times from the content table (you can find that list at
mwmaint1002.eqiad.wmnet:/home/zabe/text_table_cleanup/enwiki
). You will notice that the above 431 listed rows are a subset of this list.
In general what happened is that I ran
migrateESRefToContentTable.php
on enwiki. We did pass the former list of duplicate references to it, and it would skip deleting those for now. In line 192 of that script, it writes the es address of every content row updated to a file (for enwiki you can currently find this file at
stat1009.eqiad.wmnet:/home/zabe/enwikitest/enwiki
) regardless of whether the corresponding text row is being deleted or not. Sometimes this did not work (i.e.
T392834
), but it should cover most of it (the file is 69GB and contains 1.209.020.606 lines).
So for those 431 rows what should have happened is that on the first occurrence in the content table, the corresponding external storage address should have been written to the
content_address
field and the mapping from the text table row to the external storage address should have been written to the dump. But both of these failed. Also both of these failed again on the second (or third) occurrence of this text table row in the content table.
In the end I ran
migrateESRefToContentTableStage2.php
assuming all references to the text table, where it was possible, were migrated (this assumption was mostly based on the fact that the content table only contained around 66.000 remaining references, while the number of duped content rows was around 5,8 million, so that those must have been migrated; it turned out 431 were not). These deletions are the ones we looked at at the binlogs above.
So in the end the current this was certainly caused by me running the stage 2 script, the 431 named rows certainly were semi corrupted in some way which somehow allowed reading its content but touching them did cause issues.
There are 3 main indicators for this, 1. the script should not at all randomly skip the content table rows referencing these text table rows without any reason, especially since the script did it at least two times for each of those rows, 2. the binlogs did not, as they usually do, showed the previous content of the row when they were deleted (
T393237#10799542
) and 3. the sal entries you found which show that there were external storage saving issues at the time of the creation of these rows.
matmarex
added a comment.
May 10 2025, 2:31 AM
2025-05-10 02:31:07 (UTC+0)
Comment Actions
Thanks for the additional details. I'm still struggling to understand how the data could be available for reading by MediaWiki for all these years, but could not be updated by the maintenance script that should have migrated it, but I'll take your word for it. I guess it's too late to hope to review the script logs?
The data seems to also be available in the XML dumps – I checked just one page, but
enwiki-20250501-pages-meta-history15.xml-p14324603p14439634
has all of the revisions of Template:Purple, including the missing ones. If someone really wanted, they could probably restore it from there.
Zabe
added a comment.
May 10 2025, 8:38 AM
2025-05-10 08:38:51 (UTC+0)
Comment Actions
In
T393237#10809023
@matmarex
wrote:
Thanks for the additional details. I'm still struggling to understand how the data could be available for reading by MediaWiki for all these years, but could not be updated by the maintenance script that should have migrated it, but I'll take your word for it. I guess it's too late to hope to review the script logs?
Tbh I also do not understand how that could have been the case. Since the script ran over a period of 7 months and was restarted multiple times the script log is no longer there.
The data seems to also be available in the XML dumps – I checked just one page, but
enwiki-20250501-pages-meta-history15.xml-p14324603p14439634
has all of the revisions of Template:Purple, including the missing ones. If someone really wanted, they could probably restore it from there.
Aklapper
renamed this task from
Consistent error loading a specific enwiki page: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"
to
MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}.
May 20 2025, 8:56 AM
2025-05-20 08:56:57 (UTC+0)
Aklapper
set Request URL to https://en.wikipedia.org/w/api.php?action=purge&assert=*&forcerecursivelinkupdate=*&format=*&formatversion=*.
Aklapper
updated the task description.
(Show Details)
Aklapper
set Release Version to 1.45.0-wmf.1.
Krinkle
renamed this task from
MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}.
to
Some en.wikipedia pageviews fatal "RevisionAccessException: Failed to load data blob from {address} for revision {revision}."
May 20 2025, 5:43 PM
2025-05-20 17:43:18 (UTC+0)
Krinkle
removed Request URL.
Krinkle
updated the task description.
(Show Details)
Krinkle
moved this task from
Untriaged
to
Apr-Jun 2025
on the
Wikimedia-production-error
board.
Wahbarz01
closed this task as
Resolved
May 23 2025, 11:18 AM
2025-05-23 11:18:24 (UTC+0)
Wahbarz01
subscribed.
This comment was removed by
Ladsgroup
taavi
reopened this task as
Open
May 23 2025, 11:19 AM
2025-05-23 11:19:25 (UTC+0)
Zabe
merged a task:
T406518: MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
Oct 6 2025, 8:03 PM
2025-10-06 20:03:14 (UTC+0)
Zabe
added a subscriber:
TheresNoTime
Wellverywell
subscribed.
Jan 13 2026, 1:01 PM
2026-01-13 13:01:49 (UTC+0)
dancy
merged a task:
T417904: MediaWiki\Revision\RevisionAccessException: Failed to load data blob from {address} for revision {revision}
Feb 19 2026, 8:05 PM
2026-02-19 20:05:49 (UTC+0)
dancy
subscribed.
dancy
added a comment.
Thu, Apr 9, 3:16 PM
2026-04-09 15:16:41 (UTC+0)
Comment Actions
A few recent hits from train log triage:
[280b2715-2781-4d66-a00e-ba6417723306] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2 MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274365094. Use findBadBlobs.php to remedy. for revision 832352351. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[da0c8b3f-7fbe-4c07-9cd2-f13504e9c49a] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2 MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274360148. Use findBadBlobs.php to remedy. for revision 693354709. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[6a429a86-2c58-4d75-a201-e04c2eaf457b] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2 MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274395852. Use findBadBlobs.php to remedy. for revision 657587061. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[d4a5cc69-5210-42e0-b5b3-fbbc5a3ec5ec] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2 MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274372796. Use findBadBlobs.php to remedy. for revision 561641691. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
Can someone run
findBadBlobs
Zabe
claimed this task.
Thu, Apr 9, 3:19 PM
2026-04-09 15:19:47 (UTC+0)
Helpful_Raccoon
unsubscribed.
Thu, Apr 9, 6:56 PM
2026-04-09 18:56:03 (UTC+0)
Stashbot
added a comment.
Fri, Apr 10, 12:29 AM
2026-04-10 00:29:28 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2026-04-10T00:29:27Z]
T393237
Zabe
added a comment.
Fri, Apr 10, 12:31 AM
2026-04-10 00:31:33 (UTC+0)
Comment Actions
Marked the following revisions as bad. Please note that since revision may share the same content row, and per above analysis this should be the case for every single one of these, this also effects at least another 425 revisions.
I will double check in the next few days whether there are any revisions left that need to be marked as bad.
276101121,276101318,276101514,276101519,276101568,276101650,276101759,276101775,276101817,276101854,276102034,276102062,276102161,276102190,276102732,276102750,276102938,276103027,276103326,276103504,276103529,276103530,276103741,276103819,276104163,276104327,276104377,276104631,276104807,276104862,276104933,276105055,276105106,276105322,276105484,276105515,276105621,276105921,276105932,276105982,276106006,276106027,276106099,276106243,276106276,276106298,276106378,276106520,276106638,276106693,276106856,276106944,276107045,276107104,276107202,276107253,276107301,276107344,276107382,276107487,276107582,276107616,276107675,276107899,276107930,276108121,276108124,276108195,276108196,276108197,276108272,276108599,276108901,276108922,276108990,276109009,276109031,276109148,276109365,276109471,276109519,276110092,276110215,276110313,276110447,276110689,276110723,276110786,276110924,276111155,276111160,276111180,276111344,276111393,276111555,276111679,276111693,276111848,276111979,276112167,276112172,276112245,276112260,276112366,276112387,276112435,276112485,276112722,276112773,276113050,276113279,276113605,276113678,276113765,276113965,276114262,276114306,276114465,276114627,276114885,276115329,276115640,276115721,276115811,276115886,276116114,276116380,276116520,276116625,276116680,276116692,276116757,276116853,276116962,276117086,276117309,276117331,276117552,276117641,276117946,276117971,276118004,276118099,276118183,276118400,276118403,276118600,276118602,276118670,276118785,276118845,276119062,276119114,276119146,276119167,276119232,276119247,276119516,276119545,276120019,276120130,276120281,276120422,276120489,276120987,276121108,276121289,276121382,276121424,276121507,276121646,276121669,276121713,276122006,276122016,276122089,276122614,276123108,276123152,276123190,276123823,276123987,276124093,276124494,276124503,276124523,276124920,276125049,276125223,276125462,276125926,276126149,276126330,276126366,276126464,276126934,276127591,276127617,276127805,276127887,276127988,276128106,276128167,276128199,276128700,276128815,276128996,276129214,276129349,276129919,276130033,276130591,276130872,276130932,276131000,276131184,276131425,276131563,276131651,276131672,276131699,276131783,276132164,276132265,276132662,276132859,276133133,276133309,276133352,276133448,276133482,276133607,276133811,276133824,276134040,276134292,276134408,276134486,276134753,276134945,276134964,276135638,276135707,276135858,276136194,276136464,276136467,276136776,276136822,276136916,276138024,276138092,276138283,276138714,276138791,276139376,276139463,276139611,276139617,276139908,276140000,276140136,276140298,276140303,276140578,276140592,276140631,276140753,276141166,276141237,276141339,276141569,276141705,276141752,276141792,276141812,276141857,276142085,276142088,276142218,276142355,276142486,276142806,276143522,276143605,276144152,276144180,276144223,276144424,276144479,276144508,276144681,276144723,276144772,276144787,276144801,276144970,276145170,276145285,276145494,276145729,276145881,276146024,276146123,276146199,276146241,276146349,276146418,276146425,276146524,276147068,276147085,276147391,276147662,276147802,276147881,276147986,276148067,276148077,276148127,276148385,276148429,276148690,276148813,276149182,276149403,276149501,276149542,276149715,276149723,276149774,276149880,276149944,276150136,276150199,276150222,276150267,276150347,276150477,276150565,276150632,276150655,276150675,276150678,276150806,276150855,276150863,276151106,276151622,276152220,276152331,276152513,276152546,276152631,276152830,276152990,276153008,276153076,276153108,276153126,276153189,276153265,276153328,276153733,276153762,276154225,276154232,276154449,276154560,276154781,276154904,276154938,276155090,276155722,276155729,276155889,276155974,276156157,276156437,276157140,276157521,276157576,276158010,276158090,276158192,276158406,276158790,276158930,276159008,276159156,276159251,276159296,276159297,276159666,276159945,276159990,276160181,276160205,276160440,276160708,276160771,276160820,276160933,276162488,276162674,276163370,276163374,276163805,276164025,276164809,276166874,276177631,276178908,276179429,276179768,276180243,276184044,276184841,276185772,276185998,276186194,276186958,276187976,276198954,276198958
Stashbot
added a comment.
Tue, Apr 21, 1:02 AM
2026-04-21 01:02:54 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2026-04-21T01:02:53Z]
T393237
Zabe
added a comment.
Edited
Tue, Apr 21, 1:03 AM
2026-04-21 01:03:49 (UTC+0)
Comment Actions
The following revisions are also affected by this. I hope that is all now.
276101737,276102792,276103677,276104891,276105655,276105718,276105793,276105810,276105834,276105855,276106134,276106156,276106187,276106212,276106231,276106253,276106265,276106269,276106547,276106580,276106607,276106631,276106696,276106716,276106755,276106816,276106858,276106863,276106889,276108288,276108569,276108888,276110128,276110182,276110242,276110861,276111630,276114407,276114458,276114881,276115798,276119300,276119988,276120225,276120232,276120589,276121147,276121151,276121259,276121338,276121367,276122986,276123356,276124060,276126055,276127166,276127438,276129111,276131480,276132513,276132992,276134048,276136949,276140046,276140464,276141152,276141694,276141814,276142387,276144811,276145745,276146398,276147525,276157334,276159645,276167825,276200063,276200575,276208172,276216344,276218739,276218870,276221673,276246873,276253636,276264991,276268519,276268712,276273830,276294200,276295839,276300550,276318197,276323334,276334958,276374754,276417148,276417238,276432704,276441005,276441010,276441113,276441202,276461008,276494235,276556285,276606524,276607800,276611993,276614894,276716279,276772411,276781509,276888655,276901440,276901457,276916347,276923256,276943680,277024149,277124057,277124093,277158092,277245967,277340285,277423103,277639071,277644570,277684164,277757791,277758461,277759330,277759867,277760169,277761535,277761690,277762035,277762523,277762854,277762961,277763100,277763237,277763546,277833576,277854221,277914502,277925742,277930450,277981859,278051834,278081981,278102108,278102220,278209809,278216466,278330536,278417713,278418733,278534260,278574880,278692493,278919320,279141374,279160074,279172304,279306113,279351297,279402366,279603421,279622841,279749406,280071424,280071610,280312854,280549182,280550558,280692282,280725058,280825622,280856332,280941251,281118160,281665050,281903837,281921989,281937405,282162613,282163458,282236009,282409794,282410473,282567842,282573431,282611409,283029677,283044400,283044909,283044912,283423727,283525575,283939106,284186028,284299477,284426544,284427447,284431979,284509562,284601414,284601812,284602282,284648801,284648810,284882020,284885710,284977028,284979923,285257431,285402663,285549001,285678327,285739447,285891463,286162482,286950542,286959623,287010859,287085256,287209805,287209992,287210295,287210466,287212879,287213497,287213627,287213761,287213888,287213989,287214421,287214480,287214565,287217487,287217687,287217844,287218021,287218195,287218652,287219263,287219383,287219542,287220290,287220474,287227078,287227466,287227858,287227946,287230541,287230965,287231126,287240621,287241047,287247181,287249442,287253999,287254520,287268995,287453607,287453963,287454649,287484184,287519967,287611411,287759195,287813912,287841478,288177600,288177902,288178090,288178256,288178314,288178388,288179394,288179514,288179777,288179864,288179902,288179939,288180221,288229069,288634907,288635298,288695960,288730294,288731668,289232084,289256603,289449799,289479705,289757085,289822712,290766491,290960875,291002375,291843017,291899050,291973381,292327322,292731140,292731608,292732001,292732598,292732879,293389700,293491415,293522755,293651971,293652077,293652096,293652135,293735118,293759214,293759697,293858549,294159541,295077201,295190540,295838141,295942568,296047760,297421464,297789745,298485128,298485179,298928986,298929910,299525621,299526616,300185545,300185774,300265571,300496128,300677615,301219453,301716469,302433168,302744822,302799352,303070131,303413418,303947297,304323020,304323066,305542419,305614840,305671891,306759473,308766910,310348703,310356838,310492152,310580965,313765228,313964053,315032307,315086575,315676799,316302510,316771472,317271403,317550978,317551572,317553683,317646170,317921324,317921927,318608593,318657265,320119719,320767304,322106188,322708903,323070211,323820571,324583039,324584233,326607482,326749194,327216174,328510269,329036632,329364285,330059042,330677350,331904395,332885488,332885617,333914795,334925910,334925914,335160080,335200589,335543633,338740341,340089807,340582067,340809441,343430690,345098118,346003373,346576054,346584365,347289220,351296725,351625887,352854189,352855156,353203004,353887950,354567495,354960875,356395379,357994181,360859804,361488161,363537229,366146746,366214901,368083899,369498088,369498122,369498144,370041566,370529639,370566942,371080792,371560261,371560279,371565321,372192620,372272117,373255702,375454078,380025018,380028153,380416523,382308480,386509710,386771106,386771650,387952400,388791572,388822641,391695565,394583634,403348693,405593920,409031891,412691912,413493447,413634489,413769020,414209326,414654567,416936575,429692235,429692378,431217082,432729556,438499379,438613304,440962211,444831039,448926003,449028402,449821479,455433500,459105260,464174694,465496742,466428985,466758948,466759084,468106877,468107108,469084882,469595888,469596414,469596766,469597092,469597411,469597879,469598194,469598259,469598300,469599334,469599355,469599389,469600347,469600382,469600405,469601246,469601276,469601301,469603878,469603909,469603934,469603961,469964806,475302342,475456241,483691343,484108492,488567213,488705701,499598938,502696312,502861396,505952912,507869124,507879126,513469324,514027895,516037214,516787632,523692645,525376363,525817007,528436522,528440073,530560360,534769789,559820496,561641338,561641691,590865895,603723591,651596623,657587061,693354709,718291933,832352351,852817194,852817328,900502859
Zabe
closed this task as
Resolved
Tue, Apr 21, 11:49 PM
2026-04-21 23:49:26 (UTC+0)
Maintenance_bot
moved this task from
In progress
to
Done
on the
DBA
board.
Wed, Apr 22, 12:29 AM
2026-04-22 00:29:06 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct.
Wikimedia Foundation
Code of Conduct
Disclaimer
CC-BY-SA
GPL
Credits