⚓ T113916 Switch ResourceLoader file dependency tracking to MultiDC-friendly backend

⚓ T113916 Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Page Menu
Phabricator
Create Task
Maniphest
T113916
Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Closed, Resolved
Public
Actions
Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Assigned To
Krinkle
Authored By
Krinkle
Sep 27 2015, 11:25 PM
2015-09-27 23:25:06 (UTC+0)
Tags
MediaWiki-ResourceLoader
(Accepted Enhancement)
Performance-Team
(To-do: Goals, prioritized next 4 Quarters)
Epic
MW-1.39-notes (1.39.0-wmf.25; 2022-08-15)
MediaWiki-Platform-Team
(Inbox, needs triage)
Referenced Files
None
Subscribers
aaron
Aklapper
Catrope
Jdforrester-WMF
Krinkle
Ladsgroup
ori
View All 9 Subscribers
Description
Status quo
Following
T90001
, we spawned
T113092
for the msg_resource table. This task is for the module_deps table.
The population of the
module_deps
table is deterministic. It is currently stored in the main DB because we want high persistence, due to the high cost of regeneration.
The data is queried for thousands of modules at once from the "startup" module. Generating it all at once would be impossible within our desired HTTP response time (would take tens of seconds).
It's typically populated in a distributed fashion, e.g. from separate on-demand module requests for load.php.
We deal with absence by generating a temporary placeholder version hash. Then, after a user actually needed the module, and requests it with the temporary version hash, that request will do the in-depth computation and stores it in the database. From then on-wards, the "startup" module will contain the correct version hash.
This means that after a deployment, modules for which version hash computation is expensive, will first get invalidated to a temporary hash, and then invalidated again a few minutes later to the eventual one. This is a bit wasteful, but an intentional design decision for ResourceLoader. Improving or avoiding this aspect is outside the scope of this task.
Problem statement
Due to this data being stored in the main MySQL databases, it requires that load.php GET requests make DB-master write queries to change these rows. This is a performance and availability anti-pattern.
It has been mitigated to some extent:
Concurrent writes for the same thing thing are avoided via Memc locks (non-blocking).
These writes only happen from HTTP cache misses, and the responses have a long TTL, and the CDN cache is used and shared by both logged-in and logged-out users alike.
The objective is to store this data elsewhere, outside the databases. But ideally in a way that still upholds as much as possible the persistence.
Ideas
@Catrope
and I had a brain-storm session last week (in context of
T102578
) and came up with the following known issues:
The table stores absolute paths which means when a wmf-branch roll over, it loses track of some files, thus causing a needless cache invalidation. Since old wmf branches are not immediately removed (in part because we have multiple versions in deployment at any one time), the old file paths are not obviously wrong. As such, the table can even end up including both old and new versions of the same file. This and more is tracked under
T111481
Lots of old data is left in module_deps from modules that no longer exist in recent versions of MediaWiki core and extensions, because there is no TTL and no garbage collection.
Also, since the values are deterministic, we do not need a store that is replicated across data centres. A dc-local store is sufficient.
Details
Related Changes in Gerrit:
Subject
Repo
Branch
Lines +/-
ResourceLoader: Enable wgResourceLoaderUseObjectCacheForDeps by default
mediawiki/core
master
+13
-8
ResourceLoader: Fix confusing DependencyStoreException trace logs
mediawiki/core
master
+61
-95
Enable wgResourceLoaderUseObjectCacheForDeps for all wikis (take 2)
operations/mediawiki-config
master
+2
-7
ResourceLoader: Remove DependencyStore::renew
mediawiki/core
wmf/1.39.0-wmf.19
+21
-54
ResourceLoader: Remove DependencyStore::renew
mediawiki/core
master
+21
-54
Revert "Enable wgResourceLoaderUseObjectCacheForDeps for all wikis"
operations/mediawiki-config
master
+5
-4
Enable wgResourceLoaderUseObjectCacheForDeps for all wikis
operations/mediawiki-config
master
+5
-5
Enable wgResourceLoaderUseObjectCacheForDeps for dewiki
operations/mediawiki-config
master
+1
-0
Enable wgResourceLoaderUseObjectCacheForDeps for group0
operations/mediawiki-config
master
+2
-1
rdbms: allow merging IDatabase::upsert() rows with current ones
mediawiki/core
master
+165
-53
resourceloader: Make DepStore write lock specific to the current wiki
mediawiki/core
master
+5
-2
resourceloader: support tracking indirect module dependency paths via BagOStuff
mediawiki/core
master
+643
-140
Enable $wgResourceLoaderUseObjectCacheForDeps for testwiki/test2wiki
operations/mediawiki-config
master
+5
-0
Enable $wgResourceLoaderUseObjectCacheForDeps for Beta Cluster
operations/mediawiki-config
master
+3
-0
objectcache: clean up RedisBagOStuff and optimize changeTTLMulti()
mediawiki/core
master
+151
-81
bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
mediawiki/core
master
+312
-206
objectcache: optimize lock() and unlock() methods in SqlBagOStuff
mediawiki/core
master
+77
-4
Show related patches
Customize query in gerrit
Related Objects
Search...
Task Graph
Mentions
Status
Subtype
Assigned
Task
Resolved
aaron
T88445
MediaWiki active/active datacenter investigation and work (tracking)
Resolved
Krinkle
T270223
FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Resolved
aaron
T252951
ResourceLoader DepStore lock acquired twice?
Resolved
Krinkle
T113916
Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Resolved
aaron
T247028
Database 'INSERT' query rate doubled (module_deps regression?)
Resolved
tstarling
T253433
Web installer CSS fails with ServiceDisabledException
Declined
aaron
T253697
Support hash-based deduplication in KeyValueDependencyStore
Resolved
aaron
T212129
Move MainStash out of Redis to a simpler multi-dc aware solution
Resolved
Eevans
T222851
Improve Echo seentime code for multi-DC access
Resolved
aaron
T229062
Look into a simple way to have global keys with db-replicated
Resolved
Krinkle
T254634
Determine and implement multi-dc strategy for ChronologyProtector
Unknown Object (Task)
Resolved
Papaul
T267041
(Need By: 2020-11-29) rack/setup/install db214[234]
Resolved
Marostegui
T269324
Productionize x2 databases
Resolved
aaron
T274174
Add modtoken field and flags to objectcache table
Resolved
Krinkle
T288998
Significant ParserCache space increase after 2021-08-12 (1.37.0-wmf.18 regression)
Resolved
tstarling
T306118
Notify DBA prior to sending db traffic to x2
Resolved
tstarling
T315271
db1151, db2144 X2 masters error: Could not execute Delete_rows_v1 event on table mainstash.objectstash
Resolved
PRODUCTION ERROR
jcrespo
T315274
Wikimedia\Rdbms\DBTransactionError: Explicit transaction still active; a caller might have failed to call endAtomic() or cancelAtomic().
Resolved
tstarling
T315427
Create a puppet role for x2 hosts
Resolved
Marostegui
T315853
reclone x2 codfw hosts
Resolved
tstarling
T315995
Document how to disable x2 per DC
Mentioned In
T343492: Phase out SqlModuleDependencyStore
T312902: x2-mainstash replication lag (25min Codfw, disk saturated)
T312809: Avoid x2-mainstash replica connections (ChronologyProtector)
T252951: ResourceLoader DepStore lock acquired twice?
T280582: Reduce number of shards in redis_sessions cluster
T270101: Grants not working with DB hosts with to ipv6
T269324: Productionize x2 databases
T247638: Styles not applied during installer due to ResourceLoader trying to cache to the database
T247028: Database 'INSERT' query rate doubled (module_deps regression?)
T102578: Don't trash cache for front-end resources
T127208: Make sure ResourceLoader works with MediaWiki read-only mode
rMW280d29215a89: resourceloader: Vary module_deps on language (in addition to skin)
Mentioned Here
T252945: Avoid constant evictions on Redis main stash
T90001: Evaluate folding msg_resource and msg_resource_links tables into objectcache
T102578: Don't trash cache for front-end resources
T111481: Fix intermittend ghost entries in FileModule 'fileHashes' data
T113092: Revise the design of ResourceLoader's MessageBlobStore
Event Timeline
There are a very large number of changes, so older changes are hidden.
Show Older Changes
Krinkle
removed a parent task:
T102578: Don't trash cache for front-end resources
Jun 23 2018, 11:23 PM
2018-06-23 23:23:46 (UTC+0)
Krinkle
moved this task from
Backlog: Maintenance, non-prioritized
to
To-do: Goals, prioritized next 4 Quarters
on the
Performance-Team
board.
Aug 12 2018, 12:54 PM
2018-08-12 12:54:05 (UTC+0)
Krinkle
updated the task description.
(Show Details)
Mar 3 2019, 4:10 PM
2019-03-03 16:10:28 (UTC+0)
aaron
moved this task from
To-do: Goals, prioritized next 4 Quarters
to
To-do: Goals prioritized current Quarter
on the
Performance-Team
board.
Jun 6 2019, 11:04 AM
2019-06-06 11:04:58 (UTC+0)
Krinkle
assigned this task to
aaron
Jun 10 2019, 1:45 PM
2019-06-10 13:45:33 (UTC+0)
gerritbot
added a comment.
Jun 29 2019, 6:20 AM
2019-06-29 06:20:13 (UTC+0)
Comment Actions
Change 519741 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] [WIP] resourceloader: move indirect module dependency path tracking to BagOStuff
gerritbot
added a project:
Patch-For-Review
Jun 29 2019, 6:20 AM
2019-06-29 06:20:14 (UTC+0)
gerritbot
added a comment.
Jun 29 2019, 8:36 AM
2019-06-29 08:36:16 (UTC+0)
Comment Actions
Change 519746 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: optmize lock() and unlock() for SqlBagOStuff and clean up base method
gerritbot
added a comment.
Jul 1 2019, 7:26 PM
2019-07-01 19:26:45 (UTC+0)
Comment Actions
Change 519766 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
gerritbot
added a comment.
Jul 2 2019, 12:53 AM
2019-07-02 00:53:21 (UTC+0)
Comment Actions
Change 520148 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: clean up RedisBagOStuff and optimize changeTTLMulti()
gerritbot
added a comment.
Jul 4 2019, 2:13 PM
2019-07-04 14:13:54 (UTC+0)
Comment Actions
Change 519746
merged
by jenkins-bot:
[mediawiki/core@master] objectcache: optimize lock() and unlock() methods in SqlBagOStuff
ReleaseTaggerBot
added a project:
MW-1.34-notes (1.34.0-wmf.13; 2019-07-09)
Jul 4 2019, 3:00 PM
2019-07-04 15:00:30 (UTC+0)
gerritbot
added a comment.
Jul 10 2019, 4:34 PM
2019-07-10 16:34:51 (UTC+0)
Comment Actions
Change 519766
merged
by jenkins-bot:
[mediawiki/core@master] bagostuff: optimize SqlBagOStuff and fix failing segmentation tests
ReleaseTaggerBot
edited projects, added
MW-1.34-notes (1.34.0-wmf.14; 2019-07-16)
; removed
MW-1.34-notes (1.34.0-wmf.13; 2019-07-09)
Jul 10 2019, 5:00 PM
2019-07-10 17:00:32 (UTC+0)
gerritbot
added a comment.
Jul 18 2019, 2:07 AM
2019-07-18 02:07:36 (UTC+0)
Comment Actions
Change 520148
merged
by jenkins-bot:
[mediawiki/core@master] objectcache: clean up RedisBagOStuff and optimize changeTTLMulti()
ReleaseTaggerBot
edited projects, added
MW-1.34-notes (1.34.0-wmf.15; 2019-07-23)
; removed
MW-1.34-notes (1.34.0-wmf.14; 2019-07-16)
Jul 18 2019, 3:01 AM
2019-07-18 03:01:20 (UTC+0)
Paladox
subscribed.
Aug 31 2019, 9:26 PM
2019-08-31 21:26:36 (UTC+0)
gerritbot
added a comment.
Feb 13 2020, 11:43 PM
2020-02-13 23:43:55 (UTC+0)
Comment Actions
Change 519741
merged
by jenkins-bot:
[mediawiki/core@master] resourceloader: support tracking indirect module dependency paths via BagOStuff
ReleaseTaggerBot
added a project:
MW-1.35-notes (1.35.0-wmf.20; 2020-02-18)
Feb 14 2020, 12:00 AM
2020-02-14 00:00:53 (UTC+0)
Maintenance_bot
removed a project:
Patch-For-Review
Feb 14 2020, 12:10 AM
2020-02-14 00:10:53 (UTC+0)
Krinkle
mentioned this in
T247028: Database 'INSERT' query rate doubled (module_deps regression?)
Mar 5 2020, 8:23 PM
2020-03-05 20:23:59 (UTC+0)
Krinkle
mentioned this in
T247638: Styles not applied during installer due to ResourceLoader trying to cache to the database
Mar 13 2020, 9:29 PM
2020-03-13 21:29:44 (UTC+0)
gerritbot
added a comment.
Apr 21 2020, 5:13 PM
2020-04-21 17:13:09 (UTC+0)
Comment Actions
Change 591388 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/mediawiki-config@master] Enable $wgResourceLoaderUseObjectCacheForDeps for testwiki/test2wiki
gerritbot
added a project:
Patch-For-Review
Apr 21 2020, 5:13 PM
2020-04-21 17:13:10 (UTC+0)
Krinkle
added a subtask:
T247028: Database 'INSERT' query rate doubled (module_deps regression?)
Apr 21 2020, 5:15 PM
2020-04-21 17:15:48 (UTC+0)
Krinkle
removed projects:
MW-1.35-notes (1.35.0-wmf.20; 2020-02-18)
MW-1.34-notes (1.34.0-wmf.15; 2019-07-23)
May 15 2020, 4:23 PM
2020-05-15 16:23:45 (UTC+0)
Krinkle
updated the task description.
(Show Details)
gerritbot
added a comment.
May 15 2020, 5:00 PM
2020-05-15 17:00:16 (UTC+0)
Comment Actions
Change 596696 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] Enable $wgResourceLoaderUseObjectCacheForDeps for Beta Cluster
gerritbot
added a comment.
May 15 2020, 5:03 PM
2020-05-15 17:03:55 (UTC+0)
Comment Actions
Change 596696
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable $wgResourceLoaderUseObjectCacheForDeps for Beta Cluster
gerritbot
added a comment.
May 15 2020, 6:24 PM
2020-05-15 18:24:35 (UTC+0)
Comment Actions
Change 591388
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable $wgResourceLoaderUseObjectCacheForDeps for testwiki/test2wiki
Maintenance_bot
removed a project:
Patch-For-Review
May 15 2020, 7:10 PM
2020-05-15 19:10:37 (UTC+0)
Stashbot
added a comment.
May 16 2020, 5:49 PM
2020-05-16 17:49:26 (UTC+0)
Comment Actions
Mentioned in SAL (#wikimedia-operations)
[2020-05-16T17:49:25Z] krinkle@mwmaint1002: Running cleanupRemovedModules.php to prune old module_deps rows
T113916
Krinkle
added a comment.
Edited
May 16 2020, 10:37 PM
2020-05-16 22:37:49 (UTC+0)
Comment Actions
With
T252945
in fresh memory, I figured I'd measure the size of the data.
$ foreachwiki mysql.php -- -e 'SELECT SUM(LENGTH(md_deps)) FROM module_deps LIMIT 1;' | tee sum_len_md_deps.log
$ cat sum_len_md_deps.log | grep -E ' \d' | cut -d':' -f2 | sort -nr
54030580
41387066
26092947
21512320
19007285
14817439
14417541
14262574
13003985
12375295
11551918
9890952
9761510
9370569
9069648
8866616
7975923
7893188
7750407
7344726
7131004
6983012
6577260
6547808
6242845
6058073
5948446
5749254
5697395
5555347
5447465
5430159
5267795
5167557
5134440
5083149
5055733
4947383
4927049
4915445
4912533
4729311
4543028
4465091
4406999
4372184
4349138
4263134
4216692
4210804
4166047
4158580
4116897
4106736
4056818
3967823
3775432
3730321
3677109
3652283
3651328
3580369
3578615
3564137
3541948
3523754
3500043
3461331
3412785
3392040
3387307
3348026
3340344
3324441
3273413
3242790
3226985
3181596
3127941
3114936
3082504
3030944
2980344
2899500
2879007
2826075
2815272
2753153
2749836
2665111
2626762
2624422
2615105
2606131
2598139
2515757
2493693
2483965
2477834
2464640
2444317
2441889
2414784
2396959
2388864
2387132
2351250
2332838
2301259
2299767
2298792
2281616
2272844
2269252
2266683
2234966
2219285
2204627
2200973
2184963
2161081
2127392
2093915
2050962
2047276
2046763
2040154
2023501
2013946
2006284
2000641
1993902
1987436
1964954
1959928
1954852
1952979
1932661
1922577
1916785
1899417
1884378
1873953
1867376
1833996
1816899
1808457
1802558
1798640
1796262
1792219
1787891
1779001
1768398
1767936
1766950
1765383
1763840
1752484
1740087
1732507
1711417
1704139
1693397
1692638
1686655
1682247
1649658
1624668
1620208
1612343
1610470
1608322
1604476
1604226
1594274
1589801
1570156
1569972
1568991
1567056
1561489
1557338
1556598
1554962
1551363
1550619
1548643
1547665
1540183
1526539
1510320
1503840
1485509
1481388
1480921
1470280
1463026
1447926
1444144
1437966
1426704
1423287
1417154
1416835
1416200
1415640
1403681
1399462
1395759
1394471
1393515
1392426
1388374
1383247
1382824
1377784
1375962
1372432
1365178
1364719
1358817
1355744
1354526
1351039
1342600
1342328
1338006
1328197
1325801
1320852
1312123
1309040
1298778
1297149
1293863
1292979
1289978
1287889
1286522
1277319
1262836
1260241
1253840
1252850
1244344
1243070
1241882
1239901
1237965
1237508
1236748
1235781
1234854
1231493
1230979
1222300
1222088
1218140
1216332
1210594
1206158
1201048
1201013
1199985
1198886
1193018
1189563
1187780
1187162
1182977
1182036
1179768
1178324
1170206
1164213
1160034
1156215
1150302
1145231
1140021
1139432
1139347
1138385
1134398
1130283
1128631
1127824
1124797
1123231
1120810
1115669
1114562
1110217
1110002
1108924
1103999
1102941
1095932
1090253
1089075
1088017
1087903
1087872
1086003
1085640
1085525
1083610
1083118
1079765
1079605
1079076
1077349
1072605
1070564
1069346
1066797
1064510
1063599
1062973
1062777
1060218
1059663
1057861
1057670
1057643
1055321
1054789
1053830
1049245
1047916
1044945
1044357
1042859
1041388
1040976
1037012
1030441
1027953
1026453
1025255
1022724
1019494
1016157
1015961
1009648
1008502
1005196
1004934
1004541
1002830
1001569
1001003
998793
998753
995261
992575
989787
986962
986413
985465
983432
982969
980423
980378
980303
977538
976784
973155
972247
972058
969058
968686
965769
959118
957501
955978
950515
950232
949234
949046
948185
947364
947292
946765
943035
941139
940121
936960
933393
931979
929705
929302
927696
922878
921030
920692
919752
918550
915763
915498
915460
911978
910305
910118
907912
906935
906631
906366
905280
901330
900347
900156
895076
894679
893257
892336
889230
888246
888202
884384
881125
879907
872995
868760
867096
866133
863329
863115
862437
862075
861824
860562
859177
858541
854226
853994
850439
848767
848550
844075
842388
840748
839967
837776
833966
833795
833545
830796
829110
829063
827324
824966
824504
823879
822499
817101
816908
816309
815758
811279
810751
809409
807099
806931
804293
801235
793603
790705
790012
789342
785871
785263
785187
783574
780183
779736
779522
774615
773765
772866
772737
767916
767301
767141
763655
762829
758640
757817
753913
752269
746906
746829
746732
744990
743593
743299
742686
742594
741691
735576
733914
732153
730995
730981
730796
729238
727155
724594
723201
723117
722284
718991
718399
717919
715261
714570
714482
714305
713984
713230
712758
711017
710689
707257
706978
706877
704923
704815
703958
702847
702800
696033
693136
690478
689115
686893
686603
685630
685457
683166
681563
679556
679271
678288
677745
676320
675425
675263
675256
673063
671268
670489
667594
666979
665006
664130
661004
659811
659517
658137
652214
650917
650461
648866
647953
647865
647862
647405
646747
644282
641924
641568
637923
637156
636632
635751
635143
634508
634434
634224
633558
633419
632558
632492
629683
629352
629077
628144
627119
625835
625305
625269
624918
624468
622528
619748
619023
618942
618427
617762
617304
616754
613487
613259
612836
612210
611810
610374
610129
609758
609507
608889
606710
605907
603806
601824
601354
600631
597403
597358
597181
594748
590063
588545
587104
586038
584849
584047
583342
582152
581983
580187
579498
575551
574883
573175
571914
570763
570331
568870
568463
568080
566274
561554
561028
559571
558655
558023
557186
557167
555704
553714
552749
552508
548315
547792
546931
545011
544432
543561
541373
541257
539956
534993
533377
532115
530818
529655
525827
525799
524587
524452
524144
522777
522086
521839
521770
521536
520589
519692
518641
517830
516336
515985
514732
514723
512049
511631
510914
509919
506100
505977
504366
502692
499329
499206
496935
491171
489929
489667
487271
486315
484932
482385
480273
479775
477718
476821
474127
471841
471284
469951
469910
469798
469643
467528
467085
466494
465670
465178
462984
461388
460770
460463
456440
456156
453214
453098
452319
452239
450028
449850
444655
442508
441467
440997
437935
437195
436510
434517
434100
431500
429648
428836
428587
428513
426349
426150
424309
421098
420250
418167
417283
416982
414086
413740
413182
412194
410006
409880
406115
405015
404525
404490
403591
402200
401424
400957
399250
399003
398342
396454
395664
394403
389988
389448
388241
387660
387200
386379
385176
383516
383508
382161
381219
378690
374807
374506
374461
371894
370001
369658
366659
365517
363595
362775
356684
353466
352043
350902
350848
349886
348986
348569
347844
346958
345309
344862
344564
343746
342050
340248
337455
336756
330751
330097
329547
323797
320542
319914
319867
315646
313530
311897
311110
309028
307965
306177
306008
304878
303848
301930
301405
300928
300476
298823
292457
287598
287302
286719
280513
277871
275494
273312
268750
267027
265063
263242
256153
254118
249167
247938
241584
236462
235601
234407
233163
230372
227906
225215
224196
224157
224110
223930
221581
221260
220848
220199
220143
218356
216603
214663
214056
213809
213038
212293
211577
211199
210738
209415
208088
205886
205581
204813
203195
202386
199960
199928
198934
197651
196968
195998
193811
189896
188689
185319
183651
182239
181986
178762
178703
176952
176357
175494
171531
170543
162227
160994
152880
149729
149719
149675
148638
135813
134114
132898
128033
126254
117906
117407
111727
104949
103688
100197
89060
84832
83697
80691
72932
72005
66608
65884
62520
61386
60714
59521
56760
56151
56066
51622
48819
42964
37032
$ cat sum_len_md_deps.log | grep -E ' \d' | cut -d':' -f2 | awk '{s+=$1}END{print s}'
1330942793
On the most active and skin/language-diverse wikis it typically holds about 20MB of data over 150K rows. On the least-active/closed wikis almost no data at 37KB over a hundred rows or so. The total for all wikis combined is currenly about 1.2GB. The theoretical max for current usage is about 20GB, e.g. if every wiki lazy-computes ~ 20MB of data, or 46GB if every wiki does what commons/wikidata do and need ~50MB of data.
Ladsgroup
subscribed.
May 17 2020, 3:41 AM
2020-05-17 03:41:29 (UTC+0)
gerritbot
added a comment.
May 18 2020, 10:54 PM
2020-05-18 22:54:49 (UTC+0)
Comment Actions
Change 597171 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Make DepStore write lock specific to the current wiki
gerritbot
added a project:
Patch-For-Review
May 18 2020, 10:54 PM
2020-05-18 22:54:50 (UTC+0)
gerritbot
added a comment.
May 19 2020, 12:20 AM
2020-05-19 00:20:59 (UTC+0)
Comment Actions
Change 597171
merged
by jenkins-bot:
[mediawiki/core@master] resourceloader: Make DepStore write lock specific to the current wiki
Maintenance_bot
removed a project:
Patch-For-Review
May 19 2020, 1:10 AM
2020-05-19 01:10:57 (UTC+0)
Krinkle
added a subtask:
T253433: Web installer CSS fails with ServiceDisabledException
May 23 2020, 2:44 AM
2020-05-23 02:44:52 (UTC+0)
Krinkle
removed a parent task:
T117089: Deadlock found when trying to get lock; try restarting transaction
Krinkle
closed subtask
T253433: Web installer CSS fails with ServiceDisabledException
as
Resolved
May 26 2020, 4:59 PM
2020-05-26 16:59:00 (UTC+0)
aaron
closed subtask
T253697: Support hash-based deduplication in KeyValueDependencyStore
as
Declined
Sep 17 2020, 9:05 PM
2020-09-17 21:05:30 (UTC+0)
Krinkle
closed subtask
T247028: Database 'INSERT' query rate doubled (module_deps regression?)
as
Resolved
Nov 16 2020, 8:44 PM
2020-11-16 20:44:33 (UTC+0)
Krinkle
renamed this task from
Redesign ResourceLoader's file dependency tracking (module_deps)
to
Switch ResourceLoader file dependency tracking to MultiDC-friendly backend
Dec 16 2020, 2:10 AM
2020-12-16 02:10:07 (UTC+0)
Krinkle
added a parent task:
T270223: FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Krinkle
added a subtask:
T269324: Productionize x2 databases
Krinkle
removed a subtask:
T252951: ResourceLoader DepStore lock acquired twice?
Dec 22 2020, 9:29 PM
2020-12-22 21:29:22 (UTC+0)
Marostegui
closed subtask
T269324: Productionize x2 databases
as
Resolved
Jan 25 2021, 6:48 AM
2021-01-25 06:48:09 (UTC+0)
RhinosF1
subscribed.
Jan 25 2021, 8:26 PM
2021-01-25 20:26:50 (UTC+0)
aaron
mentioned this in
T269324: Productionize x2 databases
Jan 28 2021, 9:50 PM
2021-01-28 21:50:28 (UTC+0)
Krinkle
reopened subtask
T269324: Productionize x2 databases
as
Open
Feb 1 2021, 8:33 PM
2021-02-01 20:33:16 (UTC+0)
Marostegui
closed subtask
T269324: Productionize x2 databases
as
Resolved
Mar 5 2021, 5:53 AM
2021-03-05 05:53:05 (UTC+0)
Krinkle
raised the priority of this task from
Medium
to
High
Mar 10 2021, 9:26 PM
2021-03-10 21:26:57 (UTC+0)
Krinkle
mentioned this in
T270101: Grants not working with DB hosts with to ipv6
Mar 10 2021, 9:29 PM
2021-03-10 21:29:53 (UTC+0)
Krinkle
edited subtasks, added:
T212129: Move MainStash out of Redis to a simpler multi-dc aware solution
; removed:
T269324: Productionize x2 databases
Apr 19 2021, 6:56 PM
2021-04-19 18:56:39 (UTC+0)
Krinkle
mentioned this in
T280582: Reduce number of shards in redis_sessions cluster
Apr 19 2021, 7:00 PM
2021-04-19 19:00:26 (UTC+0)
Krinkle
moved this task from
To-do: Goals prioritized current Quarter
to
To-do: Goals, prioritized next 4 Quarters
on the
Performance-Team
board.
Sep 20 2021, 6:47 PM
2021-09-20 18:47:20 (UTC+0)
Krinkle
mentioned this in
T252951: ResourceLoader DepStore lock acquired twice?
Sep 24 2021, 9:44 PM
2021-09-24 21:44:45 (UTC+0)
Aklapper
added a parent task:
T252951: ResourceLoader DepStore lock acquired twice?
Sep 28 2021, 4:11 PM
2021-09-28 16:11:21 (UTC+0)
gerritbot
added a comment.
Nov 4 2021, 7:08 PM
2021-11-04 19:08:35 (UTC+0)
Comment Actions
Change 574101 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):
[mediawiki/core@master] rdbms: allow merging IDatabase::upsert() rows with current ones
gerritbot
added a project:
Patch-For-Review
Nov 4 2021, 7:08 PM
2021-11-04 19:08:36 (UTC+0)
gerritbot
added a comment.
Jun 9 2022, 3:13 AM
2022-06-09 03:13:07 (UTC+0)
Comment Actions
Change 574101
merged
by jenkins-bot:
[mediawiki/core@master] rdbms: allow merging IDatabase::upsert() rows with current ones
ReleaseTaggerBot
added a project:
MW-1.39-notes (1.39.0-wmf.16; 2022-06-13)
Jun 9 2022, 4:00 AM
2022-06-09 04:00:31 (UTC+0)
tstarling
closed subtask
T212129: Move MainStash out of Redis to a simpler multi-dc aware solution
as
Resolved
Jun 17 2022, 1:38 AM
2022-06-17 01:38:38 (UTC+0)
Krinkle
claimed this task.
Jul 2 2022, 8:53 PM
2022-07-02 20:53:26 (UTC+0)
Krinkle
added a subscriber:
aaron
gerritbot
added a comment.
Jul 7 2022, 12:45 AM
2022-07-07 00:45:14 (UTC+0)
Comment Actions
Change 811794 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for group0
gerritbot
added a comment.
Jul 7 2022, 11:08 PM
2022-07-07 23:08:49 (UTC+0)
Comment Actions
Change 811794
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for group0
Krinkle
added a comment.
Jul 8 2022, 12:49 AM
2022-07-08 00:49:30 (UTC+0)
Comment Actions
I tested this on mwdebug1002 with WikimediaDebug and "Verbose log" enabled on
. I picked this as being a closed wiki in group0 (since aawiki while closed is a Wikipedia and thus in group2), and then picked an uncommon styles module that has non-zero file deps, with a more unlikely skin-lang combo; thus triggering the write path. I did the same on a differnet combination before the config patch, to have to sets of logs of comparable cache-miss/write requests that would perform a write. I later realized most of this rarity selection was actually not needed for the "after" example (only for the "before" example) because MainStashDB was naturally for RL at this point as I'm the first and likely only person using it via the undeployed config on mwdebug.
Before:
SqlModuleDependencyStore
Wikimedia\DependencyStore\SqlModuleDependencyStore::fetchDependencyBlobs [0.001s] db1175:
SELECT md_module,md_skin,md_deps FROM module_deps
WHERE (md_skin = 'timeless|qqx' AND md_module = 'skins.monobook.styles')
DeferredUpdates::run: started MWCallableUpdate_MediaWiki\ResourceLoader\ResourceLoader->saveModuleDependenciesInternal #646
MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
MemcachedPeclBagOStuff debug: add(aawiki:rl-deps:skins.monobook.styles|timeless|qqx:lock)
Wikimedia\Rdbms\LoadBalancer::getLocalConnection: opened new connection for localAutoCommit/0
Wikimedia\DependencyStore\SqlModuleDependencyStore::fetchDependencyBlobs [0.001s] db1123:
SELECT md_module,md_skin,md_deps FROM module_deps
WHERE (md_skin = 'timeless|qqx' AND md_module = 'skins.monobook.styles')
Wikimedia\DependencyStore\SqlModuleDependencyStore::storeMulti [0.002s] db1123:
INSERT INTO module_deps
(md_module,md_skin,md_deps) VALUES ('skins.monobook.styles','timeless|qqx','[\"resources/src/med…\"]'
MemcachedPeclBagOStuff debug: delete(aawiki:rl-deps:skins.monobook.styles|timeless|qqx:lock)
DeferredUpdates::run: ended MWCallableUpdate_MediaWiki\ResourceLoader\ResourceLoader->saveModuleDependenciesInternal #646, processing time: 0.013
After:
KeyValueDependencyStore
SqlBagOStuff::fetchBlobs [0.001s] db1151:
SELECT keyname,value,exptime FROM objectstash
WHERE keyname = 'aawikibooks:ResourceLoaderModule-dependencies:skins.monobook.styles|vector|qqx' AND (exptime >= '20220707231951')
DeferredUpdates::run: started MWCallableUpdate_MediaWiki\ResourceLoader\ResourceLoader->saveModuleDependenciesInternal #622
MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
MemcachedPeclBagOStuff debug: add(aawikibooks:rl-deps:skins.monobook.styles|vector|qqx:lock)
SqlBagOStuff::modifyTableSpecificBlobsForSet [0.002s] db1151:
INSERT INTO objectstash
(keyname,value,exptime,modtoken) VALUES ('aawikibooks:ResourceLoaderModule-dependencies:skins.monobook.styles|vector|qqx','… ELSE modtoken END)
MemcachedPeclBagOStuff debug: delete(aawikibooks:rl-deps:skins.monobook.styles|vector|qqx:lock)
Wikimedia\Rdbms\LoadBalancer::reuseConnectionInternal: reference count for 0/mainstash reduced to 1
Wikimedia\Rdbms\LoadBalancer::getForeignConnection: reusing connection foreignInUseAutoCommit/0/mainstash
Wikimedia\Rdbms\LoadBalancer::reuseConnectionInternal: freed connection 0/mainstash
DeferredUpdates::run: ended MWCallableUpdate_MediaWiki\ResourceLoader\ResourceLoader->saveModuleDependenciesInternal #622, processing time: 0.007
I haven't done rigorous benchmarking (and won't since it's all defered post-send) but it is interesting that these anecdotally appear to be faster than before. I speculate this is likely to be fairly consistently the case given the before case had a re-select-from-replica optimisation to reduce core db writes on GET, which the after state doesn't have.
I've also monitored these two dashboards for an hour or so for any notable changes or tall spikes:
Nothing standing out so far beyond small spikes no more than regular variations already seen several times an hour, and lasting similar lengths of time. Probably unrelated.
I'll repeat the above dashboard monitoring for the group1/2 rollout as well, where I do expect an observable permanent increase in both read and write traffic on x2. Having said that, behind the CDN,
load.php
totals a mere ~150 req/s on backend appservers. We generaly batch both read and writes, so if we say each will do 1 read and 1 write (generally most will not do writes), then that might still be just noise compared to the mostly-dormant x2 already having 10K read/s and 600 write/s.
Query rates aside, the other factor is the size of the values being retained on average. I analysed that last year at
T113916#6142636
. It's a few MB per wiki, and upto 2GB for all WMF wikis with a theoretical max of 50GB if all wikis gained traffic as diverse as Commons and Wikidata.
gerritbot
added a comment.
Jul 8 2022, 1:35 AM
2022-07-08 01:35:48 (UTC+0)
Comment Actions
Change 812146 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for dewiki
gerritbot
added a comment.
Jul 8 2022, 1:35 AM
2022-07-08 01:35:49 (UTC+0)
Comment Actions
Change 812147 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for all wikis
gerritbot
added a comment.
Jul 8 2022, 1:47 AM
2022-07-08 01:47:09 (UTC+0)
Comment Actions
Change 812146
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for dewiki
gerritbot
added a comment.
Jul 11 2022, 4:01 PM
2022-07-11 16:01:06 (UTC+0)
Comment Actions
Change 812147
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for all wikis
gerritbot
added a comment.
Jul 11 2022, 8:07 PM
2022-07-11 20:07:25 (UTC+0)
Comment Actions
Change 812935 had a related patch set uploaded (by Krinkle; author: Krinkle):
[mediawiki/core@master] ResourceLoader: Fix confusing DependencyStoreException trace logs
Krinkle
mentioned this in
T312809: Avoid x2-mainstash replica connections (ChronologyProtector)
Jul 11 2022, 8:55 PM
2022-07-11 20:55:02 (UTC+0)
gerritbot
added a comment.
Jul 12 2022, 6:57 PM
2022-07-12 18:57:21 (UTC+0)
Comment Actions
Change 813296 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] Revert "Enable wgResourceLoaderUseObjectCacheForDeps for all wikis"
gerritbot
added a comment.
Jul 12 2022, 7:13 PM
2022-07-12 19:13:25 (UTC+0)
Comment Actions
Change 813296
merged
by jenkins-bot:
[operations/mediawiki-config@master] Revert "Enable wgResourceLoaderUseObjectCacheForDeps for all wikis"
Krinkle
mentioned this in
T312902: x2-mainstash replication lag (25min Codfw, disk saturated)
Jul 12 2022, 9:39 PM
2022-07-12 21:39:33 (UTC+0)
gerritbot
added a comment.
Jul 12 2022, 10:26 PM
2022-07-12 22:26:02 (UTC+0)
Comment Actions
Change 813120 had a related patch set uploaded (by Krinkle; author: Krinkle):
[mediawiki/core@master] ResourceLoader: Remove DependencyStore::renew
gerritbot
added a comment.
Jul 13 2022, 5:03 AM
2022-07-13 05:03:52 (UTC+0)
Comment Actions
Change 813120
merged
by jenkins-bot:
[mediawiki/core@master] ResourceLoader: Remove DependencyStore::renew
gerritbot
added a comment.
Jul 13 2022, 9:08 PM
2022-07-13 21:08:30 (UTC+0)
Comment Actions
Change 813670 had a related patch set uploaded (by Krinkle; author: Krinkle):
[mediawiki/core@wmf/1.39.0-wmf.19] ResourceLoader: Remove DependencyStore::renew
gerritbot
added a comment.
Jul 14 2022, 12:30 AM
2022-07-14 00:30:23 (UTC+0)
Comment Actions
Change 813670
merged
by jenkins-bot:
[mediawiki/core@wmf/1.39.0-wmf.19] ResourceLoader: Remove DependencyStore::renew
gerritbot
added a comment.
Jul 14 2022, 12:44 AM
2022-07-14 00:44:50 (UTC+0)
Comment Actions
Change 813725 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for all wikis (take 2)
gerritbot
added a comment.
Jul 14 2022, 12:55 AM
2022-07-14 00:55:31 (UTC+0)
Comment Actions
Change 813725
merged
by jenkins-bot:
[operations/mediawiki-config@master] Enable wgResourceLoaderUseObjectCacheForDeps for all wikis (take 2)
Krinkle
closed this task as
Resolved
Jul 15 2022, 6:50 AM
2022-07-15 06:50:04 (UTC+0)
gerritbot
added a comment.
Aug 3 2022, 5:36 PM
2022-08-03 17:36:48 (UTC+0)
Comment Actions
Change 812935
merged
by jenkins-bot:
[mediawiki/core@master] ResourceLoader: Fix confusing DependencyStoreException trace logs
ReleaseTaggerBot
edited projects, added
MW-1.39-notes (1.39.0-wmf.25; 2022-08-15)
; removed
MW-1.39-notes (1.39.0-wmf.16; 2022-06-13)
Aug 3 2022, 6:00 PM
2022-08-03 18:00:40 (UTC+0)
gerritbot
added a comment.
Jun 6 2023, 6:36 PM
2023-06-06 18:36:30 (UTC+0)
Comment Actions
Change 927755 had a related patch set uploaded (by Krinkle; author: Krinkle):
[mediawiki/core@master] [WIP] ResourceLoader: Enable wgResourceLoaderUseObjectCacheForDeps by default
Krinkle
mentioned this in
T343492: Phase out SqlModuleDependencyStore
Aug 3 2023, 11:09 PM
2023-08-03 23:09:25 (UTC+0)
gerritbot
added a comment.
Aug 3 2023, 11:09 PM
2023-08-03 23:09:45 (UTC+0)
Comment Actions
Change 927755
abandoned
by Krinkle:
[mediawiki/core@master] ResourceLoader: Enable wgResourceLoaderUseObjectCacheForDeps by default
Reason:
Restricted Application
added a project:
MediaWiki-Platform-Team
View Herald Transcript
Aug 3 2023, 11:09 PM
2023-08-03 23:09:45 (UTC+0)
Maintenance_bot
removed a project:
Patch-For-Review
Aug 3 2023, 11:10 PM
2023-08-03 23:10:19 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct.
Wikimedia Foundation
Code of Conduct
Disclaimer
CC-BY-SA
GPL
Credits