⚓ T50626 Provide wiki metadata in the databases similar to toolserver.wiki
Page Menu
Phabricator
Create Task
Maniphest
T50626
Provide wiki metadata in the databases similar to toolserver.wiki
Closed, Resolved
Public
Actions
Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Assigned To
coren
Authored By
scfc
May 20 2013, 4:35 AM
2013-05-20 04:35:00 (UTC+0)
Tags
Toolforge
(Backlog)
Referenced Files
None
Subscribers
Beta16
jeremyb
Krenair
liangent
MZMcBride
Petrb
Pietrodn
View All 11 Subscribers
Description
Toolserver has the local table toolserver.wiki on all databases that provides metadata about the wikis including the server the wiki's database is kept on:
| mysql> SELECT * FROM toolserver.wiki LIMIT 5;
| +----------------+------+------------+------------------+------+---------+-----------+--------------+--------------+---------------+--------+-------------+
| | dbname | lang | family | domain | size | is_meta | is_closed | is_multilang | is_sensitive | root_category | server | script_path |
| +----------------+------+------------+------------------+------+---------+-----------+--------------+--------------+---------------+--------+-------------+
| | aawikibooks_p | aa | wikibooks | NULL | 3 | 0 | 1 | 0 | 0 | NULL | 3 | /w/ |
| | aawiki_p | aa | wikipedia | NULL | 6 | 0 | 1 | 0 | 0 | NULL | 3 | /w/ |
| | aawiktionary_p | aa | wiktionary | NULL | 1 | 0 | 1 | 0 | 1 | NULL | 3 | /w/ |
| | abwiki_p | ab | wikipedia | ab.wikipedia.org | 807 | 0 | 0 | 0 | 0 | NULL | 3 | /w/ |
| | abwiktionary_p | ab | wiktionary | NULL | 0 | 0 | 1 | 0 | 1 | NULL | 3 | /w/ |
| +----------------+------+------------+------------------+------+---------+-----------+--------------+--------------+---------------+--------+-------------+
| 5 rows in set (0.00 sec)
| mysql>
Most of the information can probably be extracted from operations/mediawiki-config, but I don't know which sources there are authoritative.
See also:
T50625: Provide namespace IDs and names in the databases similar to toolserver.namespace
Details
Reference
bz48626
Related Objects
Search...
Task Graph
Mentions
Status
Subtype
Assigned
Task
Resolved
bd808
T60788
Toolserver migration to Tools (tracking)
Resolved
bd808
T60791
Missing Toolserver features in Tools (tracking)
Resolved
coren
T69476
Add wiki title case sensitivity flag (is_sensitive) to meta_p.wiki to support jbo.wp and wiktionary tools
Resolved
jcrespo
T50930
Database replication problems - production and labs (tracking)
Resolved
coren
T50626
Provide wiki metadata in the databases similar to toolserver.wiki
Mentioned In
T106989: Provide and maintain "pattern" to extract configuration information from operations/mediawiki-config
T93483: Add script_path to meta_p.wiki database
Mentioned Here
T50625: Provide namespace IDs and names in the databases similar to toolserver.namespace
Event Timeline
bzimport
raised the priority of this task from
to
Medium
Nov 22 2014, 1:18 AM
2014-11-22 01:18:01 (UTC+0)
bzimport
added a project:
Toolforge
bzimport
set Reference to bz48626.
scfc
created this task.
May 20 2013, 4:35 AM
2013-05-20 04:35:00 (UTC+0)
scfc
added a comment.
May 25 2013, 2:58 AM
2013-05-25 02:58:26 (UTC+0)
Comment Actions
Played around with:
include ($MediaWikiRepoPath . "/includes/Defines.php");
include ($WmfConfigRepoPath . "/wmf-config/InitialiseSettings.php");
var_dump ($wgConf->settings);
but it doesn't yield for example information about de.wikipedia.org.
liangent
added a comment.
May 25 2013, 7:46 AM
2013-05-25 07:46:25 (UTC+0)
Comment Actions
(In reply to comment #1)
Played around with:
include ($MediaWikiRepoPath . "/includes/Defines.php");
include ($WmfConfigRepoPath . "/wmf-config/InitialiseSettings.php");
var_dump ($wgConf->settings);
but it doesn't yield for example information about de.wikipedia.org.
Some experiments:
$ php maintenance/eval.php
$wgDBname='zhwiki';
$wmfRealm='production';
$mwConfigDir="$IP/../operations/mediawiki-config";
$wmfConfigDir="$mwConfigDir/wmf-config";
function getRealmSpecificFilename($p){global $IP,$wmfConfigDir;return str_replace($p,$IP,$wmfConfigDir);}
function wmfLoadInitialiseSettings($c){global $wmfConfigDir;require("$wmfConfigDir/InitialiseSettings.php");}
require("$wmfConfigDir/wgConf.php");
list($site,$lang)=$wgConf->siteFromDB($wgDBname);
$wikiTags=array();
$mwConfigDirHandle=opendir($mwConfigDir);
while(($f=readdir($mwConfigDirHandle))!==false){if(pathinfo($f,PATHINFO_EXTENSION)==='dblist'&&in_array($wgDBname,array_map('trim',file("$mwConfigDir/$f")))){$wikiTags[]=pathinfo($f,PATHINFO_FILENAME);}}
$dbSuffix = ( $site === 'wikipedia' ) ? 'wiki' : $site;
$wgConf->loadFullData();
$globals = $wgConf->getAll( $wgDBname, $dbSuffix,array('lang' => $lang,'site' => $site,'stdlogo' => "//upload.wikimedia.org/$site/$lang/b/bc/Wiki.png"), $wikiTags );
print_r($globals);
Array
[wgLegacyEncoding] =>
[wgCapitalLinks] => 1
...
liangent
added a comment.
Jun 2 2013, 5:41 PM
2013-06-02 17:41:58 (UTC+0)
Comment Actions
Do we want a database table consisting of three columns: wiki, config_variable_name, and config_variable_value (as a serialized blob)?
MZMcBride
added a comment.
Jun 2 2013, 5:48 PM
2013-06-02 17:48:30 (UTC+0)
Comment Actions
I think we should have a discussion about what the current "toolserver" database is, what we want in the future, and whether we care about breaking backward compatibility.
Some of the design decisions in some of the database tables could probably be re-thought, but only if we're willing to break the current interfaces.
In addition, I think we should only rely on MediaWiki's API for this information (with user authentication, as necessary). This is the cleanest and sanest way to accurately get this information, as far as I know.
coren
added a comment.
Jun 2 2013, 7:50 PM
2013-06-02 19:50:28 (UTC+0)
Comment Actions
(In reply to comment #4)
In addition, I think we should only rely on MediaWiki's API for this
information (with user authentication, as necessary).
This is particularly important in that some extensions may have hard-to-evaluate effect on some configuration values (namespaces and usergroups being the more obvious cases).
I should say that any necessary configuration value that cannot be fetched through the API should be /added/ to the API rather than fetched through an alternative scheme.
Marc
Platonides
added a comment.
Jul 12 2013, 5:53 PM
2013-07-12 17:53:32 (UTC+0)
Comment Actions
API is per wiki. toolserver.wiki is a meta table.
coren
added a comment.
Jul 12 2013, 9:33 PM
2013-07-12 21:33:15 (UTC+0)
Comment Actions
Yes, but you need to populate that table from /somewhere/. :-)
coren
added a comment.
Aug 27 2013, 8:40 PM
2013-08-27 20:40:00 (UTC+0)
Comment Actions
I've added a table with automatically maintained meta information
about the replicated databases: meta_p.wiki (which is available on every
shard).
+------------------+--------------+------+-----+---------+-------+
Field
Type
Null
Key
Default
Extra
+------------------+--------------+------+-----+---------+-------+
dbname
varchar(32)
NO
PRI
NULL
lang
varchar(12)
NO
en
name
text
YES
NULL
family
text
YES
NULL
url
text
YES
NULL
size
decimal(1,0)
NO
slice
text
NO
NULL
is_closed
decimal(1,0)
NO
has_echo
decimal(1,0)
NO
has_flaggedrevs
decimal(1,0)
NO
has_visualeditor
decimal(1,0)
NO
has_wikidata
decimal(1,0)
NO
+------------------+--------------+------+-----+---------+-------+
There is a lingering issue with the 'name' column which seems to
improperly encode the Wiki name when non-ascii characters are involved;
that will get fix once I manage to beat some sense into mysql.
Most columns are self-explanatory, and I can add a few more depending on
demand. In the meantime, (dbname, slice) provides the much requested
mapping between databases and slices.
Platonides
added a comment.
Aug 27 2013, 8:44 PM
2013-08-27 20:44:51 (UTC+0)
Comment Actions
decimal(1,0) ? This seems strange. Shouldn't those is_* and has_* be BOOL aka. TINYINT(1) ?
coren
added a comment.
Aug 27 2013, 8:47 PM
2013-08-27 20:47:12 (UTC+0)
Comment Actions
I did not want to rely on the existence of bool, which isn't ANSI; mysql "helpfully" translated my numeric(1) to decimal(1,0).
Platonides
added a comment.
Aug 27 2013, 8:48 PM
2013-08-27 20:48:10 (UTC+0)
Comment Actions
Would be a problem to rename slice to server, in order to match the column name of toolserver?
The name column looks good to me from a quick look, btw.
coren
added a comment.
Aug 27 2013, 8:50 PM
2013-08-27 20:50:05 (UTC+0)
Comment Actions
It would be possible, but probably unhelpful: from what I understand, the server column is numeric whereas I provide actual host names. Keeping the column named the same with changed semantics seems to be asking for trouble IMO (i.e.: better a select fails than return a string that is misinterpreted as an integer by code with poor error checking).
coren
added a comment.
Aug 28 2013, 7:44 PM
2013-08-28 19:44:07 (UTC+0)
Comment Actions
Added a meta_p.legacy view that has the same column name and order as toolserver.wiki for legacy purposes.
Please note that the semantics of the 'server' columns differs and there may be other subtle differences with the toolserver's table not immediately evident. Unless the same code base has to run on both labs and the toolserver for the interval while it still has replication, transitioning to use meta_p.wiki is preferable.
valhallasw
updated the task description.
(Show Details)
Jan 18 2015, 2:12 PM
2015-01-18 14:12:47 (UTC+0)
valhallasw
set Security to None.
Ricordisamoa
awarded a token.
Mar 5 2015, 10:31 PM
2015-03-05 22:31:14 (UTC+0)
Ricordisamoa
subscribed.
jeremyb
subscribed.
Mar 9 2015, 7:16 AM
2015-03-09 07:16:52 (UTC+0)
Krinkle
mentioned this in
T93483: Add script_path to meta_p.wiki database
Mar 21 2015, 8:18 PM
2015-03-21 20:18:01 (UTC+0)
scfc
mentioned this in
T106989: Provide and maintain "pattern" to extract configuration information from operations/mediawiki-config
Jul 26 2015, 6:35 PM
2015-07-26 18:35:30 (UTC+0)
Log In to Comment
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct.
Wikimedia Foundation
Code of Conduct
Disclaimer
CC-BY-SA
GPL
Credits