Help:Cloud Services introduction - Wikitech
Jump to content
From Wikitech
Poster-format overview
Wikimedia Cloud Services (WMCS)
provides tools, services, and support for technical collaborators who want to contribute to Wikimedia software projects. Use Cloud Services to host your software tools for the
Wikimedia movement
, without charge.
What can you do with Cloud Services?
Host tools on Wikimedia servers
Tools and bots make it easier to edit and maintain Wikimedia projects. For developers who support Wikimedia projects by developing tools and bots,
Toolforge
provides the following features:
Free, reliable, and scalable shared hosting, including web servers, databases and other data storage
A distributed job processing system
Support for multiple users to collaboratively maintain and manage tools
To use Toolforge you need:
Some programming knowledge
An understanding of Unix command line
To get started, visit
Help:Toolforge
. Or,
learn more about creating bots
Run scripts and visualize data
PAWS
is a Jupyter notebook installation hosted by Wikimedia. PAWS notebooks can be used for creating tutorials, running live code, creating data visualizations, running basic bots, and more.
A single PAWS notebook is maintained by a single user, but they can be downloaded and forked by other users. To use PAWS you need only a Wikimedia login and a web browser. Knowledge of Python is helpful, but not required.
Administer servers for software development
Open source software projects help the Wikimedia movement by improving core infrastructure (like MediaWiki), powering research and analytics, and supporting Wikimedia operations and software development. For advanced projects that aren't viable in the Toolforge environment,
Cloud VPS
(Virtual Private Server) provides the following features:
Free cloud computing environment, powered by
OpenStack
Collaboratively-owned collections of virtual private servers, storage, firewall, and HTTPS proxy resources to projects
Access to a variety of data services
Freedom to install packages not provided by Debian or the Wikimedia Foundation
To use Cloud VPS, you need:
An open source project that isn't viable in the Toolforge environment, or can't be accomplished using other WMCS offerings
One or more active project maintainers who meet basic requirements
Advanced programming knowledge
Advanced experience with Unix command line
The ability to administer your own servers and manage your project's applications, data, runtime, middleware, and operating systems
To get started, visit
Help:Cloud VPS
What is the difference between Cloud VPS and Toolforge?
Cloud VPS is an
Infrastructure as a service (IaaS)
solution. It provides virtual machines, storage, firewall, and HTTPS proxy resources to projects. The members of each individual project are responsible for managing applications, data, runtime, middleware, and operating systems themselves. Cloud VPS projects use the domain
wmcloud.org
(some older projects use the legacy domain
wmflabs.org
).
Toolforge is a
Platform as a service (PaaS)
solution. It provides
web servers
databases
, and a
distributed job processing system
as managed services for tool maintainers. Toolforge tools use the domain
toolforge.org
Access databases and data dumps
What is a wiki replica? What's in the dumps? Learn the basics of Wikimedia open data and how to access all available datasets at
Research:Data
Access wiki databases for tool development
Tools and software hosted on Toolforge and Cloud VPS can directly access public data dumps and production wiki replicas.
Learn about
accessing wiki replica databases from a tool account
Learn about
accessing dumps through shared storage services for Cloud VPS and Toolforge
Query wiki replicas and dumps in a browser
PAWS
provides a Jupyter notebook environment you can use to query wiki replicas and dumps, create interactive graphs, and use APIs for analysis.
Superset
and
Quarry
are web interfaces for querying live replica SQL databases of public Wikimedia wikis.
To use Superset or Quarry you need only a Wikimedia login and a web browser, but you should have a basic understanding of SQL. Learn about
SQL queries
Get Exclusive Wikimedia Enterprise API Access
The
Wikimedia Enterprise APIs
are read-only APIs providing high-volume access to Wikimedia project data. Learn more on the
Enterprise APIs page
Following the
Wikimedia Enterprise Principle
of
Access for All
, Wikimedia community members get exclusive access to the same endpoints and features as
paid accounts
. From within
PAWS
Cloud VPS
, or
Toolforge
, make your calls to any endpoint described in the Enterprise API documentation
without passing an authorization header
. More ways of accessing the Wikimedia Enterprise APIs can be found at
metawiki:Wikimedia Enterprise#Access
Which service is right for you?
Activity / Needs
PAWS
Superset
Toolforge
Cloud VPS
Data as a service
Data as a service
Platform as a service
Infrastructure as a service
Write scripts and visualize data
Write queries against wiki replica databases
✔ via Toolforge
Access on wiki database dump files
Write and run bots
✔ if not viable on Toolforge
Run web services
✔ if not viable on Toolforge
Build tools to improve Wikimedia projects
✔ if not viable on Toolforge
Schedule or run continuous jobs
✔ if not viable on Toolforge
Work with co-maintainers and co-admins
Need your own web domain
Install
apt packages
not provided by Debian or the Wikimedia Foundation
Administer your own virtual server
Retrieve Wikimedia project data with high QPS using Wikimedia Enterprise APIs
Platform / Environment
web browser
web browser
terminal
terminal
User knowledge
curious—advanced
curious—advanced
intermediate—advanced
advanced
Before you start
To use Cloud Services products, you must first
create a Wikimedia account and a developer account
🎬 Video:
Wikimedia Cloud Services introduction
(2019)
📣 Slides:
An introduction to Cloud Services presentation
(2019)
Communication and support
Support and administration of the WMCS resources is provided by the
Wikimedia Foundation Cloud Services team
and
Wikimedia movement volunteers
. Please reach out with questions and join the conversation:
Discuss and receive general support
Chat in real time in the
IRC channel
#wikimedia-cloud
connect
or the bridged
Telegram group
Discuss via email after you have subscribed to the
cloud@
mailing list
Stay aware of critical changes and plans
Subscribe to the
cloud-announce@
mailing list
(all messages are also mirrored to the
cloud@
list)
Read the
News
wiki page
Track work tasks and
report bugs
Use a subproject of the
#Cloud-Services
Phabricator
project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read stories and WMCS blog posts
Read the
Cloud Services Blog
(for the broader Wikimedia movement, see the
Wikimedia Technical Blog
Retrieved from "
Categories
Overviews
Cloud Services
Help
Cloud Services introduction
Add topic