News/2019 Toolforge Trusty deprecation - Wikitech
Jump to content
From Wikitech
News
(Redirected from
News/Toolforge Trusty deprecation
Toolforge
Cloud Services overview
Toolforge user docs
Toolforge changelog
Get started
Quickstart: set up and get access
How Toolforge works
Rules you must follow
Tutorials
Build and run tools
Navigate tool accounts and files
Build container images for tools
Run a web service
Schedule and manage jobs
Manage tool runtime configuration (envvars)
Deploy your tool on every push (beta)
Language-specific details:
Python
Pywikibot
Node.js
PHP
...more languages/frameworks
Use Redis for caching
Index content with Elasticsearch
Access shared storage and databases
Access shared storage and public wiki dumps
Access the Wiki Replicas databases
Access replica search indices
Manage
tool databases
Sending and receiving email
as tools
Share and maintain tools
Set up version control and code review
Develop successful tools
Find and share tools on Toolhub
Delete a tool
Get help
How and where to get help
Troubleshooting
Contribute to Toolforge
Useful links
Toolforge admin docs
List of tools
Toolforge Admin Console (toolsadmin)
Toolforge API
edit
This page details information about deprecating and removing hosts running
Ubuntu Trusty
(14.04) as an operating system from the
Toolforge
infrastructure. The login bastions and
Grid
execution hosts are still running Trusty and must be replaced with new instances.
The Ubuntu Trusty job grid
was shutdown on Monday 2019-03-25
. Migration steps following the shutdown have changed slightly, so be sure to read them.
What is changing?
New
job grid running Son of Grid Engine on Debian Stretch instances
New
limits on concurrent job execution
and job submission by a single tool
New
bastion hosts running Debian Stretch
with connectivity to the new job grid
New
versions of PHP, Python2, Python3, and other language runtimes
New versions of various support libraries
Timeline
Done
2019-01-11: Availability of Debian Stretch grid announced to community
Done
Week of 2019-02-04: Weekly reminders via email to tool maintainers for tools still running on Trusty
Week of 2019-03-04:
Done
Daily reminders via email to tool maintainers for tools still running on Trusty
Done
Switch
login.tools.wmflabs.org
to point to Stretch bastion
Done
2019-03-25: Shutdown Trusty grid
What should I do?
SSH to the Stretch bastion
login.tools.wmflabs.org
connects to the new Debian Stretch bastion.
Move a grid engine webservice
When possible, we recommend migrating web services to
Kubernetes
instead of the new grid:
:# Connect to the Stretch bastion
ssh
:# Become your tool account
become
YOUR_TOOL
:# Start the webservice as a Kubernetes container rather than a grid job
:#
webservice
--backend
kubernetes
start
:# -- OR --
:# Start the webservice as a Stretch grid job
:#
webservice
--backend
gridengine
start
See
Help:Toolforge/Web#Backends
for more information on migrating from grid engine to Kubernetes.
Python2 and Python3 webservices will need to
rebuild their virtualenv environments
on the new target runtime (Stretch grid or Kubernetes).
NodeJS webservices will need to rebuild their $HOME/www/js/node_modules on the new target runtime (Stretch grid or Kubernetes).
Move a continuous job
:# Connect to the Stretch bastion
ssh
:# Become your tool account
become
YOUR_TOOL
:# Start your job on the Stretch job grid
jstart
...
The exact commands needed to start each continuous job vary greatly from tool to tool. This would be a great time to make a page of reference material for yourself and other maintainers here on Wikitech in the
Tool
namespace and using the
Tool template
if you haven't already.
Move a cron job
The crontab data for all tools which still had a cron registered on the Trusty grid were backed up to
$HOME/crontab.trusty.save
before the Trusty cron server was shutdown. This backup can be used to setup your crontab on the Stretch grid.
:# Connect to the Stretch bastion
ssh
:# Become your tool account
become
YOUR_TOOL
:# Load the backup of your crontab on the Stretch job grid
crontab
$HOME
/crontab.trusty.save
If your workload permits, please avoid scheduling cronjobs from midnight to 3am so you're not competing with other cronjobs for system resources. That time window is currently very crowded.
What are the primary changes with moving to Stretch?
Language runtime and library versions
The vast majority of the language runtimes and libraries installed on the grid nodes are upgraded in Stretch.
Runtime
Trusty Version
Stretch Version
Python3
3.4.0
3.5.3
PHP
5.5.9
7.2
Python2
2.7.5
2.7.13
NodeJS
0.10.25
8.11.1
Perl
5.18.2
5.24.1
Java
1.7.0
11.0.1
Ruby
1.9.3
2.3.3
Mono
5.12.0
5.12.0
TCL
8.6.1
8.6.0
3.2.3
3.3.3
Also note that the system-installed phpunit is not going to be present due to lack of current packages for recent versions of PHP. To use phpunit, please install via composer (instructions for setting up composer are included here
Help:Toolforge#Installing_MediaWiki_core
A table of the primary packages that users are likely to notice changes in is below.
Trusty vs Stretch package version comparison
Package
Trusty Version
Stretch Version
fonts-arabeyes
2.1-3
2.1-3
fonts-arphic-ukai
0.2.20080216.2-4ubuntu2
0.2.20080216.2-4
fonts-arphic-uming
0.2.20080216.2-7ubuntu2
0.2.20080216.2-7
fonts-farsiweb
0.4.dfsg-12
0.4.dfsg-12
fonts-kacst
2.01+mry-10
2.01+mry-12
fonts-khmeros
5.0-7ubuntu1
5.0-7
fonts-lao
0.0.20060226-9
0.0.20060226-9
fonts-liberation
1.07.3-3
1:1.07.4-2
fonts-linuxlibertine
5.3.0-2
5.3.0-2
fonts-manchufont
2.007.svn0068-4
2.007.svn0068-4
fonts-nafees
1.2-5ubuntu1
1.2-5
fonts-sil-abyssinica
1.200-6
1.500-1
fonts-sil-ezra
2.51-8
2.51-8
fonts-sil-padauk
2.80-2
2.80-2
fonts-sil-scheherazade
1.001-8
2.100-1
fonts-takao-gothic
003.02.01-9ubuntu2
00303.01-1
fonts-takao-mincho
003.02.01-9ubuntu2
00303.01-1
fonts-thai-tlwg
1:0.5.1-3
1:0.6.3-1
fonts-tibetan-machine
1.901b-5
1.901b-5
fonts-unfonts-core
1.0.3.is.1.0.2-080608-10ubuntu1
1.0.2-080608-13
fonts-unfonts-extra
1.0.3.is.1.0.2-080608-4ubuntu2
1.0.2-080608+dfsg-6
texlive-fonts-recommended
2013.20140215-1ubuntu0.1
2016.20170123-5
texlive-full
2013.20140215-1ubuntu0.1
2016.20170123-5
ttf-alee
13
13.1
ttf-ubuntu-font-family
0.80-0ubuntu6
0.80-0ubuntu6+wmf1
ttf-wqy-zenhei
0.9.45-5ubuntu1
0.9.45-6
xfonts-100dpi
1:1.0.3
1:1.0.4+nmu1
xfonts-75dpi
1:1.0.3
1:1.0.4+nmu1
xfonts-base
1:1.0.3
1:1.0.4+nmu1
xfonts-mplus
2.2.4-1
1:2.2.4-2
xfonts-scalable
1:1.0.3-1
1:1.0.3-1.1
fonts-sil-nuosusil
2.1.1-7
2.1.1-8
culmus
0.130-1
0.130-1
culmus-fancy
0.0.20051018-3
0.0.20051018-3
fonts-lklug-sinhala
0.6-3
0.6-3
fonts-vlgothic
20130607-1ubuntu1
20141206-3
fonts-dejavu-core
2.34-1ubuntu1
2.37-1
fonts-dejavu-extra
2.34-1ubuntu1
2.37-1
fonts-lyx
2.0.8.1-0ubuntu1
2.2.2-1
fonts-crosextra-carlito
20130920-1
20130920-1
fonts-crosextra-caladea
20130214-1
20130214-1
fonts-smc
6.0-1
6.0-1
fonts-hosny-amiri
0.107-1
0.107-1
fonts-taml-tscu
1.0-2
1.0-2
qpdf
8.0.2-3~14.04.1
6.0.0-2
pngquant
2.0.1-1
2.5.0-2
unpaper
0.4.2-1
6.1-2+b1
ttf-bengali-fonts
1:0.5.14ubuntu1
ttf-devanagari-fonts
1:0.5.14ubuntu1
ttf-gujarati-fonts
1:0.5.14ubuntu1
ttf-kannada-fonts
1:0.5.14ubuntu1
ttf-oriya-fonts
1:0.5.14ubuntu1
ttf-punjabi-fonts
1:0.5.14ubuntu1
ttf-tamil-fonts
1:0.5.14ubuntu1
ttf-telugu-fonts
1:0.5.14ubuntu1
ttf-kochi-gothic
20030809-15
ttf-kochi-mincho
20030809-15
fonts-mgopen
1.1-9
fonts-beng
2:1.2
fonts-deva
2:1.2
fonts-gujr
2:1.2
fonts-knda
2:1.2
fonts-mlym
2:1.2
fonts-orya
2:1.2
fonts-guru
2:1.2
fonts-taml
2:1.3
fonts-telu
2:1.2
fonts-gujr-extra
1.0-5
fonts-noto-cjk
1:1.004+repack2-2
fonts-sil-lateef
1.001-1
fonts-ipafont-gothic
00303-16
fonts-ipafont-mincho
00303-16
language-pack-ar
1:14.04+20160720
language-pack-bn
1:14.04+20160720
language-pack-ca
1:14.04+20160720
language-pack-de
1:14.04+20160720
language-pack-en
1:14.04+20160720
language-pack-es
1:14.04+20160720
language-pack-eu
1:14.04+20160720
language-pack-fr
1:14.04+20160720
language-pack-he
1:14.04+20160720
language-pack-hi
1:14.04+20160720
language-pack-ja
1:14.04+20160720
language-pack-ko
1:14.04+20160720
language-pack-mr
1:14.04+20160720
language-pack-nl
1:14.04+20160720
language-pack-pa
1:14.04+20160720
language-pack-pt
1:14.04+20160720
language-pack-ru
1:14.04+20160720
language-pack-uk
1:14.04+20160720
language-pack-zh-hans
1:14.04+20160720
language-pack-zh-hant
1:14.04+20160720
gcj-jre
4:4.8.2-1ubuntu6
4:6.3.0-4
golang
2:1.3.3-1ubuntu4~ubuntu14.04.1
2:1.7~5
luarocks
2.0.9-1
2.4.2+dfsg-1
mono-complete
5.12.0.226-0xamarin3+ubuntu1404b1
5.12.0.226-0xamarin3+debian9b1
mono-fastcgi-server
4.5-0xamarin3+ubuntu1404b1
4.5-0xamarin3+debian9b1
mono-vbnc
4.7-0xamarin1+ubuntu1404b1
4.7-0xamarin1+debian9b1
npm
1.3.10~dfsg-1
5.8.0
nodejs
0.10.25~dfsg2-2ubuntu1+wmf1
8.11.1~dfsg-2~bpo9+1
openjdk-7-jre-headless
7u181-2.6.14-0ubuntu0.3
openjdk-11-jre-headless
11.0.1+13-2~bpo9+1
icedtea-7-jre-jamvm
7u181-2.6.14-0ubuntu0.3
r-base
3.2.3-1trusty0
3.3.3-1
ruby
1.9.3.484-2ubuntu1.13
2.3.3
tcl8.6
8.6.1-4ubuntu1
8.6.0+9
libberkeleydb-perl
0.54-1
0.55-1+b2
libbot-basicbot-perl
0.89-1
0.89-1
libbsd-resource-perl
1.2907-1build1
1.2910-1+b1
libcache-memcached-fast-perl
0.21-1build1
0.23-1+b2
libcgi-fast-perl
5.18.2-2ubuntu1.7
1:2.12-1
libclass-data-inheritable-perl
0.08-2
0.08-2
libcommon-sense-perl
3.72-2build1
3.74-2
libcrypt-gcrypt-perl
1.26-1build1
1.26-5+b1
libcrypt-openssl-bignum-perl
0.04-4build1
0.07-2
libcrypt-openssl-rsa-perl
0.28-1build1
0.28-5
libdata-compare-perl
1.22-0.1
1.25-0.1
libdata-dumper-simple-perl
0.11-4
0.11-5
libdatetime-format-duration-perl
1.03a-1
1.03a-1.2
libdatetime-format-strptime-perl
1.5400-1
1.7200-1
libdbd-mysql-perl
4.025-1ubuntu0.1
4.041-2
libdbd-sqlite2-perl
2:0.36-1build1
2:0.37-1+b2
libdbd-sqlite3-perl
1.40-3
1.54-1
libdbi-perl
1.630-1
1.636-1+b1
libdigest-crc-perl
0.18-1build1
0.21-1+b2
libdigest-hmac-perl
1.03+dfsg-1
1.03+dfsg-1
libfile-nfslock-perl
1.21-1
1.27-1
libgd-gd2-perl
2.46-3.1build1
2.53-3
libhtml-format-perl
2.11-1
2.12-1
libhtml-html5-entities-perl
0.003-2
0.004-1
libhtml-parser-perl
3.71-1build1
3.72-3
libhtml-template-perl
2.95-1
2.95-2
libhttp-message-perl
6.06-1
6.11-1
libimage-exiftool-perl
9.46-1
10.40-1
libio-socket-ssl-perl
1.965-1ubuntu1
2.044-1
libipc-run-perl
0.92-1
0.94-1+deb9u1
libirc-utils-perl
0.12-1
0.12-1
libjson-perl
2.61-1
2.90-1
libjson-xs-perl
2.340-1build1
3.030-1
liblog-log4perl-perl
1.41-1.1ubuntu1
1.48-1+deb9u1
liblwp-protocol-https-perl
6.04-2ubuntu0.1
6.06-2
libmediawiki-api-perl
0.39-1
0.41-1
libmediawiki-bot-perl
5.005006-1
5.006003-1
libnet-netmask-perl
1.9016-1
1.9022-1
libnet-oauth-perl
0.28-1
0.28-2
libnet-ssleay-perl
1.58-1
1.80-1
libnetaddr-ip-perl
4.071+dfsg-1
4.079+dfsg-1+b1
libobject-pluggable-perl
1.29-1
1.29-1
libparse-mediawikidump-perl
1.0.6-1
1.0.6-2
libpod-simple-wiki-perl
0.15-1
0.20-1
libpoe-component-irc-perl
6.83+dfsg-1
6.88+dfsg-1
libpoe-component-syndicator-perl
0.06-1
0.06-1
libpoe-filter-ircd-perl
2.44-1
2.44-1
libpoe-perl
2:1.3580-1
2:1.3670-1
libppix-regexp-perl
0.036-1
0.050-1
libreadonly-perl
1.04-1
2.050-1
libredis-perl
2:1.9670-1
2:1.9910-1
libregexp-common-perl
2013031301-1
2016060801-1
libsocket-getaddrinfo-perl
0.22-1build1
0.22-3
libsort-fields-perl
0.90-2
0.90-2
libstring-diff-perl
0.06-1
0.07-2
libtask-weaken-perl
1.04-1
1.04-1
libtest-exception-perl
0.32-1
0.43-1
libtext-diff-perl
1.41-1
1.44-1
libtimedate-perl
2.3000-1
2.3000-2
liburi-encode-perl
0.090-1
1.1.1-1
liburi-perl
1.60-1
1.71-1
libwww-mechanize-perl
1.73-1
1.83-1
libwww-perl
6.05-2
6.15-1
libxml-libxml-perl
2.0108+dfsg-1ubuntu0.2
2.0128+dfsg-1+deb9u1
libxml-parser-perl
2.41-1build3
2.44-2+b1
libxml-simple-perl
2.20-1
2.22-1
libxml-xpathengine-perl
0.13-1
0.13-1
perl-modules
5.18.2-2ubuntu1.7
5.24.1-3+deb9u5
python-flask-oauth
0.12-1
python-mwparserfromhell
0.3.3-1
0.4.2-1.1+b1
python-oursql
0.9.3.1-1
python-socketio-client
0.5.3-1
0.6.5-0.1
python-wikitools
1.1.1-1
python-mwclient
0.8.4-1
python-pymysql
0.6.2-2
0.7.10-1
python3-pymysql
0.7.2-1
0.7.10-1
php5-cli
5.5.9+dfsg-1ubuntu4.26
php5-common
5.5.9+dfsg-1ubuntu4.26
php5-curl
5.5.9+dfsg-1ubuntu4.26
php5-gd
5.5.9+dfsg-1ubuntu4.26
php5-imagick
3.1.2-1build1
php5-intl
5.5.9+dfsg-1ubuntu4.26
php5-mcrypt
5.4.6-0ubuntu5
php5-mysqlnd
5.5.9+dfsg-1ubuntu4.26
php5-pgsql
5.5.9+dfsg-1ubuntu4.26
php5-redis
2.2.4-1build2
php5-sqlite
5.5.9+dfsg-1ubuntu4.26
php5-xdebug
2.2.3-2build1
php5-xhprof
0.9.4-1build1
php5-xsl
5.5.9+dfsg-1ubuntu4.26
php5-readline
5.5.9+dfsg-1ubuntu4.26
php7.2-cli
7.2.13
php7.2-common
7.2.13
php7.2-curl
7.2.13
php7.2-gd
7.2.13
php-imagick
3.4.3~rc2-2
php7.2-intl
7.2.13
php7.2-mysql
7.2.13
php7.2-pgsql
7.2.13
php-redis
4.1.1
php7.2-sqlite3
7.2.13
php-xdebug
2.7.0
php7.2-xml
7.2.13
php7.2-readline
7.2.13
ttf-indic-fonts-core
1:0.5.14ubuntu1
fonts-indic
2:1.2
texlive-fonts-extra
2013.20140215-2
2016.20170123-5
mysqltcl
3.051-1
3.052-2
tcl-tls
1.6+dfsg-3
1.6.7+dfsg-1.2
tcl-trf
2.1.4-dfsg2-1
2.1.4-dfsg3-2+b1
tclcurl
7.22.0-1
7.22.0+hg20160822-1
tcllib
1.15-dfsg-2
1.18-dfsg-3
tdom
0.8.3-1
0.8.3-1
tesseract-ocr
3.03.02-3
3.04.01-5
tesseract-ocr-afr
3.02-2
3.04.00-1
tesseract-ocr-ara
3.02-2
3.04.00-1
tesseract-ocr-aze
3.02-2
3.04.00-1
tesseract-ocr-bel
3.02-2
3.04.00-1
tesseract-ocr-ben
3.02-2
3.04.00-1
tesseract-ocr-bul
3.02-2
3.04.00-1
tesseract-ocr-cat
3.02-2
3.04.00-1
tesseract-ocr-ces
3.02-2
3.04.00-1
tesseract-ocr-chi-sim
3.02-1
3.04.00-1
tesseract-ocr-chi-tra
3.02-1
3.04.00-1
tesseract-ocr-chr
3.02-2
3.04.00-1
tesseract-ocr-dan
3.02-2
3.04.00-1
tesseract-ocr-deu
3.02-2
3.04.00-1
tesseract-ocr-deu-frak
3.02-4
3.04.00-1
tesseract-ocr-ell
3.02-2
3.04.00-1
tesseract-ocr-eng
3.02-2
3.04.00-1
tesseract-ocr-enm
3.02-2
3.04.00-1
tesseract-ocr-epo
3.02-2
3.04.00-1
tesseract-ocr-equ
3.02-2
3.04.00-1
tesseract-ocr-est
3.02-2
3.04.00-1
tesseract-ocr-eus
3.02-2
3.04.00-1
tesseract-ocr-fin
3.02-2
3.04.00-1
tesseract-ocr-fra
3.02-2
3.04.00-1
tesseract-ocr-frk
3.02-2
3.04.00-1
tesseract-ocr-frm
3.02-2
3.04.00-1
tesseract-ocr-glg
3.02-2
3.04.00-1
tesseract-ocr-heb
3.02-2
3.04.00-1
tesseract-ocr-hin
3.02-2
3.04.00-1
tesseract-ocr-hrv
3.02-2
3.04.00-1
tesseract-ocr-hun
3.02-2
3.04.00-1
tesseract-ocr-ind
3.02-2
3.04.00-1
tesseract-ocr-isl
3.02-2
3.04.00-1
tesseract-ocr-ita
3.02-2
3.04.00-1
tesseract-ocr-ita-old
3.02-1
3.04.00-1
tesseract-ocr-jpn
3.02-2
3.04.00-1
tesseract-ocr-kan
3.02-2
3.04.00-1
tesseract-ocr-kor
3.02-2
3.04.00-1
tesseract-ocr-lav
3.02-2
3.04.00-1
tesseract-ocr-lit
3.02-2
3.04.00-1
tesseract-ocr-mal
3.02-2
3.04.00-1
tesseract-ocr-mkd
3.02-2
3.04.00-1
tesseract-ocr-mlt
3.02-2
3.04.00-1
tesseract-ocr-msa
3.02-2
3.04.00-1
tesseract-ocr-nld
3.02-2
3.04.00-1
tesseract-ocr-nor
3.02-2
3.04.00-1
tesseract-ocr-osd
3.02-2
3.04.00-1
tesseract-ocr-pol
3.02-2
3.04.00-1
tesseract-ocr-por
3.02-2
3.04.00-1
tesseract-ocr-ron
3.02-2
3.04.00-2
tesseract-ocr-rus
3.02.02-1
3.04.00-1
tesseract-ocr-slk
3.02-2
3.04.00-1
tesseract-ocr-slk-frak
3.02-2
3.04.00-1
tesseract-ocr-slv
3.02-2
3.04.00-1
tesseract-ocr-spa
3.02-2
3.04.00-1
tesseract-ocr-spa-old
3.02-1
3.04.00-1
tesseract-ocr-sqi
3.02-2
3.04.00-1
tesseract-ocr-srp
3.02-2
3.04.00-1
tesseract-ocr-swa
3.02-2
3.04.00-1
tesseract-ocr-swe
3.02-2
3.04.00-1
tesseract-ocr-tam
3.02-2
3.04.00-1
tesseract-ocr-tel
3.02-2
3.04.00-1
tesseract-ocr-tgl
3.02-2
3.04.00-1
tesseract-ocr-tha
3.02-2
3.04.00-1
tesseract-ocr-tur
3.02-2
3.04.00-1
tesseract-ocr-ukr
3.02-2
3.04.00-1
tesseract-ocr-vie
3.02.02-1
3.04.00-1
bison
2:3.0.2.dfsg-2
2:3.0.4.dfsg-1+b1
calibre
1.25.0+dfsg-1ubuntu1.2
2.75.1+dfsg-1
csh
20110502-2ubuntu2
20110502-2.2+b1
csvtool
1.3.2-1
1.5-1+b1
dc
1.06.95-8ubuntu1
1.06.95-9+b3
djvulibre-bin
3.5.25.4-3
3.5.27.1-7
djvulibre-plugin
4.9-4ubuntu3
4.10.6-1
doxygen
1.8.6-2
1.8.13-4+b1
doxygen-latex
1.8.6-2
1.8.13-4
expect
5.45-5ubuntu1
5.45-7+deb9u1
fabric
1.8.2-1
1.13.1-2
gawk
1:4.0.1+dfsg-2.1ubuntu2
1:4.1.4+dfsg-1
gdal-bin
1.10.1+dfsg-5ubuntu1
2.1.2+dfsg-5
git-review
1.23-1
1.25.0-2
git-svn
1:1.9.1-1ubuntu0.10
1:2.11.0-3+deb9u4
gnuplot-nox
4.6.4-2
5.0.5+dfsg1-6+deb9u1
graphicsmagick
1.3.18-1ubuntu3.1
1.3.30+hg15796-1~deb9u2
graphviz
2.36.0-0ubuntu3.2
2.38.0-17
grep
2.16-1
2.27-2
hugin-tools
2013.0.0+dfsg-1ubuntu2
2016.2.0+dfsg-1
hunspell
1.3.2-6ubuntu2.1
1.4.1-2+b2
inkscape
0.48.4-3ubuntu2
0.92.1-1
iotop
0.6-1
0.6-2
ksh
93u+20120801-1ubuntu0.14.04.1
93u+20120801-3.1
lame
3.99.5+repack1-3ubuntu1
3.99.5+repack1-9+b2
libaio1
0.3.109-4
0.3.110-3
libav-tools
6:9.20-0ubuntu0.14.04.1
7:3.2.12-1~deb9u1
libdmtx0a
0.7.4-2
0.7.4-2
libfcgi0ldbl
2.4.0-8.1ubuntu5
2.4.0-8.4+b1
libffi-dev
3.1~rc1+r3.0.13-12ubuntu0.2
3.2.1-6
libfreetype6
2.5.2-1ubuntu2.8
2.6.3-3.2
libgdbm-dev
1.8.3-12build1
1.8.3-14
libgeoip1
1.6.0-1
1.6.9-4
libhunspell-dev
1.3.2-6ubuntu2.1
1.4.1-2+b2
libjpeg-turbo-progs
1.3.0-0ubuntu2.1
1:1.5.1-2
libmpfr4
3.1.2-1
3.1.5-1
libncurses5-dev
5.9+20140118-1ubuntu1
6.0+20161126-1+deb9u2
libneon27-gnutls
0.30.0-1ubuntu1
0.30.2-2
libnfnetlink0
1.0.1-2
1.0.1-3
libnspr4
2:4.13.1-0ubuntu0.14.04.1
2:4.12-6
libnss3
2:3.28.4-0ubuntu0.14.04.3
2:3.26.2-1.1+deb9u1
libnss3-1d
2:3.28.4-0ubuntu0.14.04.3
libotf0
0.9.13-1ubuntu1
0.9.13-3+b1
libpcsclite1
1.8.10-1ubuntu1.1
1.8.20-1
libpng3/libpng16-16
1.2.50-1ubuntu2.14.04.3
1.6.28-1
libproj0/libproj12
4.8.0-2ubuntu2
4.9.3-1
libquadmath0
4.8.4-2ubuntu1~14.04.4
6.3.0-18+deb9u1
librsvg2-bin
2.40.2-1+wm2
2.40.16-1+b1
libsvn1
1.8.8-1ubuntu3.3
1.9.5-1+deb9u3
libvips-tools
7.38.5-2
8.4.5-1+b1
libxml2-utils
2.9.1+dfsg1-3ubuntu4.13
2.9.4+dfsg1-2.2+deb9u2
libyaml-dev
0.1.4-3ubuntu3.1
0.1.7-2
libzbar0
0.10+doc-9build1
0.10+doc-10.1
mailutils
1:2.99.98-1.1
1:3.1.1-1
mdbtools
0.7.1-1ubuntu1
0.7.1-5
melt
0.9.0-3
6.4.1-4
mktorrent
1.0-4
1.0-4+b2
openbabel
2.3.2+dfsg-1.1
2.3.2+dfsg-3
p7zip-full
9.20.1~dfsg.1-4+deb7u3build0.14.04.1
16.02+dfsg-3+deb9u1
pdf2svg
0.2.2-1
0.2.2-1+b1
pdf2djvu
0.7.17-3ubuntu2
0.9.4-1+b3
pdftk
2.01-1
2.02-4+b2
phantomjs
1.9.0-1
2.1.1+dfsg-2
phpunit
3.7.28-1
poppler-utils
0.24.5-2ubuntu4.14
0.48.0-2+deb9u2
postgis
2.1.2+dfsg-2ubuntu0.1
2.3.1+dfsg-2
postgresql-client
9.3+154ubuntu1.1
9.6+181+deb9u2
pstoedit
3.62-1ubuntu1
3.70-3+b2
rrdtool
1.4.7-2ubuntu5
1.6.0-1+b2
socat
1.7.2.3-1
1.7.3.1-2+deb9u1
sqlite3
3.8.2-1ubuntu2.1
3.16.2-5+deb9u1
supybot
0.83.4.1.ds-2
0.83.4.1.ds-3
tabix
0.2.6-2
1.3.2-2
texinfo
5.2.0.dfsg.1-2
6.3.0.dfsg.1-1+b2
texlive
2013.20140215-1ubuntu0.1
2016.20170123-5
ufraw-batch
0.19.2-2ubuntu2
0.22-1.1
virtualenvwrapper
4.1.1-1
4.3.1-2
whois
5.1.1
5.2.17~deb9u1
xml2
0.4-3.1
0.4-3.1+deb9u1
xsltproc
1.1.28-2ubuntu0.1
1.1.29-2.1
xvfb
2:1.15.1-0ubuntu2.11
2:1.19.2-1+deb9u5
zbar-tools
0.10+doc-9build1
0.10+doc-10.1
debootstrap
1.0.59ubuntu0.10
1.0.89
fakechroot
2.17.2-1
2.19-3
hhvm
3.12.7+dfsg-1+wmf1~trusty1
3.18.5+dfsg-1+wmf8+deb9u1
libboost-python1.##.0
1.54.0-4ubuntu3.1
1.62.0+dfsg-4
libmpc3
1.0.1-1ubuntu1
1.0.3-1+b2
libprotobuf8/10
2.5.0-9ubuntu1
3.0.0-9
libbytes-random-secure-perl
0.28-1
0.28-1
libvips37/42
7.38.5-2
8.4.5-1+b1
nodejs-legacy
0.10.25~dfsg2-2ubuntu1+wmf1
6.11.0~dfsg-1+wmf5
mariadb-client
5.5.61-1ubuntu0.14.04.1
10.1.37-0+deb9u1
opencv-data
2.4.8+dfsg1-2ubuntu1.2
2.4.9.1+dfsg1-2
python-flake8
2.1.0-1ubuntu1
3.2.1-1
python3-flake8
2.1.0-1ubuntu1
3.2.1-1
tcl-thread
1:2.7.0-1
1:2.8.0-1
python-redis
2.7.2-1build1
2.10.5-2
python
2.7.5-5ubuntu3
2.7.13-2
python3
3.4.0-0ubuntu2
3.5.3-1
python-babel
1.3+dfsg.1-2ubuntu2
2.3.4+dfsg.1-2
python3-babel
1.3+dfsg.1-2ubuntu2
2.3.4+dfsg.1-2
python-beautifulsoup
3.2.1
3.2.1-1
python-bottle
0.12.0-1
0.12.13-1
python3-bottle
0.12.0-1
0.12.13-1
python-celery
3.1.6-1ubuntu1
3.1.23-7
python3-celery
3.1.23-7
python-cffi
0.8.2-0ubuntu1
1.9.1-2
python3-cffi
0.8.2-0ubuntu1
1.9.1-2
python-egenix-mxdatetime
3.2.7-1build1
3.2.9-1
python-egenix-mxtools
3.2.7-1build1
3.2.9-1
python-enum34
0.9.23-1
1.1.6-1
python3-enum34
0.9.23-1
python-flask
0.10.1-2build1
0.12.1-1
python3-flask
0.10.1-2build1
0.12.1-1
python-flask-login
0.2.6-1
python3-flask-login
0.4.0-1
python-flickrapi
1.2-3
2.1.2-5
python3-flickrapi
2.1.2-5
python-flup
1.0.2-4
1.0.2-5
python-gdal
1.10.1+dfsg-5ubuntu1
2.1.2+dfsg-5
python3-gdal
1.10.1+dfsg-5ubuntu1
2.1.2+dfsg-5
python-gdbm
2.7.5-1ubuntu1
2.7.13-1
python3-gdbm
3.4.3-1~14.04.2
3.5.3-1
python-genshi
0.7-3
0.7-5
python3-genshi
0.7-3
0.7-5
python-genshi-doc
0.7-3
0.7-5
python-geoip
1.2.4-2ubuntu5
1.3.2-1+b2
python3-geoip
1.3.2-1+b2
python-gevent
1.1b6-1~trusty1
1.1.2-1
python3-gevent
1.1.2-1
python-gi
3.12.0-1ubuntu1
3.22.0-2
python3-gi
3.12.0-1ubuntu1
3.22.0-2
python-greenlet
0.4.2-1ubuntu0.1
0.4.11-1
python3-greenlet
0.4.2-1ubuntu0.1
0.4.11-1
python-httplib2
0.8-2build1
0.9.2+dfsg-1
python3-httplib2
0.8-2build1
0.9.2+dfsg-1
python-imaging
2.3.0-1ubuntu3.4
4.0.0-4
python3-imaging
2.3.0-1ubuntu3.4
python-ipaddr
2.1.10-1
2.1.11-2
python3-ipaddr
2.1.10-1
python-irclib
0.4.8-1.1
python-keyring
3.5-1
10.1-1
python3-keyring
3.5-1
10.1-1
python-launchpadlib
1.10.2+ds-2
1.10.4-1
python3-launchpadlib
1.10.4-1
python-lxml
3.3.3-1ubuntu0.2
3.7.1-1
python3-lxml
3.3.3-1ubuntu0.2
3.7.1-1
python-magic
1:5.14-2ubuntu3.4
1:5.30-1+deb9u2
python3-magic
1:5.14-2ubuntu3.4
1:5.30-1+deb9u2
python-matplotlib
1.3.1-1ubuntu5.1
2.0.0+dfsg1-2
python3-matplotlib
1.3.1-1ubuntu5.1
2.0.0+dfsg1-2
python-mysql.connector
1.1.6-1
2.1.6-1
python3-mysql.connector
1.1.6-1
2.1.6-1
python-mysqldb
1.2.3-2ubuntu1
1.3.7-1.1
python3-mysqldb
1.3.7-1.1
python-newt
0.52.15-2ubuntu5
0.52.19-1+b1
python3-newt
0.52.15-2ubuntu5
0.52.19-1+b1
python-nose
1.3.1-2
1.3.7-2
python3-nose
1.3.1-2
1.3.7-2
python-numpy
1:1.8.2-0ubuntu0.1
1:1.12.1-3
python3-numpy
1:1.8.2-0ubuntu0.1
1:1.12.1-3
python-opencv
2.4.8+dfsg1-2ubuntu1.2
2.4.9.1+dfsg1-2
python-pandas
0.13.1-2ubuntu2
0.19.2-5.1
python3-pandas
0.13.1-2ubuntu2
0.19.2-5.1
python-pil
2.3.0-1ubuntu3.4
4.0.0-4
python3-pil
2.3.0-1ubuntu3.4
4.0.0-4
python-problem-report
2.14.1-0ubuntu3.29
python3-problem-report
2.14.1-0ubuntu3.29
python-psycopg2
2.4.5-1build5
2.6.2-1
python3-psycopg2
2.4.5-1build5
2.6.2-1
python-pycountry
0.14.1+ds1-3build1
1.8+ds1-0.2
python3-pycountry
1.8+ds1-0.2
python-pydot
1.0.28-0ubuntu1
1.0.28-2
python3-pydot
1.0.28-2
python-pyexiv2
0.3.2-5ubuntu3
0.3.2-8+b3
python-pygments
1.6+dfsg-1ubuntu1.1
2.2.0+dfsg-1
python3-pygments
1.6+dfsg-1ubuntu1.1
2.2.0+dfsg-1
python-pyicu
1.5-2ubuntu4
1.9.5-1
python3-pyicu
1.5-2ubuntu4
python-pyinotify
0.9.4-1build1
0.9.6-1
python3-pyinotify
0.9.4-1build1
0.9.6-1
python-requests
2.2.1-1ubuntu0.4
2.12.4-1
python3-requests
2.2.1-1ubuntu0.4
2.12.4-1
python-requests-oauthlib
0.4.0-1
0.7.0-0.1
python3-requests-oauthlib
0.4.0-1
0.7.0-0.1
python-rsvg
2.32.0+dfsg-3
2.32.0+dfsg-3
python-scipy
0.13.3-1build1
0.18.1-2
python3-scipy
0.13.3-1build1
0.18.1-2
python-sqlalchemy
0.8.4-1build1
1.0.15+ds1-1
python3-sqlalchemy
0.8.4-1build1
1.0.15+ds1-1
python-svn
1.7.8-0.2
1.9.4-2
python3-svn
1.9.4-2
python-tk
2.7.5-1ubuntu1
2.7.13-1
python3-tk
3.4.3-1~14.04.2
3.5.3-1
python-twisted
13.2.0-1ubuntu1.2
16.6.0-2
python3-twisted
16.6.0-2
python-twitter
1.1+git20131227-2
1.1+git20131227-2
python-unicodecsv
0.9.4-1
0.14.1-1
python3-unicodecsv
0.14.1-1
python-unittest2
0.5.1-1ubuntu1
1.1.0-6.1
python3-unittest2
1.1.0-6.1
python-virtualenv
1.11.4-1ubuntu1
15.1.0+ds-1
python3-virtualenv
15.1.0+ds-1
python-wadllib
1.3.2-2build1
1.3.2-3
python3-wadllib
1.3.2-2build1
1.3.2-3
python-webpy
1:0.37+20120626-1
1:0.38-1
python-werkzeug
0.9.4+dfsg-1.1ubuntu2.1
0.11.15+dfsg1-1
python3-werkzeug
0.9.4+dfsg-1.1ubuntu2.1
0.11.15+dfsg1-1
python-zbar
0.10+doc-9build1
0.10+doc-10.1
python-zmq
14.0.1-1build2
16.0.2-2
python3-zmq
14.0.1-1build2
16.0.2-2
python-stdeb
0.6.0+20100620-4ubuntu1
0.8.5-1
python3-stdeb
0.8.5-1
python-coverage
3.7.1
4.2
python3-coverage
3.7.1
4.2
Concurrency limits
Tracked in
Phabricator
Task T67777
Maximum of
16
active jobs simultaneously allowed per tool user
The scheduler will hold additional job submissions in the
qw
(queued/waiting) until an active slot is available.
Maximum of
50
active and queued jobs simultaneously allowed per tool user
The scheduler will reject additional job submissions by exiting with a status code of 25 and writing "Unable to run job: job rejected: only 50 jobs are allowed per user (current job count: 50)" to stderr
Tracked in
Phabricator
Task T123270
Implementing these limits has allowed us enable job submission from the continuous and and task job queues.
Solutions to common problems
Having trouble with the new grid? If the answer to your problem isn't here, ask for help in
#wikimedia-cloud
connect
or
file a bug in Phabricator
Rebuild virtualenv for python users
Since the python executables and libraries are updated in stretch, local virtualenvs will need to be deleted and re-created on the new bastion for anything that runs from those virtualenvs to work. Several errors are likely to be caused by old virtualenvs with one obvious one being an unexpected
ImportError
Using a requirements file may make this simpler in many cases, if your project doesn't already use one. You can create one in your local directory by running
pip freeze > requirements.txt
in your tool folder with your virtualenv activated. Then later on, you can simply use
pip install -r requirements.txt
to install the new environment after you deleted the old virtualenv and created a new one. For more information on this option, see
pip's documentation on requirements files
Example 1: Upgrading a Trusty grid engine based tool to the Stretch grid
Follow these steps if you manually submit jobs using jsub, or if you submit jobs using a crontab.
ssh
become
YOUR_TOOL
rm
-rf
venv
# This will destroy the virtualenv and all libraries, so make sure you know what you will need to install later!
virtualenv
venv
source
venv/bin/activate
pip
install
--upgrade
pip
# upgrade pip itself to avoid problems with older versions
pip
install
...
# Here you'd use the requirements file syntax if you have one, or you'd manually install each needed library.
Example 2: Upgrading a uWSGI webservice into a Kubernetes container
If you are currently running your uWSGI webservice under the Grid Engine backend (i.e.,
webservice uwsgi-python
command
), and you want to upgrade to a uWSGI webservice running under Kubernetes (i.e.,
webservice --backend=kubernetes python
command)
, you should rebuild your virtualenv as follows:
ssh
become
YOUR-TOOL
webservice
--backend
kubernetes
python
stop
webservice
--backend
kubernetes
python
shell
# do not skip this step – setting up the venv directly from the bastion may result in serious performance issues, compare T214086
rm
-rf
www/python/venv/
# this will destroy the virtualenv and all libraries, so make sure you know what you will need to install later!
python3
-m
venv
www/python/venv/
source
www/python/venv/bin/activate
pip
install
--upgrade
pip
# upgrade pip itself to avoid problems with older versions
pip
install
-r
www/python/src/requirements.txt
# assuming your tool has a requirements.txt file
webservice
--backend
kubernetes
python
start
Example 3: Upgrading a Kubernetes uWSGI webservice
If you are already using the Kubernetes backend, there is nothing you need to do -- the container will use the same Debian Jessie-based image as before.
PyYAML fails to install in Debian Stretch Python3 virtualenv
Tracked in
Phabricator
Task T215434
Resolved
The new bastions are using
systemd resource control
to restrict the amount of RAM and CPU resources that a user can consume. We do this to attempt to keep a single user from using all of the shared resources of the bastion accidentally and thus making the bastion slow for everyone. The initial limits we had set were overly restrictive and caused gcc to fail when compiling PyYAML. This has been corrected by increasing the limits.
BotPassword or OAuth grant does not work from new job grid
Bot passwords
and
OAuth registrations
can both include allowed IP range restrictions. The defaults for both are to allow usage from any IPv4 and IPv6 address. If you have changed this when creating the bot password or OAuth consumer registration to restrict access to specific IP address ranges you may have issues using the password or OAuth consumer from the new job grid. The Cloud VPS environment is nearing the end of a process of moving from the
10.0.0.0/8
private address range that is shared with other internal servers operated by the Wikimedia Foundation to a new
172.16.0.0/21
private subnet. The new job grid is the first end-user facing portion of Toolforge to be migrated to the new range.
The allowed IP ranges for
bot passwords
can be changed by the owner of the account using
Special:BotPasswords
. Either add the
172.16.0.0/21
CIDR to the list of allowed ranges or reset them to the defaults of
0.0.0.0/0
and
::/0
The allowed IP ranges for an
OAuth consumer registration
can be changed by the original proposer of the registration using
Special:OAuthConsumerRegistration/list
. Either add the
172.16.0.0/21
CIDR to the list of allowed ranges or reset them to the defaults of
0.0.0.0/0
and
::/0
Lighttpd crashes on startup with message "parser failed somehow near here: (EOL)"
Lighttpd 1.4.40
made overriding keys in an existing array a fatal error
. The Stretch version of lighttpd is 1.4.45. This change in the upstream application makes the advice at
Help:Toolforge/Web/Lighttpd#Header, mimetype, character_encoding, error_handler
for replacing existing mime-type mappings with new local versions obsolete.
Look for a
$HOME/error.log
line similar to
Duplicate array-key '.js'
just prior to the parser failure error message to help you find the entry in your
$HOME/.lighttpd.conf
file that needs to be removed.
'webservice stop' says service is not running, but 'webservice start' says service is running
Tracked in
Phabricator
Task T163355
BryanDavis
has this advice:
webservice stop
rm $HOME/service.manifest
webservice [add other args here as needed] start
It is not completely well understood what causes
webservice
to become confused about the state of the process, but deleting the
service.manifest
file generally seems to fix the issue.
Python: redis.exceptions.ResponseError: value is not an integer or out of range
The Python Redis client made a
breaking change in v3.0.0 vs older versions
in renaming the prior
StrictRedis
class to
Redis
. The new behavior expects a different order of arguments for calls such as
setex()
. The expected order of arguments now matches the Redis protocol docs rather than the more "pythonic" order that the prior implementation used. Typically this means that you need to swap the order of the time and value arguments in your calling code. See
the library documentation
for more breaking changes.
Delete a tool
Tracked in
Phabricator
Task T170355
Resolved
Some tools were experiments that are done, others were made obsolete by other tools, some are just things that the original maintainer is tired of caring for. Maintainers can mark their tools for deletion using the "Disable tool" button on the tool's detail page on
. Disabling a tool will immediately stop any running jobs including webservices and prevent maintainers from logging in as the tool. Disabled tools are archived and deleted after 40 days. Disabled tools can be re-enabled at any time prior to being archived and deleted.
Python 'oursql' package fails to compile
Tracked in
Phabricator
Task T215963
The latest official release of the Python 'oursql' package will not compile against MariaDB client libraries. See upstream bug report at
. Oursql can be installed from a fork maintained at
, but the recommended long term solution is to migrate application code to the
PyMySQL
package instead.
SSH to login-stretch.tools.wmflabs.org fails with 'Permission denied (publickey)'
Tracked in
Phabricator
Task T168433
This is typically an issue with the newer Debian Stretch provided version of
sshd
on the server side refusing to authenticate an insecure or deprecated public key type. Specifically, support for DSA (ssh-dss) keys was deprecated in Openssh 7.0. If your ssh public key starts with the string "ssh-dss" you will be impacted by this. RSA keys smaller than 1024 bits are also deprecated.
First make sure that you are passing a valid key by attempting to ssh to
login-trusty.tools.wmflabs.org
using the same public key and username. If this also fails, the problem is likely something other than the ssh key type. Join us in
#wikimedia-cloud
connect
for interactive debugging help.
If you can ssh to
login-trusty.tools.wmflabs.org
with no errors, your key is probably of an unsupported type. Generate a new ssh key pair and upload the public key using the form at
. We currently recommend using either
ed25519
or
4096-bit RSA
keys. See
Production shell access#Generating your SSH key
for more information.
SSH to login-stretch.tools.wmflabs.org fails with 'Permission denied (publickey,hostbased)'
In case you face this problem, make sure to use the right shell name located on your
User Preferences
called **Instance shell account name**. It's supposed to be used in logging into the Toolforge server when need be, whether Trusty or Stretch.
"Unable to run job: Error reading answer list from qmaster"
Tracked in
Phabricator
Task T218755
Attempting to start a job with a name including non-ASCII characters using jsub, jstart, qcronsub, etc may fail with an error message written to the job's err file like "Unable to run job: Error reading answer list from qmaster". This is a
known bug in Son of Grid Engine
Monitoring tools
Tools running jobs on Trusty grid engine in last 7 days
This report updates once per hour and will not report jobs that have been seen running on the Stretch grid in the same 7 day period.
Report has drill down pages for each maintainer and tool. Examples:
bd808's tools
gridengine-status tool
Webservices that move from the Trusty grid directly to the Kubernetes cluster will not be removed from the report automatically.
Jobs running on the Stretch grid in the last 7 days
Stretch grid status
Why are we doing this?
See also:
News/2018 Trusty deprecation
Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time.
During
past operating system updates
we were able to create a mixed grid which contained hosts running multiple operating systems and control which was used to run each job using command line arguments to
jsub
and
webservice
. The current version of Sun Grid Engine (v6.2u5) that exists in Ubuntu Trusty is incompatible with "Son of" Grid Engine (v8.1.9) from Debian Stretch. Therefore the two grids must be entirely separate environments. Any cron jobs that exist or web services in the old grid (submitted from one of the current bastions) will not currently exist in the new grid. To schedule any job or service on the new Son of Grid Engine grid, one must log into a bastion dedicated to that grid (currently tools-sgebastion-06.tools.eqiad.wmflabs) to submit them.
See also
[Cloud-announce] Toolforge: Trusty deprecation and grid engine migration
[Cloud-announce] Toolforge: Trusty job grid deprecation reminders start today
[Cloud-announce] Toolforge: Bastion changes and Trusty deprecation final steps
Retrieved from "
Category
Toolforge
News/2019 Toolforge Trusty deprecation
Add topic
US