Introducing TKLDev - Turnkey's appliance development and build system in a box | TurnKey GNU/Linux

Introducing TKLDev - Turnkey's appliance development and build system in a box | TurnKey GNU/Linux
You are here
Category: All
Introducing TKLDev - Turnkey's appliance development and build system in a box
Blog Tags:
news
tkldev
github
Introducing TKLDev - Turnkey's appliance development and build system in a box
Alon Swartz
- Thu, 2013/07/18 - 12:56 -
37 comments
Today we proudly announce the official release of TKLDev, the new mothership of all TurnKey Linux appliance development. With TKLDev, building TurnKey Core from scratch is as easy as running
make
. If you can handle that you're ready to roll:
root@tkldev ~$ cd core
/turnkey/fab/products/turnkey/core
root@tkldev turnkey/core$ make
As you might expect, we're pretty stoked to finally (finally!) be able to make this announcement. It's the culmination of a couple of years worth of unglorious work behind the scenes upgrading the development plumbing that makes TurnKey possible.
I personally believe it's also a critically important milestone for TurnKey, a project I've poured my heart and soul into and would like to see grow up to embrace it's full potential.
So what's the big deal with this new build system?
Here's the problem: as much as we'd like to be the omnipresent, omnipotent, and omniscient developers that can do everything ourselves there is way more awesome open source software out there than we (or any other closed group of developers!) could ever hope to bottle up into a working library of ready-to-use solutions. Not without a great deal of help anyhow!
We started this project about five years ago. It took us a couple of years to start realizing we had a problem and another year or so to realize we had a really big problem. T
urnKey was all about open source but it wasn't truly open source itself because the relationship of the core development team with the community was pretty much one sided. We could look out, but nobody could look in.
We tried plugging the hole with
TKLPatch
, at least as a temporary stopgap, but that didn't work and we think part of the reason was that we hadn't gone nearly far enough. It wasn't enough to make it possible for the community to customize appliances. That just made the community into limited second class citizens and while it may have been better than nothing nobody really wants to be a second class citizen in a purported open source project. Give me full access to the source code and build chain or give me death! (or... a casual hrmpf of disinterest).
We realized that meant TurnKey was either doomed to stagnate or we figure out a way to re-engineer the development process in a way that truly democratized it so that everyone could join in on the same level.
So as many of you know (we've been waving our hands around about this for a while) a
couple of years ago we started working on this in earnest and we've been gradually
re-engineering all our legacy build infrastructure with the ultimate goal of creating the mother of all appliances - a self-contained appliance that can be used to build any appliance (including itself) from source code.
Like Liraz said in a
previous blog post
We've actually had this in our sights for quite a while now. We made some bad technical decisions early on with regards to how we setup our legacy build infrastructure and realized a bit too late that we would have to redo everything if we wanted to get the open source community truly on board with TurnKey's development.
With everything on our plate (e.g., developing over a hundred appliances and the TurnKey Hub) it sometimes feels like we're running in place. So it's taken longer than we would have liked to make this happen.
TKLDev and appliance sources released
Long story short we've re-engineered our entire build infrastructure, bottled it up into a self-contained appliance and battle tested it by building the entire
TurnKey 12.1 maintenance release
. We also
moved to GitHub
to streamline collaboration with the community.
TKLDev is ready for immediate
in all the standard build formats in both
32-bit / i386
and
64-bit / amd64
architectures. You can also deploy it straight in the cloud via the
TurnKey Hub
Although TKLDev is relatively easy to use, it'll probably save you some time if you read through the
documentation
first to get an overview of how things works, development best practices, etc.
TKLDev lets you build any TurnKey appliance from scratch, directly from source. You can find the
source code
for all appliances in the TurnKey library at
github.com/turnkeylinux-apps
. All our other repositories live at
github.com/turnkeylinux
. We put appliances in a separate GitHub account because with so many of them they need their own namespace.
The way forward - TKL 13.0 Wheezy, a community effort?
A while back we
announced
release candidates for TurnKey Core 13.0, but then back tracked a little with the 12.1 maintenance release, and then side tracked again with TKLDev and the move to GitHub.
Well, it was all for a very good cause but I think we're done playing hop-scotch. With all the pieces now in place, we're moving full force on the TKL 13.0 release. The first things we'll be doing is updating all the TurnKey specific components, Core and TKLDev for 13.0. With that, the infrastructure is set to start upgrading the appliances.
We're hoping with the release of TKLDev and full appliance source code that the upcoming release will be more of a community effort. Thanks to the new infrastructure there is no special sauce, no wizard behind the curtain driving TurnKey development. We're giving Linux geeks everywhere the (
super?) power to build any appliance, including Core, from scratch. What you do with it is limited only by your imagination but we're hoping at least a small part of that imagination will go towards helping us improving TurnKey for everyone's benefit.
Hopefully, we've made this stuff easy enough to use so nearly anyone with the inclination can jump in and make cool stuff happen. All the hard, really technical bits are automated so you don't need a lot of Linux experience to work with TKLDev. The same amount of experience you'd need to install and configure a regular Debian installation will take you a long way. The difference being that you can mass reproduce the results and collaborate with others on the development in ways you couldn't do otherwise.
If we've done our job well, we won't be able to anticipate all the different ways the community is going to put TKLDev to use. Some of the basic uses should be pretty obvious though. Let's say you want to improve an existing appliance. Maybe u
pgrade an appliance to use the latest upstream versions of your favorite open source application. Or maybe you've come up with a cool new feature that will make TurnKey Core (and by extension all other appliances) more useful? Fork, update, test, send a pull request, and your commit becomes an official part of TurnKey. It's that easy.
I can't wait to see how this evolves. Lets get hacking!
Comments
YES!!!!
Eric (tssgery)
- Thu, 2013/07/18 - 19:26
This is great news!

I have hobbled together a similar appliance that builds my patches, which I can now obsolete and throw away! Which is fantastic!

I will be giving this a try this weekend!
reply
Here comes the first pioneer
Liraz Siri
- Fri, 2013/07/19 - 13:54
Hi Eric! Looking forward to your feedback as soon as you give TKLDev a test ride.
reply
So far, so good!Deployed and
Eric (tssgery)
- Sat, 2013/07/20 - 03:24
So far, so good!
Deployed and setup easy enough; took me about 30 minutes to convert a tklpatch to the new fab system. I took my three most popular applainces (subsonic, nzbapp and newznabapp) and migrated them without difficulty. My github repos are updated so others can try them out if they'd like:
I did find a little issue with running "make clean" on an appliance but if you do what the error message says, your'e good to go. Details are at:
and I may find the time to submit a pull request for a fix. Gotta love opensource stuff.
reply
This was a bug in deck
Liraz Siri
- Tue, 2013/07/23 - 12:33
Hi Eric,
Awesome that you've already ported your patches to TKLDev. I'll ask Alon about forking your repos into turnkeylinux-apps so we can include them in the next release.
Anyhow, I traced the problem with make clean to a bug in deck which I committed a fix to earlier today.
Cheers!
[update]:
Alon updated the archive
with the bugfixed deck package.
reply
Added these to list of TKLDev community projects
Liraz Siri
- Thu, 2013/08/01 - 07:20
Figured it would probably be a really good idea to create a page on the wiki tracking
community TKLDev projects
like the three you listed above.
I also added a couple more TKLDev projects (Minecraft, Observium) I found on your GitHub profile that you didn't list above. You've been holding out on us!
reply
Cool!
Adrian Moya
- Thu, 2013/07/18 - 19:40
Great news again! I was about to ask via twitter as I already have a couple of projects to do with TKL! I hope to get some time to download, test and give some feedback.
For what I understood from the post, if you send this TKLDev to the space on it's own, it could reproduce herself and colonize a planet? :O
I'll check the docs and keep in touch!
reply
First the moon, then the outer planets
Liraz Siri
- Fri, 2013/07/19 - 13:51
Adrian, I like your vision. There are a few features missing for TKLDev to become a fully fledged autonomous, planet colonizing von Neumann probe, but eventually, why not! I promise to try and integrate it with first viable open source AI, then maybe it will improve itself recursively. That would be a big win.
reply
Yes there is! :)
Jeremy Davis
- Fri, 2013/07/19 - 03:24
The 64 bit version is the default (from the links on the site) but 32 bit version is available from the TKL SourceForge repo. If you have a look in the
files section
under your preferred medium (e.g.
ISO
) you'll find it (e.g.
turnkey-tkldev-12.1-squeeze-i386.iso
Happy building! :)
reply
Added a link on the blog post and tkldev app page
Liraz Siri
- Fri, 2013/07/19 - 14:17
Thanks for pointing out that it's not so obvious where the 32-bit image is. Other people might be wondering about that. I've added a couple of links to the blog post and tkldev app page so that they you don't have to read through to JedMeister's comment (helpful as always) to find it.
reply
Great News
Dan Robertson
- Thu, 2013/07/18 - 22:48
This is excellent!  I am excited to see TKL continue to grow and become even better.
reply
I'm excited that you're excited!
Liraz Siri
- Fri, 2013/07/19 - 14:20
Thanks for the encouragement Dan, and for showing up for all our important announcements. I know you've been following TurnKey's progress for a few years now...
reply
Spectacular achievement guys. Leaves Bitnami in the dust :-)
OnePressTech
- Fri, 2013/07/19 - 02:17
Well done guys. With this TKLDev release you have turbo-charged what was already a key backbone of the Cloud revolution...Turnkey Linux.
I'm out of the country for the next month but when I return I will be starting to build some multi-app appliances which I will be contributing back to the community.
Rock & roll!!
Cheers,
Tim (Managing Director - OnePressTech)
reply
Multi-app appliances are a very interesting area for development
Liraz Siri
- Fri, 2013/07/19 - 14:41
Thanks Tim, I'm looking forward to seeing what you come up with so keep us in the loop.
BTW, we've been thinking about multi-app appliances for about as long as we've been thinking about TKLDev and friends. Unfortunately we haven't had the time to tackle this one ourselves yet but we do have a few ideas. To sketch the outline, I envision a meta TurnKey appliance that supports running multiple TurnLey inside thin containers that share the same kernel. LXC would be a good candidate component for this. I gather it's improved a lot in recent Linux versions. Last I checked there were issues with the quality of isolation between containers but you don't necessarily need good isolation to get this to work, that would just give you a big security bonus.
reply
LXC would be cool!!
OnePressTech
- Fri, 2013/07/19 - 15:12
Thanks Liraz. Had a quick look at LXC and it looks pretty interesting.
FYI - my motivation for multi-app appliances is to minimise the cost for my SMB clients. There is a minimum cost for a VM since any VM is always doing something even if doing nothing (obviously). Add on a network monitor and an external heartbeat poll and the cost of a VM doing "nothing" goes up. Where an SMB I.T. infrastructure includes multiple apps that play well together (repository, network monitor hub, document management system), the overall service cost is reduced considerably by grouping apps in fewer VMs. LXC seems like it would help to simplify the lifecycle management of a multi-app VM (assuming that LXC can be run within a VMWare, AWS, etc VM of course).
Cheers,
Tim (Managing Director - OnePressTech)
reply
Yes, LXC will run inside a VM. It's an OS-level feature.
Liraz Siri
- Fri, 2013/07/19 - 15:52
Unlike other virtualization techniques, LXC is implemented inside the Linux kernel as an operating system level feature. Like chroot, but with more features. All containers share the same kernel so for security's sake you have to limit certain permissions inside the container even from root (e.g., the ability to insert/remove kernel modules, see processes outside the container, etc.) but you get much higher efficiency in return.
reply
LXC works fine with TKL
Hans Harder
- Fri, 2013/07/19 - 19:57
LXC works fine with TKL
The only problem I ran into, is that sometimes a shutdown inside a lxc container brings the host down....
But that was only if I had a redhat as host.
You can easily take the tkldev or a core and add some lxc and cgroup  package to it, and then you can start up the other TKL appliances inside that host, GREAT FOR TESTING...
I am running ( 2 years now) the TKL lamp container and TKL svn  container on the same host , no problems
startup time is a few seconds for a container and the container only runs the applications, no kernel.
The only overhead you have is the diskspace needed for a container.
And LXC runs just fine inside vmware or virtualbox
QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
reply
Great news Guys!
Jeremy Davis
- Fri, 2013/07/19 - 03:19
Congratulations. I know that this has been a big dream of yours (and mine too) so it's great to see it finally realised...
I'm still not sure when I'll have time to have a proper play with it but I'm super excited! :)
reply
Thanks! Hopefully this will be just the start
Liraz Siri
- Fri, 2013/07/19 - 14:32
Thanks Jeremy! Alon and I have tried pushing the envelope in terms of how far we could stretch out our resources, but we also ran many times head first into our limitations and I'm hoping with TKLDev in the hands of the community our limitations will no longer be TurnKey's limitations.
Fingers crossed, we'll see the open source development model finally unleashed in this neck of the woods. That would be exciting!
reply
Finally!
Carl Kiendl
- Fri, 2013/07/19 - 10:27
This looks promising...
reply
Great !!
Hans Harder
- Fri, 2013/07/19 - 11:10
Great !!
Since my vacation is just starting, I can't think of a better timing... will be a interesting learnfull vacation !
QUOTE:  ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
reply
No coincidence!
Liraz Siri
- Fri, 2013/07/19 - 14:44
We've been keeping close tabs on your activity in the forum and have identified a pattern of vacation time around July. TKLDev's release was carefully timed...
reply
Have a look at this thread
Jeremy Davis
- Wed, 2013/07/24 - 00:20
reply
Crashed with USB 3.0 only system
Timeout
- Thu, 2013/07/25 - 03:16
It run into initramfs when system has only USB 3.0. I used 'noacpi nomodeset' at grub, did I missing something?
J.
reply
+1 Docker integration
Adrian Moya
- Tue, 2013/11/05 - 22:14
+1 Docker integration
I made some experiments in the past with LXC and TKL, but never get to something useful. Now the idea of Docker for TKL sounds very good, considering that the open PAAS environment may be moving towards Docker also as a way to enable portability between platforms...
My 2 cents,
reply
Agreed.
Liraz Siri
- Mon, 2013/11/25 - 16:30
Hey Adrian! Glad to see you pitching in...
Docker is basically an advanced wrapper around LXC. I've played around with it and was sold. Unless we run into any major obstacles, TurnKey integration is in the (near-term) future.
reply
Docker integration will happen soon
Liraz Siri
- Mon, 2013/11/25 - 16:29
Hi Chux,
Thanks for the feedback and sorry for the late reply. We're aware of docker and are just as excited about it. I think it has the potential to solve two important problems:
As a distribution channel, e.g., get TurnKey into more places
Support lightweight multi-app configurations
It's on the shortlist of things we're looking at following the release.
reply
TurnKey Docker optimized builds available
Alon Swartz
- Sun, 2013/12/29 - 16:11
We've just announced TurnKey Docker optimized builds, see the
announcement
for full details.
reply
Going through the documentation... 12 pages at least..
L. Arnold
- Mon, 2013/11/25 - 04:48
I have been reading through the documentation and It took about 11 pages to get to the "answer" of my first question.  How do I run TKLDEV on a New Appliance Build?
I had been thinking that, like TKLPatch I could install TKLDEV into the system and do a build.  That appears not to be the case.
As I understand it the new Appliance needs to be built, then Forked back up to Git, then pulled from Git into TKLDEV for the ISO to be built.
Question though, if part of the appliance I am building is hosted on LaunchPad am I "out of luck" or can I somehow specify the bzr sources in the new System Build?
I am wanting to do an OpenERP appliance on top of the PostgreSQL appliance.  Perhaps there are some Git Sources for OpenERP  (a confusing number but none that I can find is CLEARLY the source).
All guidance is appreciated.
(small feedback, docs should perhaps have an outline structure... and as an aside not sure why I would want to make an ISO of Core, other than going through the steps..  Perhaps illustrating early on How to convert Core into Lamp, and make an ISO of that would be more telling of the process involved)  My thoughts anyway.  Thanks for all the hard work here!
reply
Room for improvement in the docs...
Liraz Siri
- Mon, 2013/11/25 - 16:37
Hi Arnold,
Thanks for weighing in. There's definitely room for improvement in the docs. We should look into that. The process for using TKLDev is as follows:
Deploy thel TKLDev app and follow the setup instructions. I propose building TurnKey core as a test case. If that works you know you're ready to start building your own apps.
Fork the TurnKey app sources for the app that's closest to your use case. If you want to develop OpenERP you should probably look into the LAPP or PostgreSQL apps.
I propose taking a look at the sources for a few apps to understand what is going on. You can install software from any source, including Launchpad. You just have to provide the installation instructions (e.g., like you would have to do in a TKLPatch) and test that they work.
Cheers
reply
Point m to a Surce w/ Git and BZR sources
L. Arnold
- Mon, 2013/11/25 - 19:11
It would be great if I could look at a source which is calling Git, Launchpad and Python sources (I expect Python is in them all).
If you knew one of the Git/Source/Appliances I could look at that would be great.
OpenERP, btw, should be forked from PostgreSQL Appliance as it has its own Web Server, so Apache is not needed (and is somewhat in the way).  PostGresSQL uses the "lightspeed" (or similar named) server for the Database interface.
Ideally Ports should be contollable in Webmin so that perhaps OpenERP took 80/443 and PostGre took a off highway port.
reply
A few thoughts...
Jeremy Davis
- Mon, 2013/11/25 - 22:37
The source code for all the TKLDev builds are on GitHub so there are plenty of examples (the whole TKL library of them!). Note that it is a separate repo to the TKL one (IIRC it is called TKL-apps or something similar).
There is no issue basing your appliance on the PostgreSQL appliance (rather than the LAPP appliance). Just use the PostgreSQL appliance as your base (similar to what you may have done with TKLPatch).
As for your source code on LaunchPad, is that code you have created yourself or the code for the OpenERP software? If it's code you created then you may be able to import that into GitHub. If it's the source for OpenERP itself then as Liraz said you should be able to install it, same as you would have done with TKLPatch.
reply
Pages
Add new comment
Apps
Specials
Web development
Framework
Stack
IT Infrastructure
Content management
Blogging
Ecommerce
Education
Wiki
Media
Business management
CRM
ERP
Invoicing
Messaging
Email
Forum
Chat
Issue tracking
Project management
Database
NoSQL
Developer tools
Help
Forums
Support
General
Development
Documentation
Security and News Announcements
Low-traffic newsletter: up to one email a month.
Previous issues
Categories
development
news
appliances
community
debian
release
hub
stable
iso
security
cloud
ec2
aws
proxmox
lxc
ubuntu
tkldev
tips
v16.x
drupal
More tags
Recent posts
Free up disk space
4th Aug, 2024
Python PEP 668 - working with "externally managed environment"
29th Jul, 2024
v18.0 Stable Release #6 - 10 Newly Updated ISOs, Hub Builds & Proxmox/LXC builds
17th Apr, 2024
v18.0 Stable Release #5 - 20 Updated ISOs & Hub Builds - Proxmox/LXC builds
12th Feb, 2024
v18.0 Stable Release #4 - 10 Updated ISOs & Hub Builds - Proxmox/LXC builds
5th Feb, 2024
1 of 63
next ›
Archive
August 2024
(1)
July 2024
(1)
April 2024
(1)
February 2024
(2)
November 2023
(1)
October 2023
(1)
September 2023
(1)
July 2023
(1)
April 2023
(1)
March 2023
(1)
December 2022
(1)
November 2022
(2)
Pages
Recent comments
V19?
TurnKey LXC templates are available via the Proxmox UI
Non-Aws platforms
Thanks for that!
the command for adding the
mtoolshub
Great post
randomness