Ant sucks for FTP deployment - What alternatives do we have?
Posted on 10/10/06 by Felix Geisendörfer
The authors of this post have marked it as deprecated. This means the information displayed is most likely outdated, inaccurate, boring or a combination of all three.
Policy: We never delete deprecated posts, but they are not listed in our categories or show up in the search anymore.
Comments: You can continue to leave comments on this post, but please consult Google or our search first if you want to get an answer ; ).
Update: Bruno D. has spent quite some time researching alternatives to ANT deployment and posted his results in the comments. So if you don't like my SVN deployment task for Cake 1.2, you might find his suggestions useful.
I wish I could say deploying my applications is just a matter of typing "ant deploy" into the command line these days, but the truth is - it isn't. Don't get me wrong, I like Ant a lot and I use it for some of my SVN tasks like exporting the current trunk and zipping it up for emailing it to a client and things like this, but trying to make it deploy my CakePHP app's on shared hosts via FTP turns out to be a nightmare. Here is why:
- Ant's FTP task is not capable of copying empty folders, making it really messy to copy an applications entire structure (especially tmp folders)
- Ant's FTP task cannot chmod folders
- Ant's FTP task cannot recursivly delete files if .htaccess files are hidden and can only be seen via LIST -a
Now you could work around those issues using various manual sub tasks. In fact I even got myself the source of the java commons-net package which is needed for FTP, and hacked it so ANT would automatically use "LIST -a" to work around problem #3. However, looking back, I've probably already spent more then 10 hours trying to create a smooth deploying mechanism with ANT with very little results so far - getting to the point where my automation efforts defeat their purpose.
So right now I'm half way done with an AutoHotkey script that can be called by Ant that will use my FTP client (FileZilla) to perform the required tasks (in the background). But I'm a little unhappy about the fact that this will remain a Win only solution. For that reason I would like to replace the FileZilla/Autohotkey combination with a command line utility available for Windows, Linux and Mac OS X, but to my dissapointment I could not find one fitting my needs. Such a utility would need to be able to recursivly upload, delete and chmod files + folders. If any of you knows about such a tool, please let me know!
Ok, what are the other alternatives? I know some people like Phing, but it doesn't come with a FTP task and uses php5 (I'm on php4). Anyway, "Evan Wired" mentioned that he uses (scroll down to the comments) a custom FTP task with phing, so Evan if you are still reading this blog and would like to share your task - I would love to give it a try at this point.
Another idea floating in my head is to write a CakePHP specific deployment tool that could be coupled with SVN. One that would check what files have been modified in the trunk since the last update and only overwrite those, and maybe take care of DB migration as well. This could maybe be integrated into bake.php. Cake 1.2 has a new folder inside /cake/script called tasks, so it could be a good place to hook in. But this would not be one of those little 5 minute hacks, so I'm not quite convinced it would be worth it.
So, this and the last post were more in the "ask my readers"-"whine about life" category then I would have liked them to be, but I'll make it up with some useful CakePHP posts this week, I promise. However, no cake bet will be involved this time ; ).
--Felix Geisendörfer aka the_undefined
PS: I know Daniel (cake baker) has a strong Java background, maybe we should make him enhance the ANT ftp task ; ).
You can skip to the end and add a comment.
capistrano is also a pretty popular one, they say it's much simpler then phing
maybe you'll find something usefull in here:
Ja. I second capistrano. Read this: http://www.simplisticcomplexity.com/2006/8/16/automated-php-deployment-with-capistrano
just for uploading in command line / batch via FTP, you can try ncftp (client version), it works on several OS
I use it a lot to cron backup and uploading releases for customers
Capistrano is not option for me because I deploy on shared hosts with no SSH acess. So my only option is some tool supporting FTP.
Sitecopy looks interesting even so it requires cygwin. With Ncftp I'm trying to figure out if it supports chmod or not. I guess the next ~10 hours of getting this working are already ahead of me, or like Zach Cox already said: *sigh* ...
If I ever find a good way to get all of this done, I'll write a big tutorial on it ...
You may want to simply get better hosting. Having shell access to your account is useful for more than just deployment. (And transitioning between severs is actually a non-event in your case as you're setting up an automated deployment system anyway.)
But, I am no doubt biased by the fact that I am yet another user who has happily bent Capistrano to his CakePHP will. :) ( http://scotfl.ca/2006/07/25/capistrano-and-cakephp-in-perfect-harmony/ )
scott: I'm not going to switch my host at any point soon. Having SSH access means you are responsible for your server and if you mess up, your host will charge you to fix it. The hosting company I use is really awesome because their support is unbeaten. If I ask them to update MySql, PHP or whatever, I can expect it to be done the same day, without a charge. Heck, they even helped me to figure out why one of my scripts wasn't working properly ... Unfortunatly they do not offer SSH access, not even with their dedicated servers. But again, I don't need it for anything, only deployment would be easier this way. Everything else is perfect, this site shares a server with ~80 other customers and easily surrived to digg.com traffic spikes so far ; ).
Woh! I can't think of having a dedicated server and not having ssh access :-O
That aside, I was going to suggest you to deploy using subversion but if you don't have ssh... In any case I was wondering about the same issue, because to be honest, deploying applications is a pain in the a**!
I had a brief look at capistrano but it seemed too much non-cake, I think I'll have a look at that again (and all the other suggestions that are in this page).
By the way, I never thought of using ant for deploying php. Long time since I touched a java app, I must say ... :)
ftp support should be in the upcoming 2.3.0 phing release
so if you're patient enough... ;-)
It is rather unlikely that I will work on the FTP task, because I don't use FTP for deployment purposes ;-) At the moment I use ssh and Subversion to deploy.
I looked at Capistrano, but as I read the docs I hadn't the feeling that Capistrano is what I am looking for...
I spent a few months trying to put together a workable FTP deployment system. All I got out of it was a headache and a deep, abiding love for SSH/SFTP. :)
Hopefully the situation has changed over the past year. Good luck!
In light of this, a Python script to do FTP uploading would be a worth while project. It'd satisfy the command line and platform independent requirements you mentioned above, as well.
Hi robinhoode, I'm working on a PHP script for deployment that integrates with CakePHP nicely right now. I'll post an alpha release on here soon.
Well I haven't found a platform independent solution, but one that definitely has saved me lots of headaches. www.scriptftp.com. You can write scripts so easily with it. You can even do one line uploads with recursive. And you can specify exclusions to your upload \ download. It supports CHMOD for files and folders. It will upload empty directories to my knowledge.
You can also setup your scripts to run on a schedule. You can also run your script via commandline and pass parameters. Like I have an installation script that I run that takes in parameters and grabs necessarily files to upload to their servers.
I also have a .bat file setup running the ftp scripts via commandline to update all 40 of my clients to the latest and greatest version anytime I run it.
I would love to have a Python or other OS independent solution, but right now a solution that works is so much better to me than being OS independent. I don't like to do sync because some servers write the date\time you upload and there is cases where you could update the file and bypass this check.
If anyone finds a OS independent scriptable FTP upload with chmod solution I would love it. For now ScriptFTP.com is the solution for me!
Troy: The DeploymentTask I'm working on right now is almost done. It allows you to use SVN to install / update / uninstall / activate or deactivate your project to / from any SVN revision of your repository. It's very flexible and allows you to change it's behavior entirely to fit your needs. I'll post about it soon.
I've started auto-deploying my applications using Capistrano. I've got a writeup here with some really terrible recipe code.
OMG I can relate so much to this problem!! I have been fighting for about two months now with the stupid Ant FTP task.
I've tried just about anything, even using directly the underlying FTPClient class from Apache to try and figure out what the hell was the problem. Images get corrupted (bad encoding), big files time out when transferring, etc...
I think I have found the source of this problem, something to do with encoding...Ant's manual mentions:
" Because there are at lest 4 different EBCDIC encodings, we have opted not to provide direct support for EBCDIC. To transfer EBCDIC and other unsupported file types you must create your own filter InputStreams and OutputStreams and wrap them around the streams returned or required by the FTPClient methods. FTPClient uses the NetASCII filter streams to provide transparent handling of ASCII files. We will consider incorporating EBCDIC support if there is enough demand. "
It's not clear to me what this exactly means, but the description for the "NetASCII" filter isn't reassuring:
" This class wraps an output stream, replacing all singly occurring (linefeed) characters with (carriage return followed by linefeed), which is the NETASCII standard for representing a newline. You would use this class to implement ASCII file transfers requiring conversion to NETASCII."
Hence it looks like it supports only text files... But why the hell then do they provide a binary option?? Anyways, I think at this point I'd have to post this question on the ANT mailing list or something...
Another weird thing is running "ant -diagnostics" on your command-line you'll see a line saying:
" image : Missing dependency com.sun.media.jai.codec.FileSeekableStream "
I did not find any help in the ANT docs on this error, but I've downloaded this jar file, put it in Ant's classpath but it did not fix anything... And no word whatsoever about image files in the documentations either...
So at this point I suspect that transferring image and binary files is not much supported, probably implemented but not thoroughly tested... Anyways, NOT something I want to get into, but I've already lost more than a day on trying to fix this or finding workarounds, clearly an area of improvement....
But like you say, this is frustrating to say the least, I would've liked to stick with a clean an elegant solution. I would've thought FTP is a basic part of deployments (especially in web applications), I can hardly understand why we're still fighting with this kind of stupid issues in this day and age... Sometimes the java folks just don't get it!
Good luck, I'll post back here if I find an elegant solution... Meanwhile I'll try that "ScriptFTP" thing.
It's a bummer, I was really starting to like Ant. Oh well, my little ant bubble is burst.
Hey here's a potential semi-clean fix:
I thought I'd flip the problem around, how can I make FTP transfers transparent to Ant, such as some kind of explorer extension or program? I've found two potential solutions, one called WebDrive, the other called Hummingbird FTP 6.2. Both claim to be able to connect via FTP to sites and make FTP access completely transparent in windows explorer such that you can copy files and perform any operation supported by explorer (copying / delete / rename /create dirs, etc) just like any normal local filesystem. Granted it's not a cross platform solution, but I've given up on that idea.... Aside from that, I'm not sure how well suited it is for multiple servers like you mention you have. I also wonder if it's possible to trigger it from command-line; I wouldn't want to keep a permanent connection opened to my sites ;-)
I'll give it a try and post back here later on...
Hey this also looks pretty promising, FSync :
plus it's command-line based...
Hey WebDrive seems like a pretty impressive app... Looks like command line is well supported, here's an excerpt:
WebDrive can be used in a batch file, to open a connection, copy files back and forth between the client and server and then disconnect. The following batch file is an example of how to do this. This assumes that WebDrive was installed into the default directory of "c:\program files\webdrive"
ECHO “Running FTP batch file!” start /wait /D“c:\program files\webdrive” webdrive.exe /s:“Microsoft” copy x:\dirmap.txt c:\ start /wait /D“c:\program files\webdrive” webdrive.exe X: /dType topic text here.
The file copying part could all be handled by ant instead... Ok enough of my spam here, I'll keep some for later ;-)
Ok, I've tested WebDrive a bit this afternoon, it works flawlessly! It creates a virtual drive in windows explorer with a given drive letter (say W:\ for example) and then all you have to do is set up Ant to copy your files there, or even better, use the Ant "sync" task which is already preset to synchronize two folders instead of just copying...
I have been able to transfer all sorts of files seamlessly since WebDrive manages all the FTP stuff in the background while Windows sees it as a plain normal filesystem...
I'm not sure if this would be a solution suited for people with many websites to update, allthough you can certainly connect on the fly using the command line example I gave above previously... Still I'd say it's mostly geared for 'permanent' connections. The one thing I really like is I can work on my remote files directly (for example modify configs, etc) instead of going through the usual hassle of FTP and synchronization. On the other hand this is usually not something you want to do on a production website, unless it's a minor change.
Anyways, I think it is a very elegant "patch" to the FTP hell from Ant. If I feel utterly motivated I might post my rants on Ant's mailing list though I doubt I'll get constructive answers (most likely something like "well if you don't like it just modify the code" haha.
I tried FSync also, pretty simple and straightforward, though I have two issues so far which prevent me from using it, though simple things that its developer should be able to fix quickly (hopefully). #1 is that it doesn't support the @ character in the username field (my host is setup like that and I can't change that) and #2 is that it doesn't allow using another port than the default, 21, which prevents me from using it for another special case (where I have a FTP server setup on something else than 21 to trick a dumb ISP who doesn't want me to use FTP).
Anyways, will post my final feedback on FSync later, but check out WebDrive if you get a chance, you'll never want to use plain FTP again!
Well good news, I've contacted the developer from FSync, he already fixed these issues, I got an updated copy and now it all works (he fixed the two issues I listed just above) :-)
So for you guys looking for a command-line, cheap (only 8$) FTP sync solution that can be used in batches FSync does the job perfectly... I can definitely imagine it being used for multiple sites, each with it's own connection parameters... It is very simple and straightforward to use.
Here's the link by the way if I didn't already post it:
Anyways that's it for me, Ant's FTP sucks, but at least I've found two good patches. I hope this can also help others with similar problems...
[...] One thing that always annoys me is developing code on my own machine, and transferring it onto a development/live server to see it in action. Previously I was doing all of my development in Dreamweaver, which has the ability to automatically upload files as you save each one. I got fed up with Dreamweaver, and switched to EasyEclipse (the main Eclipse bundled with useful plugins), and thought I would forever be stuck with Alt+Tab-ing to an FTP client, copying, pasting, waiting, cursing. Then I was looking through a CakePHP Bakery post, and found that Apache Ant can be used to FTP files. After seeing Francis’ script and doing some more stumbling, I came across this perfect build.xml file. [...]
[...] Apache Ant Homepage Phing Homepage Building Web Applications With Apache Ant - Julien Lecomte Trimming comments in HTML documents using Apache Ant - Julien Lecomte Improve Your Build Process with Ant - ONLamp Ant sucks for FTP deployment - What alternatives do we have? - Felix Geisendörfer Similar Posts Resources for Web Development [...]
This post is too old. We do not allow comments here anymore in order to fight spam. If you have real feedback or questions for the post, please contact us.
As long as the server you're deploying to supports it, I've found rsync the best way to go. It's super fast, only transmits file diffs, and handles hidden files & empty directories. If you can't use rsync then try sitecopy. It's a bit slower than rsync & has problems with ssh/scp/sftp but still is much faster than ftp (only transmits modified files).
My ant deploy scripts tend to have 3 components:
1) delete temp files
2) deploy files to server (rsync/sitecopy/scp/sftp/ftp, ignore .svn folders, etc)
3) deploy custom server-specific files (database.php, core.php, Google maps API keys, etc)
That usually handles the site files well, but the database is another issue. Ant can dump a local MySQL db to a file, but most servers don't allow remote MySQL access so I've had to manually import the dump on the server. Supposedly there's database migrations in CakePHP like Rails but I haven't tried it and that might not help with deploying a local db to a server anyway.
I've also heard of people using Capistrano with CakePHP but the setup instructions always seem hoaky.
My ultimate goal is to have a single Ant target that will properly deploy all files & the db to the production server. You'd think this would already exist... (sigh)