Chapter 15. Publishing

Publishing means distributing the files of your project. This is a generic mechanism. You can use it to maintain a web site or to release a new version of your application.

The most straightforward way to publish a file is with the :publish command:

        :attr {publish = ftp://ftp.my.org/upload/%file%} myfile
        :publish myfile

This uses the "publish" attribute on each of the files. When the "publish" attribute is missing the "commit" attribute is used. If both are missing this is an error.

When a file didn't change since the last time it was published, it won't be published again. This works with signatures, like building a target. The remote file is the target in this case. But Aap won't read the remote file to compute the signature, it will remember the signature from when the file was last uploaded (otherwise checking for outdated files would be slow).

In the example %file% is used. This is replaced with the name of the file being published, including the directory. This means you can use the same attribute for several files:

        Files = myfile yourfile hisfile
        :attr {publish = ftp://ftp.my.org/upload/%file%} $Files
        :publish $Files

Including the directory is sometimes not what you want. To only use the last part of the path use %basename%.

        Files = one/myfile two/yourfile three/hisfile
        :attr {publish = ftp://ftp.my.org/common/%basename%} $Files
        :publish $Files

The three files will be uploaded to the "common" directory. The directory names "one", "two" and "three" will not be used on the ftp server.

To publish all files with a "publish" attribute start Aap like this:

      aap publish

If the "publish' target is defined explicitly it will be executed. Otherwise, all files with the "publish" attribute are given to the :publish command, just like using the :publishall command.

The "publish" attribute may consist of several items. Publishing will use all these items. This means a file can be distributed to several sites at once. This is unlike fetching, which stops as soon as the file could be obtained from one of the items.

When publishing fails for one of the sites, e.g., because the server is down, this is remembered. When you publish again, uploading is done only for that site. The destinations for which pusblishing worked will be skipped then.

Using Secure Copy

Uploading files requires write access to the server. There are several methods for this, but some have the disadvantage that your password is sent as normal text over the internet. Or someone in between can change the files that you send out. There is one that provides sufficient security: scp or secure copy.

To publish a file to a server through secure copy use a URL in this form:

        :publish file {publish = scp://myname@the.server.com/dir/%file%}

"myname" is your login name on the server "the.server.com". The "file" will be written there as "dir/file", relative to your login directory.

This requires the "scp" command, which is not a standard item. If it does not appear to exist then Aap will offer you to install it for you. If the automatic mechanism fails you will have to install it yourself. You might also want to do this if you have specific preferences for how the scp command is to be installed (versions of scp exist with different kinds of encryption).

To avoid having to type a password each time you need to use public keys. For MS-Windows you can find information here:

      
      http://the.earth.li/~sgtatham/putty/0.53b/htmldoc/Chapter8.html
      

This is for the PuTTY version of scp, which is what Aap installs for you if you let it. For Unix try man ssh-keygen. Use "SSH protocol version 2" if possible.

You can specify the command to be executed with the $SCP variable, see the reference manual. Example:

          SCP = scp -i c:/private/keyfile

Using Another Method

Using "rsync" has an advantage if you have files with only a few changes. The rsync program will only transfer the differences. This adds a bit of overhead to find out what changed, thus it's a bit slower when copying whole files.

Aap uses the same secure channel as with "scp" by default. This can be changed with the $RSYNC variable, see the reference manual.

If your server does not support "scp", you might want to use "rcp". However, this is insecure, information is transferred unencoded over the internet and making a connection is not secure. Only use this when "scp" is not possible.

Using "rcp" means changing the "scp://" part of the URL to "rcp://". The $RCP variable holds the name of the command, see the reference manual.

You can also use "ftp". Some web servers require this, even though ftp sends your password as plain text over the internet, thus it is insecure. Aap uses the Python ftp library, thus an external command is not needed and there is no variable to specify the command.

Changing A Url

When you switch to another server you need to change the "publish" attribute. The next time you invoke Aap for publishing it will automatically upload all the files to the new server.

If you didn't actually change to another server, but the URL used for the server changed, invoke Aap once with the "--contents" argument:

      aap publish --contents

This will cause only files that changed to be published. It avoids that all the files with a changed "publish" attribute are published again. It will still upload files that you changed since the last upload and newly added files. But be careful: files that you once uploaded, deleted from the list of files and now added will NOT be uploaded!

Distributing Generated Files

When publishing a new version of an application, you might want to include a number of generated files. This reduces the number of tools someone needs to use when installing the application. For example, the "configure" script that "autoconf" produces from "configure.in" can be included.

To avoid these generated results to be generated again when the user runs aap, explicitly specify a signature file to be used and distribute this signature file together with the generated files. For example, suppose you have a directory with these files you created:

        main.aap
        prog.c

Additionally there is the file "version.c" that is generated by the "main.aap" recipe. It contains the date of last modification. To avoid it being generated again, include the "mysign" file in the distribution. The files to be distributed are:

        mysign
        main.aap
        prog.c
        version.c

In the "main.aap" recipe the "signfile" attribute is used for the dependency that generates version.c:

        version.c {signfile = mysign} : prog.c
                :print char *timestamp = "$TIMESTR"; >! $target

The result is that "version.c" will only be generated when "prog.c" has changed. When the "signfile" attribute would not have been used, "version.c" would have been generated after unpacking the distributed files.

Copying All Files At Once

The "finally" target is always executed after the other targets have been successfully built. Here is an example that uses the "finally" target to copy all files that need to be uploaded with one command.

        Source = *.html
        all: remote/$*Source {virtual} {remember}

        CFile =
        :rule remote/% : %
                _recipe.CFile += $source

        finally:
                @if _recipe.CFile:
                        :copy $_recipe.CFile ftp://my.org/www

Warning: When the :copy command fails, aap doesn't know the targets were not made properly and won't do it again next time. Using the "publish" attribute works better.

Publishing images for HTML files

When publishing HTML files you most likely also want to publish the image files referenced by them. You can use a wildcard to select the images files, but this may include images that are not actually used. You can explicitly specify the images, but if you miss one your locally tested web site won't work properly on the server.

A good solution is to parse the HTML files and figure out what images are used in them. However, only image files that can be found locally should be included, not ones that are on other web sites. The get_html_images() function can be used for this, as shown in an example:

        Files = index.html
                contact.html
                forward.html
        Files += `get_html_images(Files)`
        :attr {publish = scp://user@ftp.foo.org/public_html/%file%} $Files

        all: publish

Note that this still isn't perfect. Only images in plain "IMG" tags will be found and only when they use a relative path name. Thus images from Javascript or PHP code will not be included.