NetBSD Mirror FAQ

FTP and ISO Mirror FAQ

BitTorrent Mirror FAQ

WWW Mirror FAQ

CVSweb Mirror FAQ

FTP and ISO Mirror FAQ

Overview for setting up a NetBSD FTP/ISO Mirror

The basic steps for setting up a NetBSD FTP or ISO Mirror are:

  1. Prepare disk space. See "How much disk space is needed for an FTP/ISO mirror" entry below for information on amount of space required.
  2. Retrieve a copy of the NetBSD FTP site contents
  3. Set things up to synchronize the contents of your mirror with ftp.NetBSD.org.
  4. Send in the information about your mirror to . See "Contact information format for FTP/ISO Mirrors" for details on that.

How much disk space is needed for an FTP/ISO mirror?

The amount of disk space needed fluctuates over time. At this time (July 2018) a complete mirror of the supported NetBSD releases needs approximately 250 GB (and an additional 520 GB for unsupported historical releases). The daily builds of NetBSD amount to 1.3 TB and the whole of pkgsrc (source and binary packages for several branches) to 1.3 TB.

Space requirements for ISO files are dependent on how many binary releases are supported at the time. The size may be severely decreased when old branches become obsolete and will grow when new versions of NetBSD are released. Each release takes from 20 to 30 GB, and currently (July 2018) the complete iso area needs around 225 GB space.

How much bandwidth does offering an FTP/ISO mirror require?

At the moment, we do not have any detailed numbers on bandwidth usage from our mirrors. The actual usage of any mirror depends too heavily on a variety of factors such as already available bandwidth, geographical location relative to other mirrors and amount of data mirrored for us to provide you with accurate numbers.

If you have been running a full mirror for a while and have collected bandwidth statistics, please let us know.

Retrieving/Syncing a copy of the NetBSD FTP site and ISO images area

There are several ways that you can retrieve and sync the files. Depending on your personal preference, your network connection, the packages installed on your systems etc. you may choose one over the other. Either way, it's probably a good idea to first run the relevant command by hand and then, when you have confirmed that everything is working as it should, create a script to be run by cron(8).

All ISO images of the NetBSD distributions are stored on the FTP site inside images/ subdirectory. Thus, mirroring of ISO images is very similar to mirroring the entire FTP site.

Below you can find some examples of how to keep your local copy in sync. Simply replace "<FTPROOT>" with the actual root of your ftp server.

  • Using net/rsync: 

    $ rsync -v -Wartz --delete rsync://rsync.NetBSD.org/NetBSD <FTPROOT>/pub/NetBSD/

    For ISO files only: 

    $ rsync -v -Wartz --delete rsync://rsync.NetBSD.org/NetBSD/images <FTPROOT>/pub/NetBSD/images

  • Using net/wget: 

    $ cd <FTPROOT>
$ wget -v -xnH -m ftp://ftp.NetBSD.org/pub/NetBSD/

    For ISO files only: 

    $ cd <FTPROOT>
$ wget -v -xnH -m ftp://ftp.NetBSD.org/pub/NetBSD/images/

  • Using net/mirror: 

    $ cat > ftp.NetBSD.org <<EOF
package=netbsd
comment=NetBSD
site=ftp.NetBSD.org
remote_dir=/pub/NetBSD
local_dir=<FTPROOT>/pub/NetBSD
do_deletes=true
algorithm=1
EOF
$ mirror ftp.NetBSD.org

    For ISO files only: 

    $ cat > iso.NetBSD.org <<EOF
package=netbsd
comment=NetBSD
site=iso.NetBSD.org
remote_dir=/pub/NetBSD/mages
local_dir=<FTPROOT>/pub/NetBSD/images
do_deletes=true
algorithm=1
EOF
$ mirror iso.NetBSD.org

Notes about a NetBSD FTP/ISO mirror

As a NetBSD FTP/ISO mirror, it's important that you synchronize your copy of the NetBSD FTP tree and ISO images area regularly. Otherwise, old and/or obsolete files may confuse users.

Also, it might happen that due to, e.g., a security issue, file removal (or another administrative task) may be requested from the NetBSD Project.

Where should an outage be announced

If your FTP/ISO mirror experiences any significant system outage, you should notify as soon as possible so that we can temporarily redirect your CNAME (if there is one) to a nearby alternative. Also, you may wish to send a note about the outage to the netbsd-users mailing list.

If the CNAME for your mirror has been redirected, be sure to let us know when the situation is rectified, so we can revert the change.

Contact information format for FTP/ISO mirrors

Contact information needs to be submitted to .

This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD FTP/ISO mirrors should be sent in the following format:

mirror          *machine name*
country         *ISO 3166 country code of host, see /usr/share/zoneinfo/iso3166.tab*
location        *more specific location (e.g, ``University of Foo'')*
ftp		*url to NetBSD ftp mirror area*
ftp-host	*fqdn of the host providing ftp mirror service*
ftp-releng	*url to NetBSD release engineering files, aka NetBSD-daily*
iso		*url to NetBSD iso images area*
iso-host	*fqdn of the host providing iso images mirror service*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo.com
country         jp
location        Department of Computer Science, Foo University
ftp             ftp://ftp.foo-univ.ac.jp/pub/NetBSD/
ftp-host        ftp.foo-univ.ac.jp
iso		ftp://iso.foo-univ.ac.jp/pub/NetBSD/images/
iso-host	iso.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

Please use ftp* and iso* fileds in case only if you providing appropriate mirror services. For example, omit ftp-releng field if there are no releng files on your server, or don't use any of ftp* fields if you running ISO mirror only.

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get (ftp|iso).country-code.NetBSD.org

We have temporarily stopped assigning names in the .NetBSD.org domain to mirrors as we are re-evaluating our mirror policies.

If you can no longer provide a FTP/ISO mirror

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.

BitTorrent Mirror FAQ

Bit-what?

BitTorrent is a tool for distributing files. It's extremely easy to use - downloads are started by clicking on hyperlinks. Whenever more than one person is downloading at once they send pieces of the file(s) to each other, thus relieving the central server's bandwidth burden. Even with many simultaneous downloads, the upload burden on the central server remains quite small, since each new downloader introduces new upload capacity.

Overview for setting up a NetBSD Bittorrent Mirror

The basic steps for setting up a NetBSD BitTorrent Mirror are:

  1. Install net/bittorrent
  2. Fetch the .torrent files from our FTP site. The files are in the /pub/NetBSD/images/<release> subdirectory, so if you already are mirroring this directory, you're all set. For more information see this page.

  3. Start BitTorrent: 

    cd /ftp/pub/NetBSD/images/<release>; (btlaunchmany.py .) >/dev/null 2>/dev/null &

    If you didn't download the ISO images already, they will be downloaded now. You can use the --max_upload_rate arg and --max_download_rate arg options to limit the transfer rates.

Contact information format for BitTorrent mirrors

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD BitTorrent mirrors should be sent in the following format:

mirror          *machine name*
country         *ISO 3166 country code of host, see /usr/share/zoneinfo/iso3166.tab*
location        *more specific location (e.g, ``University of Foo'')*
bittorrent	*BitTorrent URL / URL with additional Information*
ipv6		*yes or no, if your server supports IPv6*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          ftp.NetBSD.org
country         us
location        Silicon Valley, California
bittorrent	http://www.NetBSD.org/mirrors/torrents/
ipv6            yes
organisation    The NetBSD Foundation, Inc.
contactperson   Admins
contactemail    <email-here>@NetBSD.org
contactkeyid	AF226A4C
contactphone    555-123-1234
contactaddress 	Somwhere in Silicon Valley, California

WWW Mirror FAQ

Overview for setting up a NetBSD WWW Mirror

The following is an overview of the recommended procedure for setting up a NetBSD WWW mirror:

  1. For setting up a mirror with this procedure, you'll need cvs, ssh, and wget (see net/wget). Of course you'll also need a web server, such as apache.
  2. Retrieve a copy of the NetBSD WWW site. See "Retrieving a copy of the NetBSD WWW site" for instructions on how to do so with anoncvs.
  3. Set things up to synchronize the contents of your mirror with www.NetBSD.org. See "Syncing the NetBSD WWW site contents" for an example of how to do so using a cron job.
  4. Send in the information about your mirror to . See "Contact information format for WWW Mirrors" for details on that.

How much disk space is needed for a WWW mirror?

The amount of diskspace needed for a complete copy of the htdocs repository is currently (July 2018) around 310 MB. The total amount should not vary too much, but will obviously grow with time.

Retrieving a copy of the NetBSD WWW site

To retrieve a copy of www.NetBSD.org via anoncvs, you should do: 

# su - nbwww
# cd /path/to/htdocs
# env CVS_RSH=ssh CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot \
  cvs checkout -P htdocs

Note: "/path/to/htdocs" should be where you want the mirror of www.NetBSD.org to reside. The directory should be owned by an e.g. "nbwww" user.

Syncing the NetBSD WWW site contents

The best way to do this is to set up a cron job for your e.g. nbwww user to update the web site at a regular interval using anoncvs and wget (more on the wget usage below): 

# crontab -u nbwww -e
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin
MAILTO=whoever@gets.the.mirror.mail
CVS_RSH=ssh
CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot
0 1 * * * cd /path/to/htdocs && cvs update -dP && wget -B http://www.NetBSD.org/ -nv -xnH -N -Fi mirrors/fetch.html

Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail. Also, "/path/to/htdocs" should be changed to where your www.NetBSD.org mirror resides.

What is the purpose of htdocs/mirrors/fetch.html

The fetch.html file links to the handful of documents on www.NetBSD.org that are generated by www.NetBSD.org. These files get generated daily from content in the doc module of the NetBSD CVS repository.

In the example cron tab above, you'll see "wget -nv -xnH -NFi mirrors/fetch.html". This means that daily, through wget, these www.NetBSD.org created files will be updated in your copy of the web site contents.

Ensuring HTTP/1.0 clients get the right page

It is much preferred that NetBSD WWW mirror sites run a separate virtual web server for a www.NetBSD.org mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site.

If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host.

Contact information format for WWW mirrors

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD WWW mirrors should be sent in the following format:

mirror          *machine name*
country         *ISO 3166 country code of host, see /usr/share/zoneinfo/iso3166.tab*
location        *more specific location (e.g, ``University of Foo'')*
www             *WWW URL*
www-host        *external WWW name*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo-univ.ac.jp
country         jp
location        Department of Computer Science, Foo University
www             http://netbsd.foo-univ.ac.jp/
www-host        netbsd.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get www.country-code.NetBSD.org

We have temporarily stopped assigning names in the .NetBSD.org domain to mirrors as we are re-evaluating our mirror policies.

If you can no longer provide a WWW mirror

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.

CVSweb Mirror FAQ

Overview for setting up a NetBSD CVSweb Mirror

The following is an overview of the recommended procedure for setting up a NetBSD CVSweb mirror:

  1. For setting up a mirror with this procedure, you'll need cvs, cvsup, and the cvsweb CGI script. Both cvsup and the cvsweb CGI script are available from the NetBSD packages collection (see devel/cvsup-bin and www/cvsweb). Of course you'll also need a web server, such as www/apache.
  2. Retrieve a copy of the NetBSD CVS tree. See "Retrieving a copy of the NetBSD CVS tree" for instructions on how to do so with cvsup.
  3. Set things up to synchronize the contents of your mirror with the cvsup server. See "Syncing the NetBSD CVS tree contents" for an example of how to do so using a cron job.
  4. Send in the information about your mirror to . See "Contact information format for CVSweb Mirrors" for details on that.

How much disk space is needed for a CVSweb mirror?

The amount of disk space needed is rather large due to the fact that you are providing an interface to the entire CVS repository. Currently (2006-05), the total disk space required for all sources in CVS is around 4.3 GB.

Retrieving a copy of the NetBSD CVS tree

To retrieve a copy of the CVS tree via cvsup, you should do: 

# su - cvsweb
# mkdir -p /home/cvsweb/netbsd
# cd /home/cvsweb

Note: "/home/cvsweb/netbsd" should be where you want the mirror of the tree to reside. The directory should be owned by an e.g. "cvsweb" user. Create an "supfile" using your editor (For examples, see netbsd-supfile). Please take a look at the list of available public cvsup servers, you'll find the list here. 

# cvsup -g -L 2 netbsd-supfile

Syncing the NetBSD CVS tree contents

The best way to do this is to set up a cron job for your e.g. cvsweb user to update the tree at a regular interval using cvsup: 

# crontab -u cvsweb -e
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin
MAILTO=whoever@gets.the.mirror.mail
0 1 * * * cvsup -g /home/cvsweb/netbsd-supfile

Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail.

How to setup the cvsweb CGI script

Take a look at the cvsweb.conf and be sure to edit the following options:

@CVSrepositories = (
        'netbsd'  => ['NetBSD',           '/home/cvsweb/netbsd'],
);

$mancgi = "http://man.NetBSD.org/%s.%s";

Place cvsweb.cgi in your web servers' cgi-bin directory and be sure to edit it to point to your cvsweb.conf.

Ensuring HTTP/1.0 clients get the right page

It is much preferred that NetBSD CVSweb mirror sites run a separate virtual web server for a CVSweb mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site.

If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host.

Contact information format for CVSweb mirrors

Contact information needs to be submitted to . This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD CVSweb mirrors should be sent in the following format:

mirror          *machine name*
country         *ISO 3166 country code of host, see /usr/share/zoneinfo/iso3166.tab*
location        *more specific location (e.g, ``University of Foo'')*
cvsweb          *CVSweb URL*
cvsweb-host     *external CVSweb name*
organisation    *organisation name*
contactperson   *contact person's name*
contactemail    *contact person's electronic mail address*
contactkeyid	*contact person's pgp key id*
contactphone    *contact person's telephone number*
contactaddress  *contact person's postal address*

An example submission might look like this.

mirror          netbsd.foo-univ.ac.jp
country         jp
location        Department of Computer Science, Foo University
cvsweb          http://netbsd.foo-univ.ac.jp/cgi-bin/cvsweb.cgi/
cvsweb-host     cvsweb.netbsd.foo-univ.ac.jp
organisation    Foo University
contactperson   Foo Bar
contactemail    Bar@foo-univ.ac.jp
contactkeyid	GA23FG12
contactphone    +81 3 xxxx yyyy
contactaddress  9876 Foo City, Tokyo 1234567 Japan

If the contact information for your mirror ever needs to be changed, please feel free to let us know at .

How to get cvsweb.country-code.NetBSD.org

We have temporarily stopped assigning names in the .NetBSD.org domain to mirrors as we are re-evaluating our mirror policies.

If you can no longer provide a CVSweb mirror

Please let us know at so that we can remove your site from our list and redirect the DNS name to an alternative site.