#

Tap the Usenet

The Usenet offers lots of information - I wouldn't like it to miss some of them.

A powerful package is Rich $alz's "InterNetNews", say INN. It isn't easy to master the INN. Who made the `sendmail` may be happy: it becomes more difficult...!

INN would be able to fetch and send news by itself, but we will get some cooperation from a more intelligent package: `newsflash` and `postit`.

Again I'll describe it depending on the S.u.S.E. 4.2 Linux distribution. Here INN is easily installable in the `/usr/lib/news` directory (with the "YaST" tool). By the way you can find here a (sorry, not very useful) overview: `/usr/lib/news/README.linux`, and INN is pre-installed (if marked in YaST), but all the Documentation is missing. You can get the documentation, if you send an email to majordomo@warren.mentorg.com

      Subject: empty
      sendme faq-inn-1
      sendme faq-inn-2
      sendme faq-inn-3

Then you will receive a long FAQ describing how INN works and how to fix many problems. You also should read the tutorial (in German language) news-0.3.ps by Markus Dickebohm. It describes in a very good way how to install a `INN/newsflash/postit` system. Have a look at this server: Maybe there are newer tutorials available. And you could also load down the packages `newsflash` and `postit` (see below: "Give and Take").

## InterNetNews

The first thing we will install is INN. A properly installed `sendmail` is required.

### Preworking

The INN daemon `innd` does not like any other program listening at the news port, and there may be a line in `/etc/inetd.conf` causing a fatal error:

      nntp stream tcp  nowait root  /usr/sbin/tcpd /usr/sbin/in.nntpd

You must add a "#" in the first column to make it a comment, otherwise `innd` will not start (I needed several days to notice that). INN does not run together with C News or any other news daemon, too. It isn't necessary, by the way.

There is one thing you also should do. If INN does not like something it writes an email to "`usenet@localhost`". You must tell your system that "usenet" is a user of your system - some other mail addresses too. You do it by editing `/etc/aliases`. Add the following lines (perhaps you have to create a new file):

    /etc/aliases
      news: root
      newsadm: news
      newsadmin: news
      usenet: news

Type in to tell sendmail the new aliases:

      root:# newaliases
      root:# _

Now all emails concerning your Usenet system come to `root`.

But before we start configuring INN we have to edit the file `/etc/hosts` and to insert one entry for the news server. Take the address from the database or call the provider's hotline. Add:

    /etc/hosts (cutting)
      <IP address>  <name_of_mail_server>    getmail

### INN configuration files

There are some configuration files required.

    /var/lib/news/inn.conf  (Slackware: /usr/lib/news/inn.conf)
      # Basic installation of INN
      server:	your.computer
      organization:	as you like
      fromhost:     Email_adresse@Host.Computer # for posting header

Thomas Schoepf <schoepf@informatik.tu-muenchen.de> wrote me: if there appears a error message like `ME: cant GetConfigValue pathhost...` while starting INN, you should add the line `pathhost: <Host.Fuer.News.Path>`.

    /var/lib/news/hosts.nntp (Slackware: /usr/lib/news/hosts.nntp)
      # Which site feeds our newsreader?
      localhost:
      Rechner.name:

    /var/lib/news/nnrp.access (Slackware: /usr/lib/news/nnrp.access)
      # Which computers may tap us?
      localhost:Read Post:::*
      (none):Read Post:::*
      your.computer:Read Post:::*

    /var/lib/news/newsfeeds (Slackware: /usr/lib/news/newsfeeds)
      # Where do the news go?
      # We don't feed our news to anywhere.
      ME\
              :*::
      # for NOV database
      @overview:*:Tc,WO:/usr/lib/news/bin/overchan
      # Feed the LRZ with all we have:
      get/getlist\
              :*:Ap,Tf,Wnm:get

Please refer to the INN documentation for more information. `providername` is the name the provider writes into the news header in the "Path:" line. You must exacltly write it the way your provider does. If you don't the article negotiation will take much more time.

There are some files missing, which can be fetched from the news server. Therefore you could use the script `getlist`, which should be part of the INN package. If not, load it via ftp:

    unix.hensa.ac.uk/mirrors/yggdrasil.fall95.disk1/usr/lib/news/bin/getlist.

Store it in `/usr/lib/news/bin` and make it executable: "`chmod news getlist`".

So you can load two files from `lrznews` using `getlist`:

      root:# /usr/lib/news/bin/getlist -h getnews active >/var/lib/news/active.raw
      root:# /usr/lib/news/bin/getlist -h getnews newsgroups >/var/lib/news/newsgroups
      root:# _

If you have the Slackware distribution change `/var/lib/news` against `/usr/lib/news`.

> You even could manage this manually (without "getlist") using telnet:

>       root: # telnet getnews 119 >/var/lib/news/active.raw
>     now wait 10 sec. - *no* prompt appears!
>       list active
>     wait again a few seconds
>       quit
>     wait until your own prompt appears (takes some time):
>       root: # telnet getnews 119 >/var/lib/news/newsgroups
>     now wait 10 sec. - *no* prompt appears!
>       list newsgroups
>     wait again a few seconds
>       quit
>     wait until your own prompt appears (takes some time):
>       root: # _

The fresh file `active` must be edited now. In it you find all the subsribable Usenet groups and the actual posting numbers. Warning: this file is sensitive. It does not like any changes (words per line, number of white spaces or number of digits)! You may not change anything except the things we discuss here!!!

You should do now two things. First, set all posting numbers zo zero. Do it this way:

      root:# cd /var/lib/news           # Slackware: /usr/lib/news
      root:# sed 's/ [0-9]* [0-9]* / 0000000000 0000000001 /' active.raw > active
      root:# _

Second, look if there are entries for the following newsgroups. If they are missing add them now.

      to 0000000000 0000000001 y
      junk 0000000000 0000000001 y
      control 0000000000 0000000001 y

### Polish

I found some errors the hard way. INN needs total access to the whole directory `/var/run`:

      root:# chown root.root /var/run   # (not root.uucp!)
      root:# _

(perhaps you need to play something with the access bits)

For testing look into `/var/spool/news` whether the directories `in.coming, news.archive, out.going` and `over.view` are present - if not, create them using `mkdir`!

Finally you must give all files in `/var/lib/news` (Slackware: `/usr/lib/news` and in `/var/spool/news` the user `news`:

      root:# chown -R news.news /usr/lib/news/*    # for Slackware only this one
      root:# chown -R news.news /var/lib/news/*    # if present
      root:# chown -R news.news /var/spool/news/*  # if present
      root:# _

For basic installation call now some assisting programs: `makehistory` and `news.daily`. Do it as user `news` (important!):

      root# cd /
      root# su news -c "/usr/lib/news/bin/makehistory -o"
      root# su news -c "/usr/lib/news/bin/news.daily"
      root# _

### The Big Question: Does it Work?

A test start:

      root:# /usr/lib/news/etc/rc.news
      Starting innd.
      root:# telnet localhost 119
      Trying 127.0.0.1...
      Connected to localhost.
      Escape character is '^]'.
      200 my.computer InterNetNews NNRP server INN version date ready (posting ok).
      no prompt appears!

If this message appears you may breath deeply and say loudly "uff!!!". You can escape here typing in `quit`.

If the message does not appear, sorry: you need to find the bug, and this may take some time. Let the INN-FAQ help you. Perhaps INN could send you an email to "`usenet`" - respectively "`root`", in which you may read what happened. If there is no mail perhaps you can find some hints in the `syslog`, in `/var/log/messages` or in the log files in `/var/lib/news/log`.

The INN installation made some gray hairs onto my head and took more than one month time. I tried to tell you all the mistake possibilities, but I feel I could not cover all of them. The INN is not very cooperative while debugging... But I think I could help you in most cases

### Start-up of the System

The INN is started while system start in rc.news. You must include it in the start-up sequence.

Again we have the possibls cases `sinpleinit` or `sysvinit`.

#### simpleinit (e.g. Slackware)

In `/etc/rc.d/rc.M` we define a new file:

    /etc/rc.d/rc.M; Sektion "# Initialize the NET subsystem."
      . /etc/rc.d/rc.inet1
      . /etc/rc.d/rc.isdn
      . /etc/rc.d/rc.inet2
      . /usr/lib/news/etc/rc.news   # <-- add this line!

You also must find the file which is called while system reboot (often this is `/etc/rc.d/rc.6`). Here you should add before the shutdown commands:

    /etc/rc.d/rc.6
      echo -n "Shutting down NNTP port:"
      /usr/lib/news/bin/ctlinnd throttle "shutdown"
      /usr/lib/news/bin/ctlinnd shutdown "shutdown"
      [more shutdown commands]

If you don't some (harmless) error messages appear while shutdown.

#### sysvinit (z.B. S.u.S.E, Caldera, Red Hat...)

If you have sysvinit (like S.u.S.E. 4.2) it is a bit more complicated. First, create a start/stop file:

    /sbin/init.d/news # (Red Hat: /etc/rc.d/news)
      #! /bin/sh
      # Author: Bernhard Hailer <dl4mhk@lrz.uni-muenchen.de>, 24-Jun-96
      # /sbin/init.d/news
      case "$1" in
        start)
          echo "Initializing NNTP port. (INN)"
          /usr/lib/news/etc/rc.news       #  rc.news wird beim INN mitgeliefert
          ;;
        stop)
          echo -n "Shutting down NNTP port:"
          /usr/lib/news/bin/ctlinnd throttle "shutdown"
          /usr/lib/news/bin/ctlinnd shutdown "shutdown"
          ;;
        *)
          echo "Usage: $0 {start|stop}"
          exit 1
      esac
      exit 0

You must set some soft links, because the system needs to know which files have to be executed while starting or stopping:

      root:# cd /sbin/init.d/rc2.d  #  There are other path definitions
      root:# ln -s ../news K20news  #  in other Linux distributions!
      root:# ln -s ../news S20news
      root:# _

Now while booting the INN start-up should appear on the screen - together with the sendmail daemon. Also while shutdown some messages should appear.

### Daily Work

The file `news.daily` we used above should be called daily - if not, INN will mail some reproaches to the user `usenet`. And therefore we create a small crontab entry for the user `news`, who should be the only user allowed to execute this script:

    /var/cron/tabs/news # (perhaps /var/spool/cron/crontabs/news)
      PATH=/usr/lib/news/bin:/usr/lib/news/etc:/usr/lib/news:/usr/bin:/usr/sbin
      10 07 * * * /usr/lib/news/bin/news.daily

We tell this now `crond`:

      root:# chown news.news /var/cron/tabs/news
      root:# crontab -u news /var/cron/tabs/news
      root:# _

Now the script should be called every day at 07:10 a.m.

`news.daily` does the cleaning and garbage collecting. Old news postings will be erased, and some old log files and so on will be deleted, too. There are two control files which may be interesting:

    /var/lib/news/expire.ctl  # Slackware: /usr/lib/news/expire.ctl
      # If posting is erased we will remember its ID 14 days, because
      # we don't want to receive it again:
      /remember/:14
      # (A)ll postings will be kept min. 3, regularly 10 and max. 30 days
      # (min. and max., if an expire date is set in the posting)
      *:A:3:10:30
      # some special newsgroups may remain much shorter:
      junk:A:1:1:2
      control:A:1:1:2

    /var/lib/news/control.ctl  # Slackware: /usr/lib/news/control.ctl
      # preset: if INN wants to tell something it has to mail it to admin.
      all:*:*:mail

## Give and Take

To operate posting and polling some software packages exist, which are able to cooperate with INN, like `newsflash+postit` by Chris Blum or `suck+rpost` by Tim Smith. INN also has some scripts on board which could do it. But the NNTP handshake is very complicated, and the telephone connection should be used very effectively, it is a good choice to take a more intelligent package.

The program `suck` is more known, but I choosed the (very compact) programs by Chris Blum, because they are highly configurable. They use the telephone line very efficiently, because they open more than one connection to the news server. So, while the news server may create a news packet, there may take place some discussions about the next packet at the same time.

Load these programs down using ftp at

      pople.pc.uni-koeln.de/pub/newsflash-0.99.tgz
      pople.pc.uni-koeln.de/pub/postit-jun96.tar.gz

(please look if there are newer versions!) Store it in the directory `/usr/src` and extract them:

      root:# tar zxfv newsflash-0.99.tgz
      root:# tar zxfv postit-jun96.tar.gz
      root:# _

#### Installation of `newsflash`

Newsflash fetches some choosed newsgroups (the ones not existing yet at home).

Before compilation you should edit `/usr/src/newsflash/config.h` and set the path variables correctly for your system (`/var/lib/news` or `/usr/lib/news/newsflash` for Slackware). Then:

      root:# cd /usr/src/newsflash
      root:# make dep
      ...
      root:# make
      ...
      root:# cp newsflash /usr/lib/news/bin
      root:# cp printdate /usr/lib/news/bin
      root:# _

`newsflash` needs to know which newsgroups it shall retrieve, so you must tell this in a file: it contains the files listing the desired groups:

    /var/lib/news/serverlist # or path defined in config.h!
      get1 getnews 119 /var/lib/news/packet1 timestamp
      get2 getnews 119 /var/lib/news/packet2 timestamp
      get3 getnews 119 /var/lib/news/packet3 timestamp
      get4 getnews 119 /var/lib/news/packet4 timestamp

Here four packets are executed, all aim at the news server of the provider. The first word is an ID string, the second is the server defined in /etc/hosts, 119 is the standard news port. The packet description follows, and the last number is a timestamp. You can calculate an active timestamp using the program `printdate` (included in the newsflash package); the produced 9 digit number tells us how many seconds are gone since January 1st at midnight. To retrieve the new news since the last two days you should subtract about 150000 or so (one day are 86400 seconds) and type in the result as timestamp.

Now we must prepare the packet files we defined above. Newsflash can optimize very good the data transmission parallelizing more than one packet. Here we tell it how to execute the transmission:

    /var/lib/news/packet1  # Slackware: /usr/lib/news/newsflash/packet1
      de.comm.isdn
      fido.ger.isdn
      de.comp.os.linux.networking

    /var/lib/news/packet2  # Slackware: /usr/lib/news/newsflash/packet2
      de.comp.os.linux.networking
      fido.ger.isdn
      de.comm.isdn

    /var/lib/news/packet3  # Slackware: /usr/lib/news/newsflash/packet3
      de.comp.os.linux.misc
      de.comp.os.linux.hardware
      de.comp.os.linux.x
      lrz.*

    /var/lib/news/packet4  # Slackware: /usr/lib/news/newsflash/packet4
      lrz.*
      de.comp.os.linux.x
      de.comp.os.linux.hardware
      de.comp.os.linux.misc

As you see the packets 1/2 and 3/4 have the same contents, but in reverse order. So one packet is executed from both ends, and the transmission is optimized. The packets should have the same estimated number of postings.

#### Installation of `postit`

Again here: before compilation you should edit `/usr/src/postit/config.h` and correct the defined paths there (`/var/spool/news`). Most times this file should be ok.

      root:# cd /usr/src/postit
      root:# make dep
      ...
      root:# make
      ...
      root:# cp postit /usr/lib/news/bin
      root:# cp samples/sendnews /usr/lib/news/bin
      root:# _

The last file we copied here (`/usr/lib/news/bin/sendnews` ) must be edited:

    /usr/lib/news/bin/sendnews
      SPOOLPATH=/var/spool/news
      OUTGOING=${SPOOLPATH}/out.going
      BINPATH=/usr/lib/news/bin

      ${BINPATH}/ctlinnd flush get
      mv ${OUTGOING}/get ${OUTGOING}/get.postit
      ${BINPATH}/ctlinnd begin get
      ${BINPATH}/postit get.postit getnews

### Another Big Question: Working Anyway?

If you call now `newsflash` the telephone wire should get hot now for some minutes. You should be able to see what's going on on your console. The messages may seem to be mis-ordered, because the process is parallelized.

If it does not work, you must look for the bug. Read the README. In most cases you need to edit again one of the files `hosts.nntp, nnrp.access, newsfeeds` etc. and play with the entries.

If it works you can read now the news groups with a news reader like `nn, tin` or Netscape).

To fetch the news automatically you may use the following script, which is very similar to the email fetcher script.

    /sbin/transnews
      #!/bin/sh
      # transnews: gets and sends usenet news.
      # Author:    Bernhard Hailer <dl4mhk@lrz.uni-muenchen.de>
      # Version:   4.0 (27-Jan-97)

      PATH=/sbin:/usr/sbin:/bin:/usr/bin

      . /etc/i4l.provider
      . /etc/i4l.secrets

      cd /                                # against PATH=. effect

      echo -----------------------------------------------------------------------
      echo -e "Starting news transfer: \c"; date

      . /sbin/connect on

      echo "Checking news server..."
      set `ping -qc5 -i1 getnews | grep transmitted`
      if [ $4 -gt 0 ];
      then
        echo "8-) News server online - starting news poll"
        echo "Sending news..."
        su news -c "/usr/lib/news/bin/sendnews"
        echo "Again sleeping 1 sec for PPP handshaking"
        sleep 1
        echo "Fetching news..."
        su news -c "/usr/lib/news/bin/newsflash -v"
        echo "Again sleeping 1 sec for PPP handshaking"
        sleep 1
      else
        echo ":-[ Sorry - news server is down. Try later..."
      fi

      . /sbin/isdn off

      echo -e "Finished news transfer: \c"; date

## Does Your Computer Run All the Time?

Yes? Then you should edit the `crontab` of `root` and add the following:

      # Transfer news every night at 04:29 a.m.
      29 04 * * *       /sbin/transnews 1>> /var/log/news-log 2>> /var/log/news-log

As described in the email chapter cron must be called with the new crontab now. The script scould be started one minute before the email transfer starts (not at the same time!!), so perhaps one tariff unit can be saved. If you would start the transfer at the same time, one of the applications would see a open line where no one is (opening a line takes some time). The connection would be interrupted.

Have Fun in the Usenet!!