28 November 2011

Packetlogger documentation: nmea0183-serial and iptee

Although I love NMEA 2000 there is a role for (some) continued NMEA 0183 use. Since last week I run a Digital Yacht AIS receiver at home and upload the resulting AIS data to the internet. I'm station 727 at Marine Traffic and I also contribute the data to AISHUB.

This is where nmea0183-serial and iptee come in. These are new Linux and OS X programs available at my website as part of the packetlogger download (hey, maybe I should rename that!)

The nmea0183-serial program reads data from a serial port at 38,400 baud and sends it to stdout. The iptee program reads lines from stdin and sends it to any number of TCP or UDP servers. To make sure it does not block one service when another is unavailable it drops messages when a service is not available.

I use this as follows:
nmea0183-serial -r /dev/ttyUSB0 | 
   iptee -u data.aishub.net 2112 -u 195.251.168.18 5321 |
   tee /volume1/ais/ais.log
In the above example I feed the data to AIShub.net and Marine traffic via two UDP channels, and I log the data on disk as well. For display reasons the code was split over multiple lines; it was originally a single line.

Packetlogger documentation: n2kd - producing a json server for a webpage

As you may have guessed from the -json option to the analyzer, I use it myself to produce data that is consumed by a webpage.
The webpage reads the data using AJAX and XmlHttpRequest. But for that to work nicely it needs a server that produces JSON, and that JSON should be as short as possible with each PGN only produced once. In other words, a webpage that is updated every 2 seconds does not want to know about the 19 rudder angle changes that were sent on the CAN bus in the mean time, only about the last one (or even better, the average -- but that is something my code doesn't do yet). Furthermore, if it is interested in AIS messages it wants to know about all AIS messages, not just the ones produced in the last two seconds.
So that's where a new program comes in, called n2kd (for NMEA 2000 deamon). It is available for Linux and OS X. This isn't very configurable yet, but I also don't know if anyone is interested. It is what I am using on-board at the moment and can be improved considerably. Let me know if you think this is useful if only it had such-and-such feature and I will consider it!
Here's the actual command that I use on board:
actisense-serial -r $ACTISENSE_PRIMARY | 
  analyzer -clocksrc 35 -json |
  n2kd
As you can see my GPS has CAN bus ID 35, so I give that to the analyzer to correct the computer's clock based on the GPS data.
The N2KD program is then contacted on port 597, and it will wait for 2 seconds and then spit out a full JSON structure containing all knowledge it has over all PGNs.
Note that this program will intentionally crash when it has no further input. I'm using a watchdog to restart the above command string when something goes wrong. It should probably just quit instead, but that's for next time.

Packetlogger documentation: the analyzer program

The analyzer program contains the 'secret sauce'. It uses a database of PGN contents that has been painstakingly reverse engineered from observation of my network. I have not seen, read or otherwise obtained any material that I was not allowed to redistribute. Again this utility is very simple. It consumes the 'raw' ASCII PGNs produced by the reader utilities on stdin, and writes out the parsed PGNs. In a second mode it can also produce a human or computer readable form of the PGN database that it contains. So if you have saved the output of the reader in a file, you can analyze the data at a later data like this:
analyzer < your-actisense-output.log
For instance if we do that for the following output:
2011-11-24-22:42:04.388,2,127251,36,255,8,7d,0b,7d,02,00,ff,ff,ff
2011-11-24-22:42:04.390,2,127250,36,255,8,00,5a,7c,00,00,00,00,fd
2011-11-24-22:42:04.437,2,130306,36,255,8,b1,5c,00,ee,f0,fa,ff,ff
and use the above command to analyze this we get:
N2K packet analyzer $Rev: 233 $ from $Date: 2011-11-27 22:21:08 +0100 (zo, 27 nov 2011) $
(C) 2009-2011 Keversoft B.V., Harlingen, The Netherlands
http://yachtelectronics.blogspot.com

New PGN 127251 for device 36 (heap 5452 bytes)
2011-11-24-22:42:04.388 2  36 255 127251 Rate of Turn:  SID = 125; Rate = 0.0934 deg/s
New PGN 127250 for device 36 (heap 5467 bytes)
2011-11-24-22:42:04.390 2  36 255 127250 Vessel Heading:  SID = 0; Heading = 182.4 deg; Deviation = 0.0 deg; Variation = 0.0 deg; Reference = Magnetic
New PGN 130306 for device 36 (heap 5480 bytes)
2011-11-24-22:42:04.437 2  36 255 130306 Wind Data:  SID = 177; Wind Speed = 0.92 m/s; Wind Angle = 353.4 deg; Reference = Apparent
Now this output is readable for humans, but not very good for computers to parse and process further. If you want to use the analyzer in your software you're better of using the -json option which will produce lines like this:
{"timestamp":"2011-11-24-22:42:04.388","prio":"2","src":"36","dst":"255","pgn":"127251","description":"Rate of Turn","fields":{"SID":"125","Rate":"0.0934"}}
{"timestamp":"2011-11-24-22:42:04.390","prio":"2","src":"36","dst":"255","pgn":"127250","description":"Vessel Heading","fields":{"SID":"0","Heading":"182.4","Deviation":"0.0","Variation":"0.0","Reference":"Magnetic"}}
{"timestamp":"2011-11-24-22:42:04.437","prio":"2","src":"36","dst":"255","pgn":"130306","description":"Wind Data","fields":{"SID":"177","Wind Speed":"0.92","Wind Angle":"353.4","Reference":"Apparent"}}
The analyzer's second mode of producing the database that is kept within it can be accessed with the -explain and -explain-xml arguments. Further arguments are used when you are analyzing a particular PGN or a particular device.
ArgumentPurpose
-rawShow the raw bytes of the message data following the parsed data
-debugShows internal values of the parser; probably not readable for anyone but me.
-geo {dd|dm|dms}Choose which format is used to print geographical locations.
-jsonFormat the output as JSON computer readable data.
-src <src>Ignore input coming from other sources than the specified one.
<pgn>Ignore PGNs other than this one.
-clocksrc <src>Set the computer's clock with data from this source.
-explainProduce the human readable list of PGNs that the parser understands.
-explain-xmlProduce the computer readable list, in XML format, of PGNs that the parser understands.
The -clocksrc argument is only supported on Linux and OS X. There is a -data argument, but that is not complete yet and will crash the analyzer.

Packetlogger documentation: ASCII raw format explained

In this post we'll go into the format produced by the reader programs and consumed by the analyzer program. Here's a typical example of some typical PGNs that have 8 bytes of data:
2011-11-24-22:42:04.388,2,127251,36,255,8,7d,0b,7d,02,00,ff,ff,ff
2011-11-24-22:42:04.390,2,127250,36,255,8,00,5a,7c,00,00,00,00,fd
2011-11-24-22:42:04.437,2,130306,36,255,8,b1,5c,00,ee,f0,fa,ff,ff
2011-11-24-22:42:04.490,2,127251,36,255,8,7e,0b,7d,02,00,ff,ff,ff
2011-11-24-22:42:04.493,2,127250,36,255,8,01,5a,7c,00,00,00,00,fd

8 bytes is the usual amount because that can be transmitted in a single CAN message. Longer messages are transmitted in multiple CAN messages. The reader programs 'solve' this and only send out coalesced complete PGN messages, for example:
2011-11-24-22:41:10.034,6,129540,36,255,123,24,ff,0a,03,68,03,a2,0d,00,00,ff,ff,ff,7f,f1,05,44,2a,c2,9a,04,10,ff,ff,ff,7f,f2,07,dc,17,e7,28,04,10,ff,ff,ff,7f,f2,08,0a,2f,0a,2f,e4,0c,ff,ff,ff,7f,f2,0a,f4,0c,c3,6d,68,10,ff,ff,ff,7f,f2,0f,f4,0c,9e,c1,10,0e,ff,ff,ff,7f,f2,13,68,03,16,22,00,00,ff,ff,ff,7f,f1,15,7f,07,ef,de,48,0d,ff,ff,ff,7f,f2,1a,dc,26,fb,c2,a0,0f,ff,ff,ff,7f,f2,1c,2e,17,2c,62,10,0e,ff,ff,ff,7f,f2

So what is this format?

Packetlogger Documentation: Getting data out of your Actisense NGT-1

The programs that I make available are all command line utilities. Normal for UNIX folks, somewhat exotic for OS X folks and downright frightening for Microsoft Windows users. Or maybe the other way around. Anyway, I don't have the time to develop GUI programs.

If you are not used to command line tools, and don't fancy learning them these programs are not for you.

So the first thing you need to do is open up a terminal or console. Or for those still used to that terminology, open up a DOS box. It hasn't been a DOS box for ten years, but that's what my sailing friends still call this. Click Start -> Run then type cmd.exe<enter> to get a console.

On Microsoft Windows you'll use the program actisense-reader.exe. First run it without any arguments. It will show you a list of COM ports. One of them should be your Actisense NGT-1 NMEA 2000 gateway, and it should say "Available" as well.

For instance:
C:\packetlogger\windows-x86>actisense-reader.exe
actisenser-reader $Rev: 235 $ $Date: 2011-11-28 09:23:14 +0100 (ma, 28 nov 2011) $
(C) 2009-2011 Keversoft, Harlingen, The Netherlands.

Usage: actisense-reader.exe <com-port> [<baud-rate>]

<com-port> is an integer from the following list:
1: COM1 - Communications Port (Available)
2: COM2 - Communications Port (Available)
3: COM3 - Actisense NGW/NGT NMEA 2000 Gateway (Available)

<baud-rate> is an integer from the following list: 4800 38400 115200 230400 (default is 115200)

This program reads from the indicated COM port until input on stdin is received. Normally this means until you press the 'Enter' key on the keyboard.
Once you have found the appropriate free NGT com port you can start the program again and pass it the number of the COM port, for instance:
C:\packetlogger\windows-x86>actisense-reader.exe 3
If your NGT-1 is connected to a network you'll see the program start emitting ASCII data, one line per NMEA 2000 PGN:
2011-11-24-22:42:04.388,2,127251,36,255,8,7d,0b,7d,02,00,ff,ff,ff
2011-11-24-22:42:04.390,2,127250,36,255,8,00,5a,7c,00,00,00,00,fd
2011-11-24-22:42:04.437,2,130306,36,255,8,b1,5c,00,ee,f0,fa,ff,ff
2011-11-24-22:42:04.490,2,127251,36,255,8,7e,0b,7d,02,00,ff,ff,ff
2011-11-24-22:42:04.493,2,127250,36,255,8,01,5a,7c,00,00,00,00,fd
On Linux and OS X the program to use is called actisense-serial, and it needs a device name. You'll have to dig around in your OS console logs (OS X: console.app, Linux: dmesg) to see how your NGT-1 has been recognized by the operating system. Let's assume it is /dev/ttyUSB0, in that case you use
actisense-serial -r /dev/ttyUSB0
Why the -r? Well, this program can also write PGNs to the bus using the same ASCII protocol as it uses to receive PGNs. This means that if you have, for instance, two NMEA2000 networks and you want messages to be passed from network A to B -- in other words read from A and write to B you do this like this:
actisense-serial -r /dev/networkA | actisense-serial -w /dev/networkB
Or if you want to read PGNs from A, inject those into B and then view the output from both networks you do this like this:
actisense-serial -r /dev/networkA | actisense-serial /dev/networkB | ....
If you do not want to inject the messages from A into network B, but you do want to view the output from both networks you use the 'pass thru' option:
actisense-serial -r /dev/networkA | actisense-serial -p /dev/networkB | ....
In a next blog post I'll show you what the output format generated by these programs contains and what you can do with the output.

Packetlogger Late 2011 edition

This blog has been pretty quiet over the last few months, but now I've finally found some time to update the public version of my NMEA 2000 reverse engineering research and NMEA utility programs. In the last year I have had the chance to observe my personal network and fix a large number of small bugs in the definitions. It now looks as if my programs and database are able to decode all common NMEA 2000 messages and a number of the proprietary ones that occur on my own network. This means that the following messages are all in order:
  • Basic NMEA 2000 (ISO request, claim, acknowledgement etc.)
  • Basic sensor data (wind, depth, heading, GPS, rudder)
  • Some electrical sensor data (switch status, battery, inverter, …)
  • Normal AIS messages (class A, class B)
  • The 'basic' route message
What I have not tested are things such as:
  • AIS uncommon messages such as SAR
  • Engine parameters, transmission parameters, trip parameters
  • DSC call information
  • Most route and WP service messages
This does not mean that these messages aren't in the database or that they are incorrect -- just that I don't know exactly how well they match up. For these 'incomplete' messages your help is appreciated. In my next post I'll tell you how you can do this. Also I'll be publishing some documentation posts on how to use the programs. To download the programs please go to the packet logger page on www.keversoft.com.