bug#11749: 24.1; C-mode indentation gives wrong-type-argument error.

bug#11749: 24.1; C-mode indentation gives wrong-type-argument error.

[Kim F. Storm]

Every now and then, while editing some C files, indentation fails
with a wrong-type-argument number-or-marker-p error.

I have seen it happen when yanking text - but also just typing code
and all of a sudden it fails to indent after a {.

Here is the last incident:

bug#11749: 24.1; C-mode indentation gives wrong-type-argument error.

[Glenn Morris]

Kim F. Storm wrote:

> Here is the last incident:

At this point, the report ended... :)
Could you (re)send the details?
Does they resemble http://debbugs.gnu.org/9957 ?

bug#11749: 24.1; C-mode indentation gives wrong-type-argument error.

[Kim Storm]

On 06/21/2012 09:17 AM, Glenn Morris wrote:
> Kim F. Storm wrote:
>
>> Here is the last incident:
> At this point, the report ended... :)
> Could you (re)send the details?
> Does they resemble http://debbugs.gnu.org/9957 ?
>
I no longer have the original message - but yes, from what I remeber, it 
resembles that other bug.
It was also related to indentation.  I'll send another backtrace next 
time it happens.

About the truncated mail:

I remember that while sending the mail, emacs barfed that there were NUL 
characters (or similar) in the message
and what I would like to do about it.  I hit the "d" option (delete 
non-printable characters and send the messsage).
Seems it removed a lot more than just non-printable characters (or is 
has a rather stupid interpretation of "non-printing").

Also the message log contains only the truncated message.

Still that's a different issue.

Kim

bug#11749: 24.2; wrong-type-argument

[Kim F. Storm]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 5756 bytes --]


Tried 24.2 RC2 - and once again I hit this annoying indentation bug i c mode:


In GNU Emacs 24.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.10)
 of 2012-08-24 on kfs-lx2
Windowing system distributor `The X.Org Foundation', version 11.0.11104000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: en_DK.utf8
  value of $LC_NUMERIC: en_DK.utf8
  value of $LC_TIME: en_DK.utf8
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Debugger

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  display-time-mode: t
  cua-mode: t
  ido-everywhere: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <S-right> <S-right> 
o b j C-z <left> <left> <S-right> <S-right> <S-right> 
o b j M-v <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <left> <left> 
<delete> <delete> <delete> x t r M-v <backspace> <backspace> 
<backspace> <backspace> x x ½ <end> <down> <up> <up> 
<up> <up> <up> <up> <left> <C-left> <C-left> <left> 
1 <down> <left> 1 <down> <left> 1 <down> <left> <M-backspace> 
<C-right> 1 <C-right> <C-right> <up> <up> <C-left> 
M-d a c t <down> <left> <left> <left> M-d o b j <down> 
<backspace> <backspace> <backspace> x t r <right> <S-end> 
<S-left> <delete> <up> <up> <up> <C-left> <C-left> 
<C-left> <C-left> <C-left> <C-left> <C-return> <down> 
<down> <down> <right> <right> <right> <right> <right> 
<right> <left> <delete> <right> <right> <right> <right> 
<right> <delete> <down> <delete> <down> <delete> <down> 
<delete> <down> <down> <left> <S-end> <S-left> <delete> 
( x d o m SPC | SPC x a c t SPC ' <backspace> | SPC 
a x <backspace> o b j <backspace> <backspace> <backspace> 
<backspace> x o b j SPC | SPC x x t r ) SPC & S-SPC 
0 x 8 0 <home> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> ( ! <end> ) <down> 
<down> <down> <left> SPC & & S-SPC <up> <up> <up> <C-left> 
<C-left> <C-left> <C-left> <C-right> <right> <right> 
<S-end> C-c <timeout> <down> <down> <down> <left> C-v 
<C-left> <right> <right> <delete> 1 <backspace> <right> 
1 <C-left> <C-left> <C-left> <C-left> <left> <left> 
<left> M-d <end> <backspace> <backspace> ) <up> <S-down> 
<S-down> <S-down> <S-down> <S-down> <S-down> C-c <timeout> 


Recent messages:
Mark set
Undo!
Mark set [4 times]
Auto-saving...done
Mark set [3 times]
Undo!
Mark set
Auto-saving...done
Mark set
Entering debugger...

Load-path shadows:
/home/kfs/emacs/common/emacs/user/kfs/nero hides /home/kfs/emacs/common/emacs/lisp/nero
/home/kfs/emacs/common/emacs/lisp/mwheel hides /home/kfs/Downloads/emacs-24.2/lisp/mwheel
/home/kfs/emacs/common/emacs/lisp/misc hides /home/kfs/Downloads/emacs-24.2/lisp/misc
/home/kfs/emacs/common/emacs/user/kfs/vcursor hides /home/kfs/Downloads/emacs-24.2/lisp/vcursor
/home/kfs/emacs/common/emacs/lisp/avoid hides /home/kfs/Downloads/emacs-24.2/lisp/avoid
/home/kfs/emacs/common/emacs/lisp/printing hides /home/kfs/Downloads/emacs-24.2/lisp/printing
/home/kfs/emacs/common/emacs/lisp/pong hides /home/kfs/Downloads/emacs-24.2/lisp/play/pong
/home/kfs/emacs/common/emacs/lisp/smtpmail hides /home/kfs/Downloads/emacs-24.2/lisp/mail/smtpmail

Features:
(shadow emacsbug message mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mail-utils debug perl-mode make-mode conf-mode
jka-compr woman man skeleton tramp-cache tramp-sh tramp tramp-compat
auth-source eieio assoc gnus-util mm-util mail-prsvr password-cache
format-spec advice help-fns advice-preload tramp-loaddefs sh-script
executable help-mode view vc-annotate log-view vc-rcs js byte-opt
bytecomp byte-compile cconv macroexp json thingatpt newcomment
log-edit add-log diff-mode vc ediff-merg ediff-diff ediff-wind
ediff-help ediff-util ediff-mult ediff-init ediff vc-dispatcher
sgml-mode cua-gmrk cc-mode cc-fonts cc-guess cc-menus cc-cmds
find-dired grep-x grep compile pcmpl-unix shell pcomplete dabbrev
tabify cua-rect rect misearch multi-isearch parse-time vc-cvs pcvs
pcvs-parse pcvs-info pcvs-defs easy-mmode pcvs-util ewoc dired imenu
cgi-mode derived windmove kmacro paren keypad time cua-base cus-start
cus-load telnet mailcrypt rfc822 comint ansi-color ring easymenu make
begendol ido server cc-styles cc-align cc-engine cc-vars cc-defs
regexp-opt disp-table match aux-fct common printing ps-print ps-def
lpr time-date tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
loaddefs button faces cus-face files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar
gtk x-toolkit x multi-tty emacs)


-- 
Kim F. Storm  <storm@cua.dk>      http://www.cua.dk

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

M-x report-emacs-bug obviously refuses to include any useful information 
-- so I send it through my Thunderbird client... this time.

Tried 24.2 RC2 - and once again I hit this annoying indentation bug i c 
mode:

Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 
(5457 . 5720))
   c-guess-basic-syntax()
   c-indent-line()
   indent-according-to-mode()
   c-electric-paren(nil)
   call-interactively(c-electric-paren nil nil)

The offsets corresponds to a preceding { ... } block.

The code I was working on is the following (sorry, I cannot show more 
than that):

       if (!((xdom | xact | xobj | xxtr) & 0x80))
     continue;

       if (domains && !alldom && !((xdom | xobj | xxtr) & 0x01))
     {


I get a lot of errors -- making 24.2 impossible to work with for my C code.

Here are some more backtraces:

Debugger entered--Lisp error: (error "Invalid search bound (wrong side 
of point)")
   re-search-backward("^[     ]*:[     ]*\\(.\\)?" 5920 t)
   c-lineup-gcc-asm-reg((arglist-cont-nonempty . 5920))
   c-evaluate-offset(c-lineup-gcc-asm-reg (arglist-cont-nonempty 5920 
5950) arglist-cont-nonempty)
   c-evaluate-offset((c-lineup-gcc-asm-reg c-lineup-arglist) 
(arglist-cont-nonempty 5920 5950) arglist-cont-nonempty)
   c-calc-offset((arglist-cont-nonempty 5920 5950))
   c-get-syntactic-indentation(((arglist-cont-nonempty 5920 5950)))
   c-indent-line()
   indent-according-to-mode()
   c-electric-paren(nil)
   call-interactively(c-electric-paren nil nil)


Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 
(5874 . 5881))
   c-guess-basic-syntax()
   c-indent-line()
   indent-according-to-mode()
   c-electric-paren(nil)
   call-interactively(c-electric-paren nil nil)

The last one came when I inserted the final ; in the last line of the 
following snippet:

static void add_hash_ext(idhash_t **hp, id_t *idtab, int n)
{

  }


   static idhash_t *lookup_ext(idhash_t **hp, );


Yes, there are syntax errors in that code -- but indentation should be 
able to
handle that without signalling errors all the time.



I have seen similar problems in 24.1 - but not in 23.4 that I usually
use (for this very reason).


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
     `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/home/kfs/Downloads/emacs-24.2/etc/DEBUG.


In GNU Emacs 24.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.10)
  of 2012-08-24 on kfs-lx2
Windowing system distributor `The X.Org Foundation', version 11.0.11104000
Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: en_DK.utf8
   value of $LC_NUMERIC: en_DK.utf8
   value of $LC_TIME: en_DK.utf8
   value of $LANG: en_US.utf8
   value of $XMODIFIERS: nil
   locale-coding-system: utf-8-unix
   default enable-multibyte-characters: t

Major mode: Debugger

Minor modes in effect:
   diff-auto-refine-mode: t
   shell-dirtrack-mode: t
   display-time-mode: t
   cua-mode: t
   ido-everywhere: t
   tooltip-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <S-right> <S-right>
o b j C-z <left> <left> <S-right> <S-right> <S-right>
o b j M-v <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <right> <right>
<right> <right> <right> <right> <right> <left> <left>
<delete> <delete> <delete> x t r M-v <backspace> <backspace>
<backspace> <backspace> x x ½ <end> <down> <up> <up>
<up> <up> <up> <up> <left> <C-left> <C-left> <left>
1 <down> <left> 1 <down> <left> 1 <down> <left> <M-backspace>
<C-right> 1 <C-right> <C-right> <up> <up> <C-left>
M-d a c t <down> <left> <left> <left> M-d o b j <down>
<backspace> <backspace> <backspace> x t r <right> <S-end>
<S-left> <delete> <up> <up> <up> <C-left> <C-left>
<C-left> <C-left> <C-left> <C-left> <C-return> <down>
<down> <down> <right> <right> <right> <right> <right>
<right> <left> <delete> <right> <right> <right> <right>
<right> <delete> <down> <delete> <down> <delete> <down>
<delete> <down> <down> <left> <S-end> <S-left> <delete>
( x d o m SPC | SPC x a c t SPC ' <backspace> | SPC
a x <backspace> o b j <backspace> <backspace> <backspace>
<backspace> x o b j SPC | SPC x x t r ) SPC & S-SPC
0 x 8 0 <home> <right> <right> <right> <right> <right>
<right> <right> <right> <right> ( ! <end> ) <down>
<down> <down> <left> SPC & & S-SPC <up> <up> <up> <C-left>
<C-left> <C-left> <C-left> <C-right> <right> <right>
<S-end> C-c <timeout> <down> <down> <down> <left> C-v
<C-left> <right> <right> <delete> 1 <backspace> <right>
1 <C-left> <C-left> <C-left> <C-left> <left> <left>
<left> M-d <end> <backspace> <backspace> ) <up> <S-down>
<S-down> <S-down> <S-down> <S-down> <S-down> C-c <timeout>


Recent messages:
Mark set
Undo!
Mark set [4 times]
Auto-saving...done
Mark set [3 times]
Undo!
Mark set
Auto-saving...done
Mark set
Entering debugger...

Load-path shadows:
/home/kfs/emacs/common/emacs/user/kfs/nero hides 
/home/kfs/emacs/common/emacs/lisp/nero
/home/kfs/emacs/common/emacs/lisp/mwheel hides 
/home/kfs/Downloads/emacs-24.2/lisp/mwheel
/home/kfs/emacs/common/emacs/lisp/misc hides 
/home/kfs/Downloads/emacs-24.2/lisp/misc
/home/kfs/emacs/common/emacs/user/kfs/vcursor hides 
/home/kfs/Downloads/emacs-24.2/lisp/vcursor
/home/kfs/emacs/common/emacs/lisp/avoid hides 
/home/kfs/Downloads/emacs-24.2/lisp/avoid
/home/kfs/emacs/common/emacs/lisp/printing hides 
/home/kfs/Downloads/emacs-24.2/lisp/printing
/home/kfs/emacs/common/emacs/lisp/pong hides 
/home/kfs/Downloads/emacs-24.2/lisp/play/pong
/home/kfs/emacs/common/emacs/lisp/smtpmail hides 
/home/kfs/Downloads/emacs-24.2/lisp/mail/smtpmail

Features:
(shadow emacsbug message mml mml-sec mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mail-utils debug perl-mode make-mode conf-mode
jka-compr woman man skeleton tramp-cache tramp-sh tramp tramp-compat
auth-source eieio assoc gnus-util mm-util mail-prsvr password-cache
format-spec advice help-fns advice-preload tramp-loaddefs sh-script
executable help-mode view vc-annotate log-view vc-rcs js byte-opt
bytecomp byte-compile cconv macroexp json thingatpt newcomment
log-edit add-log diff-mode vc ediff-merg ediff-diff ediff-wind
ediff-help ediff-util ediff-mult ediff-init ediff vc-dispatcher
sgml-mode cua-gmrk cc-mode cc-fonts cc-guess cc-menus cc-cmds
find-dired grep-x grep compile pcmpl-unix shell pcomplete dabbrev
tabify cua-rect rect misearch multi-isearch parse-time vc-cvs pcvs
pcvs-parse pcvs-info pcvs-defs easy-mmode pcvs-util ewoc dired imenu
cgi-mode derived windmove kmacro paren keypad time cua-base cus-start
cus-load telnet mailcrypt rfc822 comint ansi-color ring easymenu make
begendol ido server cc-styles cc-align cc-engine cc-vars cc-defs
regexp-opt disp-table match aux-fct common printing ps-print ps-def
lpr time-date tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core frame cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
loaddefs button faces cus-face files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar
gtk x-toolkit x multi-tty emacs)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Eli Zaretskii]

> Date: Wed, 29 Aug 2012 00:49:53 +0200
> From: Kim Storm <storm@cua.dk>
> 
> M-x report-emacs-bug obviously refuses to include any useful information 

How does it refuse to do that?  Any error messages, or some other
hints?

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2012-08-31 13:01, Eli Zaretskii wrote:
>> Date: Wed, 29 Aug 2012 00:49:53 +0200
>> From: Kim Storm<storm@cua.dk>
>>
>> M-x report-emacs-bug obviously refuses to include any useful information
> How does it refuse to do that?  Any error messages, or some other
> hints?
>
No, it sent the message, but the text I had pasted into the mail buffer 
had vanished
when I checked in debbugs. So fortunately I had saved a copy of the message
(based on previous experience *) so I could send it again.

*) The first time I sent that bug report emacs had removed the entire 
body (at that
time I was asked something about problems with the character set or 
something,
and I guess I said "go ahead").

Still the original problem makes 24.1 and 24.2 absolutely useless for me.
It works fine for some time -- and then suddenly it barfs about 
indentation all the time.
I know I should start from -Q etc. - but I need to get some work done ...

I really wonder why I'm the only one seeing this -- can my C style 
really be that different :-)

BTW - 23.4 has quite some display problems with Debian / Gnome - it 
leaves highlight artifacts
and missing screen updates when scrolling and when killing and inserting 
text
  ... those problems are still present in 24.2 ... and maybe they've 
gotten worse.

I remember having seen some bug reports about it, so I haven't bothered
reporting them -- but it happens "all the time".

Kim

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hello, Kim.

On Wed, Aug 29, 2012 at 12:49:53AM +0200, Kim Storm wrote:
> M-x report-emacs-bug obviously refuses to include any useful information 
> -- so I send it through my Thunderbird client... this time.

> Tried 24.2 RC2 - and once again I hit this annoying indentation bug i c 
> mode:

> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 
> (5457 . 5720))
>    c-guess-basic-syntax()
>    c-indent-line()
>    indent-according-to-mode()
>    c-electric-paren(nil)
>    call-interactively(c-electric-paren nil nil)

> The offsets corresponds to a preceding { ... } block.

Yes.  It looks like the "state cache" (a cache of certain brace, paren
and bracket positions) has got corrupted.  This is difficult to make
repeatable, and is very dependent upon the exact navigation taken around
the C buffer.

> The code I was working on is the following (sorry, I cannot show more 
> than that):

>        if (!((xdom | xact | xobj | xxtr) & 0x80))
>      continue;

>        if (domains && !alldom && !((xdom | xobj | xxtr) & 0x01))
>      {


> I get a lot of errors -- making 24.2 impossible to work with for my C code.

> Here are some more backtraces:

> Debugger entered--Lisp error: (error "Invalid search bound (wrong side 
> of point)")
>    re-search-backward("^[     ]*:[     ]*\\(.\\)?" 5920 t)
>    c-lineup-gcc-asm-reg((arglist-cont-nonempty . 5920))
>    c-evaluate-offset(c-lineup-gcc-asm-reg (arglist-cont-nonempty 5920 
> 5950) arglist-cont-nonempty)
>    c-evaluate-offset((c-lineup-gcc-asm-reg c-lineup-arglist) 
> (arglist-cont-nonempty 5920 5950) arglist-cont-nonempty)
>    c-calc-offset((arglist-cont-nonempty 5920 5950))
>    c-get-syntactic-indentation(((arglist-cont-nonempty 5920 5950)))
>    c-indent-line()
>    indent-according-to-mode()
>    c-electric-paren(nil)
>    call-interactively(c-electric-paren nil nil)

That looks a bit like bug #10941, but isn't it.  It could be something
quite easy inside `c-lineup-gcc-asm-reg' (which I've not seen being used
before).  Is this an easily repeatable bug?  Any chance of a recipe?
Would you also dump your CC Mode configuration (C-c C-b) please.

> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p 
> (5874 . 5881))
>    c-guess-basic-syntax()
>    c-indent-line()
>    indent-according-to-mode()
>    c-electric-paren(nil)
>    call-interactively(c-electric-paren nil nil)

> The last one came when I inserted the final ; in the last line of the 
> following snippet:

> static void add_hash_ext(idhash_t **hp, id_t *idtab, int n)
> {

>   }


>    static idhash_t *lookup_ext(idhash_t **hp, );

I've not been able to reproduce it in that limited context.  It looks
like another bug in the state cache, and they're bastards.

> Yes, there are syntax errors in that code -- but indentation should be 
> able to
> handle that without signalling errors all the time.

Yes.

> I have seen similar problems in 24.1 - but not in 23.4 that I usually
> use (for this very reason).

Might you have something unusual in your CC Mode config?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2012-09-02 23:16, Alan Mackenzie wrote:
> Yes. It looks like the "state cache" (a cache of certain brace, paren 
> and bracket positions) has got corrupted.
> This is difficult to make repeatable, and is very dependent upon the 
> exact navigation taken around the C buffer.
Well - my navigation triggers that bug a lot.


>> Debugger entered--Lisp error: (error "Invalid search bound (wrong side
>> of point)")
>>     re-search-backward("^[     ]*:[     ]*\\(.\\)?" 5920 t)
>>     c-lineup-gcc-asm-reg((arglist-cont-nonempty . 5920))
>>     c-evaluate-offset(c-lineup-gcc-asm-reg (arglist-cont-nonempty 5920
>> 5950) arglist-cont-nonempty)
>>     c-evaluate-offset((c-lineup-gcc-asm-reg c-lineup-arglist)
>> (arglist-cont-nonempty 5920 5950) arglist-cont-nonempty)
>>     c-calc-offset((arglist-cont-nonempty 5920 5950))
>>     c-get-syntactic-indentation(((arglist-cont-nonempty 5920 5950)))
>>     c-indent-line()
>>     indent-according-to-mode()
>>     c-electric-paren(nil)
>>     call-interactively(c-electric-paren nil nil)
> That looks a bit like bug #10941, but isn't it.  It could be something
> quite easy inside `c-lineup-gcc-asm-reg' (which I've not seen being used
> before).  Is this an easily repeatable bug?  Any chance of a recipe?
> Would you also dump your CC Mode configuration (C-c C-b) please.
I think I only saw this once - the other bug already happened, and I 
just started typing some random code snippets, and this bug happened.
>
>> Debugger entered--Lisp error: (wrong-type-argument integer-or-marker-p
>> (5874 . 5881))
>>     c-guess-basic-syntax()
>>     c-indent-line()
>>     indent-according-to-mode()
>>     c-electric-paren(nil)
>>     call-interactively(c-electric-paren nil nil)
>
> I've not been able to reproduce it in that limited context.  It looks
> like another bug in the state cache, and they're bastards.
Any way of clearing the cache?
>
> Might you have something unusual in your CC Mode config?
I have this:

(modify-syntax-entry ?_ "w" c-mode-syntax-table)

But otherwise, I don't seem to have any customizations ... only an 
awkward programming style :-)


_Subject: CC Mode 5.32.3 (C/l); emacs bug 11749

_ X-Reporter-Void-Vars-Found: auto-fill-mode (setq
c-basic-offset 2
c-comment-only-line-offset '(0 . 0)
c-indent-comment-alist '((anchored-comment column . 0) (end-block space 
. 1) (cpp-end-block space . 2))
c-indent-comments-syntactically-p nil
c-block-comment-prefix ""
c-comment-prefix-regexp '((pike-mode . "//+!?\\|\\**") (awk-mode . "#+") 
(other . "//+\\|\\**"))
c-doc-comment-style '((java-mode . javadoc) (pike-mode . autodoc) 
(c-mode . gtkdoc))
c-cleanup-list '(scope-operator)
c-hanging-braces-alist '((substatement-open before after) 
(arglist-cont-nonempty))
c-hanging-colons-alist nil
c-hanging-semi&comma-criteria '(c-semi&comma-inside-parenlist)
c-backslash-column 48
c-backslash-max-column 72
c-special-indent-hook '(c-gnu-impose-minimum)
c-label-minimum-indentation 1
c-offsets-alist '((inexpr-class . +)
(inexpr-statement . +)
(lambda-intro-cont . +)
(inlambda . c-lineup-inexpr-block)
(template-args-cont c-lineup-template-args +)
(incomposition . +)
(inmodule . +)
(innamespace . +)
(inextern-lang . +)
(composition-close . 0)
(module-close . 0)
(namespace-close . 0)
(extern-lang-close . 0)
(composition-open . 0)
(module-open . 0)
(namespace-open . 0)
(extern-lang-open . 0)
(objc-method-call-cont c-lineup-ObjC-method-call-colons 
c-lineup-ObjC-method-call +)
(objc-method-args-cont . c-lineup-ObjC-method-args)
(objc-method-intro . [0])
(friend . 0)
(cpp-define-intro c-lineup-cpp-define +)
(cpp-macro-cont . +)
(cpp-macro . [0])
(inclass . +)
(stream-op . c-lineup-streamop)
(arglist-cont-nonempty c-lineup-gcc-asm-reg c-lineup-arglist)
(arglist-cont c-lineup-gcc-asm-reg 0)
(comment-intro c-lineup-knr-region-comment c-lineup-comment)
(catch-clause . 0)
(else-clause . 0)
(do-while-closure . 0)
(access-label . -)
(case-label . 0)
(substatement . +)
(statement-case-intro . +)
(statement . 0)
(brace-entry-open . 0)
(brace-list-entry . 0)
(brace-list-intro . +)
(brace-list-close . 0)
(block-close . 0)
(block-open . 0)
(inher-cont . c-lineup-multi-inher)
(inher-intro . +)
(member-init-cont . c-lineup-multi-inher)
(member-init-intro . +)
(annotation-var-cont . +)
(annotation-top-cont . 0)
(topmost-intro . 0)
(knr-argdecl . 0)
(func-decl-cont . +)
(inline-close . 0)
(class-close . 0)
(class-open . 0)
(defun-block-intro . +)
(defun-close . 0)
(defun-open . 0)
(c . c-lineup-C-comments)
(string . c-lineup-dont-change)
(topmost-intro-cont first c-lineup-topmost-intro-cont 
c-lineup-gnu-DEFUN-intro-cont)
(brace-list-open . +)
(inline-open . 0)
(arglist-close . c-lineup-arglist)
(arglist-intro . c-lineup-arglist-intro-after-paren)
(statement-cont . +)
(statement-case-open . +)
(label . 0)
(substatement-label . 0)
(substatement-open . +)
(knr-argdecl-intro . 5)
(statement-block-intro . +)
)
c-buffer-is-cc-mode 'c-mode
c-tab-always-indent t
c-syntactic-indentation t
c-syntactic-indentation-in-macros t
c-ignore-auto-fill '(string cpp code)
c-auto-align-backslashes t
c-backspace-function 'backward-delete-char-untabify
c-delete-function 'delete-char
c-electric-pound-behavior nil
c-default-style '((java-mode . "java") (awk-mode . "awk") (other . "gnu"))
c-enable-xemacs-performance-kludge-p nil
c-old-style-variable-behavior nil
defun-prompt-regexp nil
tab-width 8
comment-column 32
parse-sexp-ignore-comments t
parse-sexp-lookup-properties t
auto-fill-function nil
comment-multi-line t
comment-start-skip "\\(//+\\|/\\*+\\)\\s *"
fill-prefix nil
fill-column 70
paragraph-start "[ ]*\\(//+\\|\\**\\)[ ]*$\\|^\f"
adaptive-fill-mode t
adaptive-fill-regexp "[ ]*\\(//+\\|\\**\\)[ ]*\\([ 
]*\\([-–!|#%;>*·•‣⁃◦]+[ ]*\\)*\\)"
)

Emacs : GNU Emacs 24.2.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.10)
of 2012-08-24 on kfs-lx2
Package: CC Mode 5.32.3 (C/l)
Buffer Style: gnu
c-emacs-features: (pps-extended-state col-0-paren posix-char-classes 
gen-string-delim gen-comment-delim syntax-properties 1-bit)

current state:
==============

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Stefan Monnier]

> I have this:
> (modify-syntax-entry ?_ "w" c-mode-syntax-table)

I don't know if it's the culprit, or if Alan considers this is a broken
config, but at least for the major modes I maintain, I consider this as
a broken config.


        Stefan

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2012-09-03 15:56, Stefan Monnier wrote:
>> I have this:
>> (modify-syntax-entry ?_ "w" c-mode-syntax-table)
> I don't know if it's the culprit, or if Alan considers this is a broken
> config, but at least for the major modes I maintain, I consider this as
> a broken config.
>
Still, it never caused any problems in emacs 23.

Kim

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Stefan Monnier]

>>> I have this:
>>> (modify-syntax-entry ?_ "w" c-mode-syntax-table)
>> I don't know if it's the culprit, or if Alan considers this is a broken
>> config, but at least for the major modes I maintain, I consider this as
>> a broken config.
> Still, it never caused any problems in Emacs 23.

A broken config is not guaranteed to trigger problems ;-)
In any case, I do not know if it's the culprit (or even if Alan
considers it to be broken).  Still, I comment you try to remove it, to
see if it helps, and if it does, try to find some other way to get the
same result (e.g. using forward-symbol instead of forward-word).


        Stefan

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hello, Kim.

On Mon, Sep 03, 2012 at 11:47:00AM +0200, Kim Storm wrote:
> On 2012-09-02 23:16, Alan Mackenzie wrote:
> > Yes. It looks like the "state cache" (a cache of certain brace, paren 
> > and bracket positions) has got corrupted.
> > This is difficult to make repeatable, and is very dependent upon the 
> > exact navigation taken around the C buffer.
> Well - my navigation triggers that bug a lot.

It's strange that it doesn't seem to be happening much to other people.

[ .... ]

> > I've not been able to reproduce it in that limited context.  It looks
> > like another bug in the state cache, and they're bastards.
> Any way of clearing the cache?

M-: c-state-cache-init.  As a workaround, you could turn that into a
command and bind it to a spare C-c C-<letter>.

> > Might you have something unusual in your CC Mode config?
> I have this:

> (modify-syntax-entry ?_ "w" c-mode-syntax-table)

That shouldn't be causing the current problem.

> But otherwise, I don't seem to have any customizations ... only an 
> awkward programming style :-)

Thanks for the config!

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Kim.
>
> On Mon, Sep 03, 2012 at 11:47:00AM +0200, Kim Storm wrote:
>> On 2012-09-02 23:16, Alan Mackenzie wrote:
>> > Yes. It looks like the "state cache" (a cache of certain brace, paren 
>> > and bracket positions) has got corrupted.
>> > This is difficult to make repeatable, and is very dependent upon the 
>> > exact navigation taken around the C buffer.
>> Well - my navigation triggers that bug a lot.
>
> It's strange that it doesn't seem to be happening much to other people.

It happens to me irregularly.  There is generally no way to re-create
it, so I've stopped reporting these.  I wish there were some way to
record all actions in c-mode buffers such that they could be saved and
re-played when this type of problem happens.  If there were some sort of
debug flag I could turn on, I would turn it on by default and hopefully
be able to catch some useful information.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Stefan Monnier]

>>> > Yes. It looks like the "state cache" (a cache of certain brace, paren 
>>> > and bracket positions) has got corrupted.
>>> > This is difficult to make repeatable, and is very dependent upon the 
>>> > exact navigation taken around the C buffer.
>>> Well - my navigation triggers that bug a lot.
>> It's strange that it doesn't seem to be happening much to other people.
> It happens to me irregularly.  There is generally no way to re-create
> it, so I've stopped reporting these.  I wish there were some way to
> record all actions in c-mode buffers such that they could be saved and
> re-played when this type of problem happens.  If there were some sort of
> debug flag I could turn on, I would turn it on by default and hopefully
> be able to catch some useful information.

If the problem happens while browsing a file (i.e. without any
intervening buffer modifications), then a trace of calls to
font-lock-default-fontify-region (but only those in the affected buffer)
might do the trick.

This said, I'm wondering why cc-mode's cache of parsing state is
so fragile.


        Stefan

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2012-09-07 16:53, Stefan Monnier wrote:
> If the problem happens while browsing a file (i.e. without any
> intervening buffer modifications), then a trace of calls to
> font-lock-default-fontify-region (but only those in the affected buffer)
> might do the trick.
I have never seen it just by browsing a C file.

When I see it, I have made "a number of changes".

Maybe changes have to include kill&yanks.

As I said, it's hard to work with 24.x due to this bug, but I'll give it 
another try
now that I know how to clear the cache.

Kim

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hello again, Michael!

On Thu, Sep 06, 2012 at 11:45:48PM -0400, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Kim.

> > On Mon, Sep 03, 2012 at 11:47:00AM +0200, Kim Storm wrote:
> >> On 2012-09-02 23:16, Alan Mackenzie wrote:
> >> > Yes. It looks like the "state cache" (a cache of certain brace, paren 
> >> > and bracket positions) has got corrupted.
> >> > This is difficult to make repeatable, and is very dependent upon the 
> >> > exact navigation taken around the C buffer.
> >> Well - my navigation triggers that bug a lot.

> > It's strange that it doesn't seem to be happening much to other people.

> It happens to me irregularly.

Once a week?  Once a day?  Is it associated with buffer changes, or does
it just happen?

> There is generally no way to re-create it, so I've stopped reporting
> these.  I wish there were some way to record all actions in c-mode
> buffers such that they could be saved and re-played when this type of
> problem happens.  If there were some sort of debug flag I could turn
> on, I would turn it on by default and hopefully be able to catch some
> useful information.

M-x c-toggle-parse-state-debug

(or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
`c-debug-parse-state' is not buffer local.  I wrote this on 19th October
last year to help sort out the bug you reported a little earlier.  :-)

It works by calculating c-parse-state twice for each call - The first
time normally, then again with the internal state bound to "newly
initialised".  If the the two results differ, they are printed to
*Messages*, together with the saved previous state.  If this does
trigger, please note any recent buffer changes.  It may make editing
intolerably slow.

Kim, would you please try this out on a buffer you could actually send to
me.  Thanks!

> -- 
> Michael Welsh Duggan
> (md5i@md5i.com)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Hello again, Michael!
>
> On Thu, Sep 06, 2012 at 11:45:48PM -0400, Michael Welsh Duggan wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>> > Hello, Kim.
>
>> > On Mon, Sep 03, 2012 at 11:47:00AM +0200, Kim Storm wrote:
>> >> On 2012-09-02 23:16, Alan Mackenzie wrote:
>> >> > Yes. It looks like the "state cache" (a cache of certain brace, paren 
>> >> > and bracket positions) has got corrupted.
>> >> > This is difficult to make repeatable, and is very dependent upon the 
>> >> > exact navigation taken around the C buffer.
>> >> Well - my navigation triggers that bug a lot.
>
>> > It's strange that it doesn't seem to be happening much to other people.
>
>> It happens to me irregularly.
>
> Once a week?  Once a day?  Is it associated with buffer changes, or does
> it just happen?

It happens when I'm editing C code.  I can't say once a week/day, etc.,
because I don't edit C code every day.  Most days I don't see it at all.
When it does happen, it is either extremely ephemeral (and goes away
immediately), or happens frequently in the same file, and I have to
reload the file completely just to get back to status quo.  It's very
hard to characterize.  (It used to happen a lot more often before you
fixed this in a few places.)

>> There is generally no way to re-create it, so I've stopped reporting
>> these.  I wish there were some way to record all actions in c-mode
>> buffers such that they could be saved and re-played when this type of
>> problem happens.  If there were some sort of debug flag I could turn
>> on, I would turn it on by default and hopefully be able to catch some
>> useful information.
>
> M-x c-toggle-parse-state-debug
>
> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
> last year to help sort out the bug you reported a little earlier.  :-)
>
> It works by calculating c-parse-state twice for each call - The first
> time normally, then again with the internal state bound to "newly
> initialised".  If the the two results differ, they are printed to
> *Messages*, together with the saved previous state.  If this does
> trigger, please note any recent buffer changes.  It may make editing
> intolerably slow.

I will turn this on.

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Michael Welsh Duggan <mwd@cert.org> writes:

>>> There is generally no way to re-create it, so I've stopped reporting
>>> these.  I wish there were some way to record all actions in c-mode
>>> buffers such that they could be saved and re-played when this type of
>>> problem happens.  If there were some sort of debug flag I could turn
>>> on, I would turn it on by default and hopefully be able to catch some
>>> useful information.
>>
>> M-x c-toggle-parse-state-debug
>>
>> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
>> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
>> last year to help sort out the bug you reported a little earlier.  :-)
>>
>> It works by calculating c-parse-state twice for each call - The first
>> time normally, then again with the internal state bound to "newly
>> initialised".  If the the two results differ, they are printed to
>> *Messages*, together with the saved previous state.  If this does
>> trigger, please note any recent buffer changes.  It may make editing
>> intolerably slow.
>
> I will turn this on.

And I have already hit several warnings.  I noticed nothing when
editing.  The latest one seemed to happen when I was cut/pasting code
from one place to another.  Now that I noticed the problems, I can't
recreate them.  I will modify my sources to beep at me when the messages
come up again so I notice them more immediately.

c-parse-state inconsistency at 6033: using cache: (6031 (5900 . 6025) 5819 (5689 . 5767) 5456 2847 2824), from scratch: (6031 (5900 . 6025) 5819 (5689 . 5767) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6644 . 6646) 2847 2824)  c-state-cache-good-pos 6646  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5602)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6033: using cache: (6031 (5900 . 6025) 5819 (5689 . 5767) 5456 2847 2824), from scratch: (6031 (5900 . 6025) 5819 (5689 . 5767) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6031 (5900 . 6025) 5819 (5689 . 5767) 5456 2847 2824)  c-state-cache-good-pos 6032  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5602)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6044: using cache: (6042 (5911 . 6036) 5830 (5700 . 5778) 5456 2847 2824), from scratch: (6042 (5911 . 6036) 5830 (5700 . 5778) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6655 . 6657) 2847 2824)  c-state-cache-good-pos 6657  c-state-nonlit-pos-cache '(9001 6001 3001)  c-state-nonlit-pos-cache-limit 6001  c-state-brace-pair-desert '(2847 . 5582)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6044: using cache: (6042 (5911 . 6036) 5830 (5700 . 5778) 5456 2847 2824), from scratch: (6042 (5911 . 6036) 5830 (5700 . 5778) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6042 (5911 . 6036) 5830 (5700 . 5778) 5456 2847 2824)  c-state-cache-good-pos 6043  c-state-nonlit-pos-cache '(9001 6001 3001)  c-state-nonlit-pos-cache-limit 6001  c-state-brace-pair-desert '(2847 . 5582)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
Opening nnimap server on home...done
nnimap read 0k from localhost
No more unseen articles
No more unread articles
Mark set [5 times]
Undo! [12 times]
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6607 . 6609) 2847 2824)  c-state-cache-good-pos 6609  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6271: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6446: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6446: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6477  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6506: using cache: ((6257 . 6497) 5456 2847 2824), from scratch: ((6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6477  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6506: using cache: ((6257 . 6497) 5456 2847 2824), from scratch: ((6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6214: using cache: ((5759 . 6205) 5456 2847 2824), from scratch: ((5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6525: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6205  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6525: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6516 (6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6517: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6516 (6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6517  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6607 . 6609) 2847 2824)  c-state-cache-good-pos 6609  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6271: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6294: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6271: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6271: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6446: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6446: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6306: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6477: using cache: (6476 6257 (5759 . 6205) 5456 2847 2824), from scratch: (6476 6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6477  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6506: using cache: ((6257 . 6497) 5456 2847 2824), from scratch: ((6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6476 6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6477  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6506: using cache: ((6257 . 6497) 5456 2847 2824), from scratch: ((6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6214: using cache: ((5759 . 6205) 5456 2847 2824), from scratch: ((5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6525: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6205  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6525: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6516 (6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6497  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6517: using cache: (6516 (6257 . 6497) 5456 2847 2824), from scratch: (6516 (6257 . 6497) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6516 (6257 . 6497) 5456 2847 2824)  c-state-cache-good-pos 6517  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '((6607 . 6609) 2847 2824)  c-state-cache-good-pos 6676  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 6259: using cache: (6257 (5759 . 6205) 5456 2847 2824), from scratch: (6257 (5759 . 6205) 5456 (5210 . 5330) 2847 2824)
Old state:
(setq c-state-cache '(6257 (5759 . 6205) 5456 2847 2824)  c-state-cache-good-pos 6258  c-state-nonlit-pos-cache '(9010 6010 3001)  c-state-nonlit-pos-cache-limit 6010  c-state-brace-pair-desert '(2847 . 5546)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Michael Welsh Duggan <mwd@cert.org> writes:

> Michael Welsh Duggan <mwd@cert.org> writes:
>
>>>> There is generally no way to re-create it, so I've stopped reporting
>>>> these.  I wish there were some way to record all actions in c-mode
>>>> buffers such that they could be saved and re-played when this type of
>>>> problem happens.  If there were some sort of debug flag I could turn
>>>> on, I would turn it on by default and hopefully be able to catch some
>>>> useful information.
>>>
>>> M-x c-toggle-parse-state-debug
>>>
>>> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
>>> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
>>> last year to help sort out the bug you reported a little earlier.  :-)
>>>
>>> It works by calculating c-parse-state twice for each call - The first
>>> time normally, then again with the internal state bound to "newly
>>> initialised".  If the the two results differ, they are printed to
>>> *Messages*, together with the saved previous state.  If this does
>>> trigger, please note any recent buffer changes.  It may make editing
>>> intolerably slow.
>>
>> I will turn this on.

I turned this on, and inserted a (ding) where cc-mode outputs an
inconsistency message.  I can barely stand to edit code now, due to
what seem to be almost every key press causing a beep.  Have you made
any progress here, or do you need more data?

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael.

On Fri, Sep 21, 2012 at 01:47:15PM -0400, Michael Welsh Duggan wrote:
> Michael Welsh Duggan <mwd@cert.org> writes:

> > Michael Welsh Duggan <mwd@cert.org> writes:

> >>>> There is generally no way to re-create it, so I've stopped reporting
> >>>> these.  I wish there were some way to record all actions in c-mode
> >>>> buffers such that they could be saved and re-played when this type of
> >>>> problem happens.  If there were some sort of debug flag I could turn
> >>>> on, I would turn it on by default and hopefully be able to catch some
> >>>> useful information.

> >>> M-x c-toggle-parse-state-debug

> >>> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
> >>> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
> >>> last year to help sort out the bug you reported a little earlier.  :-)

> >>> It works by calculating c-parse-state twice for each call - The first
> >>> time normally, then again with the internal state bound to "newly
> >>> initialised".  If the the two results differ, they are printed to
> >>> *Messages*, together with the saved previous state.  If this does
> >>> trigger, please note any recent buffer changes.  It may make editing
> >>> intolerably slow.

> >> I will turn this on.

> I turned this on, and inserted a (ding) where cc-mode outputs an
> inconsistency message.  I can barely stand to edit code now, due to
> what seem to be almost every key press causing a beep.  Have you made
> any progress here, or do you need more data?

I have found the bug which is causing (most of) these dings, though I
don't think it is the one which caused Kim's original bug.  Could you try
out the patch below, please.  (I have also enhanced/corrected the
debugging routines a bit, too.)



diff -r ac6584d14c06 cc-engine.el
--- a/cc-engine.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-engine.el	Sun Oct 07 10:54:51 2012 +0000
@@ -2648,17 +2648,19 @@
 	;; If we're essentially repeating a fruitless search, just give up.
 	(unless (and c-state-brace-pair-desert
 		     (eq cache-pos (car c-state-brace-pair-desert))
+		     (> from (car c-state-brace-pair-desert))
 		     (<= from (cdr c-state-brace-pair-desert)))
-	  ;; DESERT-LIM.  Only search what we absolutely need to,
+	  ;; DESERT-LIM.  Avoid repeated searching through the cached desert.
 	  (let ((desert-lim
 		 (and c-state-brace-pair-desert
 		      (eq cache-pos (car c-state-brace-pair-desert))
+		      (>= from (cdr c-state-brace-pair-desert))
 		      (cdr c-state-brace-pair-desert)))
 		;; CACHE-LIM.  This limit will be necessary when an opening
 		;; paren at `cache-pos' has just had its matching close paren
-		;; inserted.  `cache-pos' continues to be a search bound, even
-		;; though the algorithm below would skip over the new paren
-		;; pair.
+		;; inserted into the buffer.  `cache-pos' continues to be a
+		;; search bound, even though the algorithm below would skip
+		;; over the new paren pair.
 		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
 	    (narrow-to-region
 		(cond
@@ -3354,13 +3356,19 @@
   (fset 'c-real-parse-state (symbol-function 'c-parse-state)))
 (cc-bytecomp-defun c-real-parse-state)
 
+(defvar c-parse-state-point nil)
 (defvar c-parse-state-state nil)
 (make-variable-buffer-local 'c-parse-state-state)
 (defun c-record-parse-state-state ()
+  (setq c-parse-state-point (point))
   (setq c-parse-state-state
 	(mapcar
 	 (lambda (arg)
-	   (cons arg (symbol-value arg)))
+	   (let ((val (symbol-value arg)))
+	     (cons arg
+		   (if (consp val)
+		       (copy-tree val)
+		     val))))
 	 '(c-state-cache
 	   c-state-cache-good-pos
 	   c-state-nonlit-pos-cache
@@ -3373,7 +3381,8 @@
 	   c-state-point-min-lit-start
 	   c-state-min-scan-pos
 	   c-state-old-cpp-beg
-	   c-state-old-cpp-end))))
+	   c-state-old-cpp-end
+	   c-parse-state-point))))
 (defun c-replay-parse-state-state ()
   (message
    (concat "(setq "
@@ -3416,7 +3425,8 @@
       (message "Old state:")
       (c-replay-parse-state-state))
     (c-record-parse-state-state)
-    res1))
+    res2 ; res1 correct a cascading series of errors ASAP
+    ))
 
 (defun c-toggle-parse-state-debug (&optional arg)
   (interactive "P")


> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 2358 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Michael.
>
> On Fri, Sep 21, 2012 at 01:47:15PM -0400, Michael Welsh Duggan wrote:
>> Michael Welsh Duggan <mwd@cert.org> writes:
>
>> > Michael Welsh Duggan <mwd@cert.org> writes:
>
>> >>>> There is generally no way to re-create it, so I've stopped reporting
>> >>>> these.  I wish there were some way to record all actions in c-mode
>> >>>> buffers such that they could be saved and re-played when this type of
>> >>>> problem happens.  If there were some sort of debug flag I could turn
>> >>>> on, I would turn it on by default and hopefully be able to catch some
>> >>>> useful information.
>
>> >>> M-x c-toggle-parse-state-debug
>
>> >>> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
>> >>> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
>> >>> last year to help sort out the bug you reported a little earlier.  :-)
>
>> >>> It works by calculating c-parse-state twice for each call - The first
>> >>> time normally, then again with the internal state bound to "newly
>> >>> initialised".  If the the two results differ, they are printed to
>> >>> *Messages*, together with the saved previous state.  If this does
>> >>> trigger, please note any recent buffer changes.  It may make editing
>> >>> intolerably slow.
>
>> >> I will turn this on.
>
>> I turned this on, and inserted a (ding) where cc-mode outputs an
>> inconsistency message.  I can barely stand to edit code now, due to
>> what seem to be almost every key press causing a beep.  Have you made
>> any progress here, or do you need more data?
>
> I have found the bug which is causing (most of) these dings, though I
> don't think it is the one which caused Kim's original bug.  Could you try
> out the patch below, please.  (I have also enhanced/corrected the
> debugging routines a bit, too.)

Still doesn't seem to help much here.  I have attached a file which
reliably causes a cache failure.  I have attached the smallest file of
the set of files I am working on that causes this particular problem.
Load the attached file and toggle on parse state debugging.  Then scroll
to the bottom of the file.  (Use C-v multiple times, or just M->.)  One
reason I have attached this file is that it only triggers the warning
message once.  Most of my larger files cause this to happen quite a lot.


[-- Attachment #2: mod_fan_in.lzz --]
[-- Type: text/plain, Size: 11027 bytes --]

/*
** Copyright (C) 2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  mod_fan_in.lzz
**
**    Implementation of the "Fan-In" module.
**
**    The Fan-In module copies blocks of data from a user-specified
**    number of inputs to one output.
**
*/

#hdr
#include <silk/silk.h>

RCSIDENTVAR(rcsID_MOD_FAN_OUT_H, "$Id$");

#include "module.hh"
#end

#src
RCSIDENT("$Id$");

#include "modsys.hh"

#define MOD_FAN_IN_CLASS_NAME "Fan-In"
#define DEFAULT_READ_BLOCK_SIZE 10000

#end

namespace silk {

class ModFanIn : public Module
{
    friend int mod_fan_in_so_loader(
        uint16_t        major_version,
        uint16_t        minor_version,
        void           *so_loader_data);

  private:

    static const Module::port_t ports_out[] = {
        {"out",  "Record output"},
        SK_MODULE_PORT_SENTINEL
    };

    std::vector<Module::port_t> ports_in;
    std::vector<std::string> port_names;

    enum params_id_t {
        OPT_INPUT_COUNT
    };

    /** The definition of parameters for the ModFanOut module. */
    struct class_params_t {
        /** name of the parameter */
        const char         *key;
        /** an integer identifier for it  */
        int                 val;
        /** min number of values it supports */
        long                lower;
        /** max number of values it supports */
        long                upper;
        /** description of parameter */
        const char         *description;
    };

    static ModFanIn::class_params_t class_params[] = {
        {"input_count",        OPT_INPUT_COUNT,       1, -1,
         "Number of input connectors to create"},
        {0, 0, 0, 0, 0}         /* sentinel entry */
    };

    ModFanIn()
    {
        conns_out.push_back(NULL);
    }

    ~ModFanIn() {}

  public:

    const char *get_type(void) const
    {
        return MOD_FAN_IN_CLASS_NAME;
    }

  private:

    static Module *_create() {
        return new ModFanIn();
    }

    void _init()
    {
        unsigned int i;

        if (conns_out[0] == NULL) {
            std::ostringstream os;
            os << "The " << get_type() << " module's "
               << ports_out[0].name
               << " connector is not connected";
            throw sk_error_module_bad_param_t(os);
        }
        for (i = 0; i < conns_in.size(); i++) {
            if (conns_in[i]) {
                continue;
            }
            std::ostringstream os;
            os << "The " << get_type() << " module's "
               << ports_in[i].name
               << " connector is not connected";
            throw sk_error_module_bad_param_t(os);
        }
    }

    void _run()
    {
        std::vector<Connector *> ready;
        std::vector<Connector *>::iterator conn;
        Connector *out = conns_out[0];
        void *in_block;
        void *out_block = NULL;
        size_t in_size;
        size_t out_size;
        size_t active_inputs;
        int rv;

        /* create and initialize the connector poll object */
        PollConn poller(PollConn::READ);

        for (conn = conns_in.begin(); conn != conns_in.end(); ++conn) {
            (*conn)->add_to_poll(&poller);
        }

        active_inputs = conns_in.size();

        /* run until no one is giving us data */
        while (active_inputs > 0) {
            /* Which of the upstream connectors have data?  If none have
             * data, wait forever for data to become available. */
            poller.poll(1, ready);

            if (stopped || ready.empty()) {
                goto END;
            }

            in_size = DEFAULT_READ_BLOCK_SIZE;

            /* Process all the connectors that have data */
            for (conn = ready.begin(); conn != ready.end(); ++conn) {
                rv = (*conn)->get_read_block(&in_block, &in_size);
                if (rv) {
                    --active_inputs;
                    (*conn)->remove_from_poll(&poller);
                    continue;
                }

                /* Copy the block to the downstream connector */
                out_size = in_size;
                rv = out->get_write_block(&out_block, &out_size);
                if (rv || stopped) {
                    (*conn)->return_read_block(in_block);
                    goto END;
                }
                if (stopped) {
                    (*conn)->return_read_block(in_block);
                    goto END;
                }
                assert(out_size >= in_size);
                memcpy(out_block, in_block, in_size);

                out->return_write_block(out_block, in_size);
                out_block = NULL;
                (*conn)->return_read_block(in_block);

                if (stopped) {
                    goto END;
                }
            }
        }

      END:
        if (out_block) {
            out->return_read_block(out_block);
        }
        out->stop_no_more_data();

        for (conn = conns_in.begin(); conn != conns_in.end(); ++conn) {
            (*conn)->stop();
            try {
                (*conn)->remove_from_poll(&poller);
            } catch (sk_error_conn_poll_add_remove_t &e) {
                // ignore this exception
            }
        }
    }

    void _get_ports(
        std::vector<port_t> &inputs,
        std::vector<port_t> &outputs)
    {
        const Module::port_t *port;

        inputs = ports_in;
        outputs.clear();
        for (port = ports_out; port->name; ++port) {
            outputs.push_back(*port);
        }
    }

    static Parameterized::param_registry_t *create_param_registry() {
        param_registry_t *registry = new param_registry_t();
        class_params_t *param = class_params;

        while (param->key) {
            registry->insert(
                Definition(param->key, param->description,
                           &param->val, param->lower, param->upper));
            ++param;
        }
        return registry;
    }

    static std::auto_ptr<Parameterized::param_registry_t> param_registry =
        std::auto_ptr<Parameterized::param_registry_t>(
            ModFanIn::create_param_registry());

    const Parameterized::param_registry_t& get_param_registry() const
    {
        return *param_registry;
    }

    void _set_param(
        const Definition& defn,
        const std::vector<std::string>& values) throw (Error)
    {
        int rv;
        uint32_t u32;
        unsigned int i;

        switch (*(params_id_t *)(defn.get_context())) {
          case OPT_INPUT_COUNT:
            if (!conns_in.empty()) {
                std::ostringstream os;
                os << "Invalid " << defn.get_name()
                   << " '" << values[0]
                   << "': Cannot change value once connectors are created";
                throw sk_error_module_bad_param_t(os);
            }
            rv = skStringParseUint32(&u32, values[0].c_str(), 1, 0);
            if (rv) {
                goto PARSE_ERROR;
            }
            for (i = 0; i < u32; i++) {
                Module::port_t port = {NULL, NULL};
                char name[32];
                snprintf(name, sizeof(name), "in.%u", i);
                port_names.push_back(name);
                port.name = port_names.back().c_str();
                ports_in.push_back(port);
                conns_in.push_back(NULL);
            }
            break;
        }

        return;

      PARSE_ERROR:
        std::ostringstream os;
        os << "Invalid " << defn.get_name() << " '" << values[0] << "': "
           << skStringParseStrerror(rv);
        throw sk_error_conn_param_value_t(os.str());
    }
};

int mod_fan_in_so_loader(
    uint16_t        major_version,
    uint16_t        minor_version,
    void           *so_loader_data)
{
    ModuleSystem::loader_data_t *so_loader =
        static_cast<ModuleSystem::loader_data_t *>(so_loader_data);
    ModuleSystem *module_system = so_loader->modsys;

    // ADD VERSION CHECK

    try {
        module_system->register_module(
            MOD_FAN_IN_CLASS_NAME, &ModFanIn::_create);
    }
    catch (Error &e) {
        skAppPrintErr("Failed to load " MOD_FAN_IN_CLASS_NAME ": %s",
                      e.get_string().c_str());
        return -1;
    }
    catch (...) {
        skAppPrintErr("Failed to load " MOD_FAN_IN_CLASS_NAME ": %s",
                      "Unknown error");
        return -1;
    }

    return 0;
}


}

/*
** Local Variables:
** mode:c++
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/


[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael,

On Tue, Oct 09, 2012 at 10:05:07AM -0400, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

[ .... ]

> >> >>> M-x c-toggle-parse-state-debug

> >> >>> It works by calculating c-parse-state twice for each call - The first
> >> >>> time normally, then again with the internal state bound to "newly
> >> >>> initialised".  If the the two results differ, they are printed to
> >> >>> *Messages*, together with the saved previous state.  If this does
> >> >>> trigger, please note any recent buffer changes.  It may make editing
> >> >>> intolerably slow.

> >> >> I will turn this on.

> >> I turned this on, and inserted a (ding) where cc-mode outputs an
> >> inconsistency message.  I can barely stand to edit code now, due to
> >> what seem to be almost every key press causing a beep.  Have you made
> >> any progress here, or do you need more data?

> > I have found the bug which is causing (most of) these dings, though I
> > don't think it is the one which caused Kim's original bug.  Could you try
> > out the patch below, please.  (I have also enhanced/corrected the
> > debugging routines a bit, too.)

> Still doesn't seem to help much here.  I have attached a file which
> reliably causes a cache failure.  I have attached the smallest file of
> the set of files I am working on that causes this particular problem.
> Load the attached file and toggle on parse state debugging.  Then scroll
> to the bottom of the file.  (Use C-v multiple times, or just M->.)  One
> reason I have attached this file is that it only triggers the warning
> message once.  Most of my larger files cause this to happen quite a lot.

What is happening in this file is another bug, arising from historical
assumptions which are no longer valid.

The "from scratch" calculation notes that the starting scanning position
would be a long way (>5000) back, hence it tries going back to the second
"beginning-of-defun" to get a top-level starting point.  This
"beginning-of-defun" is a pure "brace in column zero" test.

This doesn't work in C++ when constructs inside a namespace have braces
at column zero, something I believe has become very common in recent
years.  Namespaces didn't exist in C++ when c-parse-state was originally
written.

Obviously this optimisation is no longer valid.  I wouldn't be surprised
if it has caused quite a bit of buggy behaviour.  I'll need to think it
over for a few days to decide what to do.

Again, thanks for taking so much trouble in submitting these bug
reports.


> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael,

On Wed, Oct 10, 2012 at 08:00:25PM +0000, Alan Mackenzie wrote:
> On Tue, Oct 09, 2012 at 10:05:07AM -0400, Michael Welsh Duggan wrote:
> > Alan Mackenzie <acm@muc.de> writes:

[ .... ]

> > > I have found the bug which is causing (most of) these dings, though I
> > > don't think it is the one which caused Kim's original bug.  Could you try
> > > out the patch below, please.  (I have also enhanced/corrected the
> > > debugging routines a bit, too.)

> > Still doesn't seem to help much here.  I have attached a file which
> > reliably causes a cache failure.  I have attached the smallest file of
> > the set of files I am working on that causes this particular problem.
> > Load the attached file and toggle on parse state debugging.  Then scroll
> > to the bottom of the file.  (Use C-v multiple times, or just M->.)  One
> > reason I have attached this file is that it only triggers the warning
> > message once.  Most of my larger files cause this to happen quite a lot.

> What is happening in this file is another bug, arising from historical
> assumptions which are no longer valid.

> The "from scratch" calculation notes that the starting scanning position
> would be a long way (>5000) back, hence it tries going back to the second
> "beginning-of-defun" to get a top-level starting point.  This
> "beginning-of-defun" is a pure "brace in column zero" test.

> This doesn't work in C++ when constructs inside a namespace have braces
> at column zero, something I believe has become very common in recent
> years.  Namespaces didn't exist in C++ when c-parse-state was originally
> written.

> Obviously this optimisation is no longer valid.  I wouldn't be surprised
> if it has caused quite a bit of buggy behaviour.  I'll need to think it
> over for a few days to decide what to do.

The only reasonable thing to do is to disable the heuristic for C++ Mode.
This is what the patch below now does.  Could you try it out as usual,
please.  Thanks!




diff -r ac6584d14c06 cc-engine.el
--- a/cc-engine.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-engine.el	Sun Oct 14 17:02:08 2012 +0000
@@ -2560,8 +2560,11 @@
 	      start-point cache-pos)))
 
     ;; Might we be better off starting from the top level, two defuns back,
-    ;; instead?
-    (when (> how-far c-state-cache-too-far)
+    ;; instead?  This heuristic no longer works well in C++, where
+    ;; declarations inside namespace brace blocks are frequently placed at
+    ;; column zero.
+    (when (and (not (c-major-mode-is 'c++-mode))
+	       (> how-far c-state-cache-too-far))
       (setq BOD-pos (c-get-fallback-scan-pos here)) ; somewhat EXPENSIVE!!!
       (if (< (- here BOD-pos) how-far)
 	  (setq strategy 'BOD
@@ -2648,17 +2651,19 @@
 	;; If we're essentially repeating a fruitless search, just give up.
 	(unless (and c-state-brace-pair-desert
 		     (eq cache-pos (car c-state-brace-pair-desert))
+		     (> from (car c-state-brace-pair-desert))
 		     (<= from (cdr c-state-brace-pair-desert)))
-	  ;; DESERT-LIM.  Only search what we absolutely need to,
+	  ;; DESERT-LIM.  Avoid repeated searching through the cached desert.
 	  (let ((desert-lim
 		 (and c-state-brace-pair-desert
 		      (eq cache-pos (car c-state-brace-pair-desert))
+		      (>= from (cdr c-state-brace-pair-desert))
 		      (cdr c-state-brace-pair-desert)))
 		;; CACHE-LIM.  This limit will be necessary when an opening
 		;; paren at `cache-pos' has just had its matching close paren
-		;; inserted.  `cache-pos' continues to be a search bound, even
-		;; though the algorithm below would skip over the new paren
-		;; pair.
+		;; inserted into the buffer.  `cache-pos' continues to be a
+		;; search bound, even though the algorithm below would skip
+		;; over the new paren pair.
 		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
 	    (narrow-to-region
 		(cond
@@ -3354,13 +3359,19 @@
   (fset 'c-real-parse-state (symbol-function 'c-parse-state)))
 (cc-bytecomp-defun c-real-parse-state)
 
+(defvar c-parse-state-point nil)
 (defvar c-parse-state-state nil)
 (make-variable-buffer-local 'c-parse-state-state)
 (defun c-record-parse-state-state ()
+  (setq c-parse-state-point (point))
   (setq c-parse-state-state
 	(mapcar
 	 (lambda (arg)
-	   (cons arg (symbol-value arg)))
+	   (let ((val (symbol-value arg)))
+	     (cons arg
+		   (if (consp val)
+		       (copy-tree val)
+		     val))))
 	 '(c-state-cache
 	   c-state-cache-good-pos
 	   c-state-nonlit-pos-cache
@@ -3373,7 +3384,8 @@
 	   c-state-point-min-lit-start
 	   c-state-min-scan-pos
 	   c-state-old-cpp-beg
-	   c-state-old-cpp-end))))
+	   c-state-old-cpp-end
+	   c-parse-state-point))))
 (defun c-replay-parse-state-state ()
   (message
    (concat "(setq "
@@ -3416,7 +3428,8 @@
       (message "Old state:")
       (c-replay-parse-state-state))
     (c-record-parse-state-state)
-    res1))
+    res2 ; res1 correct a cascading series of errors ASAP
+    ))
 
 (defun c-toggle-parse-state-debug (&optional arg)
   (interactive "P")


> > -- 
> > Michael Welsh Duggan
> > (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Michael,
>
> On Wed, Oct 10, 2012 at 08:00:25PM +0000, Alan Mackenzie wrote:
>> On Tue, Oct 09, 2012 at 10:05:07AM -0400, Michael Welsh Duggan wrote:
>> > Alan Mackenzie <acm@muc.de> writes:
>
> [ .... ]
>
>> > > I have found the bug which is causing (most of) these dings, though I
>> > > don't think it is the one which caused Kim's original bug.  Could you try
>> > > out the patch below, please.  (I have also enhanced/corrected the
>> > > debugging routines a bit, too.)
>
>> > Still doesn't seem to help much here.  I have attached a file which
>> > reliably causes a cache failure.  I have attached the smallest file of
>> > the set of files I am working on that causes this particular problem.
>> > Load the attached file and toggle on parse state debugging.  Then scroll
>> > to the bottom of the file.  (Use C-v multiple times, or just M->.)  One
>> > reason I have attached this file is that it only triggers the warning
>> > message once.  Most of my larger files cause this to happen quite a lot.
>
>> What is happening in this file is another bug, arising from historical
>> assumptions which are no longer valid.
>
>> The "from scratch" calculation notes that the starting scanning position
>> would be a long way (>5000) back, hence it tries going back to the second
>> "beginning-of-defun" to get a top-level starting point.  This
>> "beginning-of-defun" is a pure "brace in column zero" test.
>
>> This doesn't work in C++ when constructs inside a namespace have braces
>> at column zero, something I believe has become very common in recent
>> years.  Namespaces didn't exist in C++ when c-parse-state was originally
>> written.
>
>> Obviously this optimisation is no longer valid.  I wouldn't be surprised
>> if it has caused quite a bit of buggy behaviour.  I'll need to think it
>> over for a few days to decide what to do.
>
> The only reasonable thing to do is to disable the heuristic for C++ Mode.
> This is what the patch below now does.  Could you try it out as usual,
> please.  Thanks!

Better, but doesn't solve all problems.  I'll see if I can't find
something reproducible for you.

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

A new recipe.  This is a strange one.  I hope it works for you as well.

emacs -Q
M-x tool-bar-mode
resize the frame to be height 56 (according to the window manager)
Load the included file
M-x c-toggle-parse-state-debug
C-s FIXME C-s
Move cursor to the "o" on the next line (1175).
Type: 
PyTYPE( 
C-f 
)

I get errors at the first open paren, and the close paren.  Resizing the
fame is important!  I think it's due to caching that happens during font
locking. 


[-- Attachment #2: pysilk.c --]
[-- Type: text/plain, Size: 236217 bytes --]

/*
** Copyright (C) 2007-2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  Python wrappers
**
*/


#include <Python.h>             /* Must be included before any system
                                   headers */
#include <silk/silk.h>

RCSIDENT("$Id$");

#include <silk/rwrec.h>
#include <silk/skbag.h>
#include <silk/skcountry.h>
#include <silk/skipaddr.h>
#include <silk/skipset.h>
#include <silk/skprefixmap.h>
#include <silk/sksite.h>
#include <silk/skstream.h>
#include <silk/skvector.h>
#include <silk/utils.h>
#include <datetime.h>
#include <structmember.h>
#include "pysilk_common.h"


/* LOCAL DEFINES AND TYPEDEFS */

#if PY_MAJOR_VERSION >= 3
#  ifndef PyMODINIT_FUNC        /* declarations for DLL import/export */
#    define PyMODINIT_FUNC PyObject *
#  endif
#  define INIT_RETURN(x) return x
#  define XMOD_INIT(x) PyInit_ ## x
#else
#  ifndef PyMODINIT_FUNC        /* declarations for DLL import/export */
#    define PyMODINIT_FUNC void
#  endif
#  define INIT_RETURN(x) x
#  define XMOD_INIT(x) init ## x
#endif
#define MOD_INIT(x) XMOD_INIT(x) /* Avoid argument prescan problem by
                                    indirection */

/* get an object's type; added in Python 2.6 */
#ifndef Py_TYPE
#define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)
#endif


#define XSTR(s) #s
#define STR(s) XSTR(s)

#define PYSILK_NAME     pysilk
#define PYSILK_NL_NAME  pysilk_nl
#define PYSILK_INIT     MOD_INIT(PYSILK_NAME)
#define PYSILK_NL_INIT  MOD_INIT(PYSILK_NL_NAME)

/* In Python 2.4, length()-type functions are 'inquiry's that return
 * int.  In Python 2.5, they are 'lenfunc's that return Py_ssize_t. */
#if PY_VERSION_HEX < 0x02050000
#define Py_ssize_t     int
#define PY_SSIZE_T_MAX INT_MAX
#endif

#define NOT_SET -9999

#if SK_ENABLE_IPV6
#define UNUSED_NOv6(x) x
#else
#define UNUSED_NOv6(x) UNUSED(x)
#endif


#define CHECK_SITE(err)                         \
    do {                                        \
        if (init_site(NULL)) {                  \
            return err;                         \
        }                                       \
    } while(0)


/*
 *  ASSERT_RESULT(ar_func_args, ar_type, ar_expected);
 *
 *    ar_func_args  -- is a function and any arguments it requires
 *    ar_type       -- is the type that ar_func_args returns
 *    ar_expected   -- is the expected return value from ar_func_args
 *
 *    If assert() is disabled, simply call 'ar_func_args'.
 *
 *    If assert() is enabled, call 'ar_func_args', capture its result,
 *    and assert() that its result is 'ar_expected'.
 */
#ifdef  NDEBUG
/* asserts are disabled; just call the function */
#define ASSERT_RESULT(ar_func_args, ar_type, ar_expected)  ar_func_args
#else
#define ASSERT_RESULT(ar_func_args, ar_type, ar_expected)       \
    do {                                                        \
        ar_type ar_rv = (ar_func_args);                         \
        assert(ar_rv == (ar_expected));                         \
    } while(0)
#endif


typedef skBagErr_t (*silkBagModFn)(
    skBag_t *,
    const skBagTypedKey_t *,
    const skBagTypedCounter_t *,
    skBagTypedCounter_t *);

typedef struct silkpy_globals_st {
    PyObject *silkmod;
    PyObject *timedelta;
    PyObject *datetime;
    PyObject *maxelapsed;
    PyObject *minelapsed;
    PyObject *zerotime;
    PyObject *thousand;
    PyObject *havesite;
    PyObject *sensors;
    PyObject *classes;
    PyObject *flowtypes;
    PyObject *newrawrec;
    PyObject *maxintipv4;
#if SK_ENABLE_IPV6
    PyObject *maxintipv6;
#endif
    int site_configured;
} silkpy_globals_t;


/* LOCAL VARIABLE DEFINITIONS */

static  char error_buffer[1024];

#if PY_MAJOR_VERSION >= 3
/* Global state for 3.0 Python is found by searching for this
 * module */
static struct PyModuleDef *pysilk_module = NULL;

#define GLOBALS                                                         \
    ((silkpy_globals_t*)PyModule_GetState(PyState_FindModule(pysilk_module)))

#else

/* Global state for Pre-3.0 Python */
static silkpy_globals_t silkpy_globals_static;

#define GLOBALS   (&silkpy_globals_static)

#endif  /* #else of #if PY_MAJOR_VERSION >= 3 */


/* EXPORTED FUNCTION PROTOTYPES */

PyMODINIT_FUNC PYSILK_INIT(void);
PyMODINIT_FUNC PYSILK_NL_INIT(void);


/* LOCAL FUNCTION PROTOTYPES */

/* basic support functions */
static PyObject *any_obj_error(
    PyObject   *exc,
    const char *format,
    PyObject   *obj);
#ifdef TEST_PRINTF_FORMATS
#define error_printf  printf
#else  /* TEST_PRINTF_FORMATS */
static int error_printf(
    const char *fmt,
    ...)
    SK_CHECK_PRINTF(1, 2);
#endif
static int init_classes(void);
static int init_flowtypes(void);
static int init_sensors(void);
static int init_silkfile_module(
    PyObject   *mod);
static int init_site(
    const char *site_file);
static PyObject *initpysilkbase(
    char*   name);
static PyObject *iter_iter(
    PyObject   *self);
static void obj_dealloc(
    PyObject   *obj);
static PyObject *obj_error(
    const char *format,
    PyObject   *obj);
static PyObject *reduce_error(
    PyObject   *self);
static int silkPyDatetimeToSktime(
    sktime_t   *silktime,
    PyObject   *datetime);
#if !SK_ENABLE_IPV6
static PyObject *silkPyNotImplemented(
    PyObject    UNUSED(*self),
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds));
#endif


/* FUNCTION DEFINITIONS */


/*
 *************************************************************************
 *   IPAddr
 *************************************************************************
 */

typedef struct silkPyIPAddr_st {
    PyObject_HEAD
    skipaddr_t addr;
} silkPyIPAddr;

/* function prototypes */
static PyObject *silkPyIPAddr_country_code(
    silkPyIPAddr   *self);
static long silkPyIPAddr_hash(
    silkPyIPAddr   *obj);
static PyObject *silkPyIPAddr_int(
    silkPyIPAddr   *obj);
static PyObject *silkPyIPAddr_is_ipv6(
    silkPyIPAddr   UNUSED_NOv6(*self));
static PyObject *silkPyIPAddr_isipv6_deprecated(
    silkPyIPAddr   *self);
static PyObject *silkPyIPAddr_mask(
    silkPyIPAddr   *self,
    PyObject       *mask);
static PyObject *silkPyIPAddr_mask_prefix(
    silkPyIPAddr   *self,
    PyObject       *prefix);
static PyObject *silkPyIPAddr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyIPAddr_octets(
    silkPyIPAddr   *self);
static PyObject *silkPyIPAddr_padded(
    silkPyIPAddr   *obj);
static PyObject *silkPyIPAddr_repr(
    silkPyIPAddr   *obj);
static PyObject *silkPyIPAddr_richcompare(
    silkPyIPAddr   *self,
    PyObject       *obj,
    int             cmp);
static int silkPyIPAddr_setup(
    PyObject   *mod);
static PyObject *silkPyIPAddr_str(
    silkPyIPAddr   *obj);
static PyObject *silkPyIPAddr_to_ipv4(
    PyObject   *self);
#if SK_ENABLE_IPV6
static PyObject *silkPyIPAddr_to_ipv6(
    PyObject   *self);
#endif
static int silkPyIPv4Addr_init(
    silkPyIPAddr   *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyIPv4Addr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds);
static int silkPyIPv6Addr_init(
    silkPyIPAddr   *self,
    PyObject       *args,
    PyObject       *kwds);
#if SK_ENABLE_IPV6
static PyObject *silkPyIPv6Addr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds);
#endif
static PyObject *silkPyIPvXAddr_new(
    PyTypeObject   *basetype,
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds);

/* define docs and methods */
static char silkPyIPAddr_doc[] =
    ("IPAddr(string) -> ip address\n"
     "IPAddr(ipaddr) -> copy of ip address");

static char silkPyIPv4Addr_doc[] =
    ("IPv4Addr(string) -> IPv4 address\n"
     "IPv4Addr(int) -> IPv4 address\n"
     "IPv4Addr(IPV6Addr) -> IPv4 from IPv4 in IPv6 address\n"
     "IPv4Addr(IPv4Addr) -> copy of ip address");

static char silkPyIPv6Addr_doc[] =
    ("IPv6Addr(string) -> IPv6 address\n"
     "IPv6Addr(int) -> IPv6 address\n"
     "IPv6Addr(IPV4Addr) -> IPv4 in IPv6 address\n"
     "IPv6Addr(IPv6Addr) -> copy of ip address");

static PyNumberMethods silkPyIPAddr_number_methods;

static PyMethodDef silkPyIPAddr_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"isipv6", (PyCFunction)silkPyIPAddr_isipv6_deprecated, METH_NOARGS,
     "addr.isipv6() -> bool -- return whether addr is an IPv6 address"},
    {"is_ipv6", (PyCFunction)silkPyIPAddr_is_ipv6, METH_NOARGS,
     "addr.isipv6() -> bool -- return whether addr is an IPv6 address"},
    {"to_ipv6", (PyCFunction)
#if SK_ENABLE_IPV6
     silkPyIPAddr_to_ipv6, METH_NOARGS,
#else
     silkPyNotImplemented, METH_VARARGS | METH_KEYWORDS,
#endif
     "addr.to_ipv6() -- return addr converted to an IPv6 address"},
    {"to_ipv4", (PyCFunction)silkPyIPAddr_to_ipv4, METH_NOARGS,
     "addr.to_ipv4() -- return addr converted to an IPv4 address"},
    {"padded", (PyCFunction)silkPyIPAddr_padded, METH_NOARGS,
     "addr.padded() -> str -- return zero-padded IP string"},
    {"mask", (PyCFunction)silkPyIPAddr_mask, METH_O,
     "addr.mask(addr2) -> addr3 -- return addr masked by addr2"},
    {"mask_prefix", (PyCFunction)silkPyIPAddr_mask_prefix, METH_O,
     ("addr.mask(prefix) -> addr2 -- "
      "return addr masked by the top prefix bits")},
    {"country_code", (PyCFunction)silkPyIPAddr_country_code, METH_NOARGS,
     "addr.country_code() -> string -- 2-character country code"},
    {"octets", (PyCFunction)silkPyIPAddr_octets, METH_NOARGS,
     ("addr.octets() = (o1, o2 ...) -- "
      "return the octets of addr as a tuple")},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

/* define the object types */
static PyTypeObject silkPyIPAddrType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.IPAddr",              /* tp_name */
    sizeof(silkPyIPAddr),       /* tp_basicsize */
    0,                          /* tp_itemsize */
    obj_dealloc,                /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    (reprfunc)silkPyIPAddr_repr, /* tp_repr */
    &silkPyIPAddr_number_methods, /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    (hashfunc)silkPyIPAddr_hash, /* tp_hash  */
    0,                          /* tp_call */
    (reprfunc)silkPyIPAddr_str, /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
#if PY_MAJOR_VERSION < 3
    Py_TPFLAGS_HAVE_RICHCOMPARE |
#endif
    Py_TPFLAGS_DEFAULT,         /* tp_flags */
    silkPyIPAddr_doc,           /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    (richcmpfunc)silkPyIPAddr_richcompare, /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    silkPyIPAddr_methods,       /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    silkPyIPAddr_new,           /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyIPv4AddrType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.IPv4Addr",            /* tp_name */
    0,                          /* tp_basicsize */
    0,                          /* tp_itemsize */
    0,                          /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyIPv4Addr_doc,         /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    &silkPyIPAddrType,          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyIPv4Addr_init, /* tp_init */
    0,                          /* tp_alloc */
    silkPyIPv4Addr_new,         /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyIPv6AddrType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.IPv6Addr",            /* tp_name */
    0,                          /* tp_basicsize */
    0,                          /* tp_itemsize */
    0,                          /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyIPv6Addr_doc,         /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    &silkPyIPAddrType,          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyIPv6Addr_init, /* tp_init */
    0,                          /* tp_alloc */
#if SK_ENABLE_IPV6
    silkPyIPv6Addr_new,         /* tp_new */
#else
    (newfunc)silkPyNotImplemented, /* tp_new */
#endif
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyIPAddr_Check(op)                  \
    PyObject_TypeCheck(op, &silkPyIPAddrType)
#define silkPyIPv4Addr_Check(op)                \
    PyObject_TypeCheck(op, &silkPyIPv4AddrType)
#define silkPyIPv6Addr_Check(op)                \
    PyObject_TypeCheck(op, &silkPyIPv6AddrType)

static PyObject *silkPyIPAddr_country_code(
    silkPyIPAddr   *self)
{
    char name[3];
    sk_countrycode_t code;
    int rv;

    rv = skCountrySetup(NULL, error_printf);
    if (rv != 0) {
        PyErr_SetString(PyExc_RuntimeError, error_buffer);
        return NULL;
    }

    code = skCountryLookupCode(&self->addr);
    if (code == SK_COUNTRYCODE_INVALID) {
        Py_RETURN_NONE;
    }

    return PyUnicode_FromString(skCountryCodeToName(code, name, sizeof(name)));
}

static long silkPyIPAddr_hash(
    silkPyIPAddr   *obj)
{
    long retval;
#if SK_ENABLE_IPV6
    if (skipaddrIsV6(&obj->addr)) {
        uint8_t v6[16];
        skipaddrGetAsV6(&obj->addr, v6);
#if SK_SIZEOF_LONG == 8
        retval = *(long*)(&v6[8]);
#else  /* SK_SIZEOF_LONG */
        /* Assume long is 4 bytes */
        retval = *(long*)(&v6[12]);
#endif  /* SK_SIZEOF_LONG */
        return retval;
    }
#endif  /* SK_ENABLE_IPV6 */
    retval = skipaddrGetV4(&obj->addr);
    if (retval == -1) {
        retval = 0;
    }
    return retval;
}

static PyObject *silkPyIPAddr_int(
    silkPyIPAddr   *obj)
{
    PyObject *result;

#if SK_ENABLE_IPV6
    if (skipaddrIsV6(&obj->addr)) {
        char      buf[33];
        char     *p;
        int       i;
        uint32_t *v6;

        p = buf;
        v6 = (uint32_t*)obj->addr.ip_ip.ipu_ipv6;
        for (i = 0; i < 4; i++) {
            sprintf(p, "%08" PRIx32, ntohl(*v6));
            p += 8;
            ++v6;
        }
        result = PyLong_FromString(buf, NULL, 16);
    } else
#endif  /* SK_ENABLE_IPV6 */
    {
        result = PyLong_FromUnsignedLong(skipaddrGetV4(&obj->addr));
    }

    return result;
}

static PyObject *silkPyIPAddr_is_ipv6(
    silkPyIPAddr   UNUSED_NOv6(*self))
{
#if SK_ENABLE_IPV6
    return PyBool_FromLong(skipaddrIsV6(&self->addr));
#else
    Py_RETURN_FALSE;
#endif
}

static PyObject *silkPyIPAddr_isipv6_deprecated(
    silkPyIPAddr   *self)
{
    PyErr_Warn(PyExc_DeprecationWarning,
               ("IPAddr.isipv6() is deprecated.  "
                "Use IPAddr.is_ipv6() instead."));
    return silkPyIPAddr_is_ipv6(self);
}

static PyObject *silkPyIPAddr_mask(
    silkPyIPAddr   *self,
    PyObject       *mask)
{
    silkPyIPAddr *retval;
    skipaddr_t addr;
    PyTypeObject *type;

    if (!silkPyIPAddr_Check(mask)) {
        PyErr_SetString(PyExc_TypeError, "Argument must be an IPAddr");
        return NULL;
    }

    skipaddrCopy(&addr, &self->addr);
    skipaddrMask(&addr, &((silkPyIPAddr*)mask)->addr);
    if (skipaddrIsV6(&addr)) {
        type = &silkPyIPv6AddrType;
    } else {
        type = &silkPyIPv4AddrType;
    }
    retval = PyObject_New(silkPyIPAddr, type);
    if (retval == NULL) {
        return NULL;
    }
    skipaddrCopy(&retval->addr, &addr);

    return (PyObject*)retval;
}

static PyObject *silkPyIPAddr_mask_prefix(
    silkPyIPAddr   *self,
    PyObject       *prefix)
{
    silkPyIPAddr *retval;
    PyTypeObject *type;
    long p;
    int max;

    if (!IS_INT(prefix)) {
        PyErr_SetString(PyExc_TypeError, "Prefix must be an integer");
        return NULL;
    }

    if (skipaddrIsV6(&self->addr)) {
        type = &silkPyIPv6AddrType;
        max = 128;
    } else {
        type = &silkPyIPv4AddrType;
        max = 32;
    }

    p = PyInt_AsLong(prefix);
    if (PyErr_Occurred()) {
        return NULL;
    }

    if (p < 0 || p > max) {
        return PyErr_Format(PyExc_ValueError,
                            "Prefix must be between 0 and %d", max);
    }

    if (skipaddrIsV6(&self->addr)) {
        type = &silkPyIPv6AddrType;
    } else {
        type = &silkPyIPv4AddrType;
    }
    retval = PyObject_New(silkPyIPAddr, type);
    if (retval == NULL) {
        return NULL;
    }

    skipaddrCopy(&retval->addr, &self->addr);
    skipaddrApplyCIDR(&retval->addr, p);

    return (PyObject*)retval;
}

static PyObject *silkPyIPAddr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds)
{
    static char  *kwlist[] = {"address", NULL};
    silkPyIPAddr *self;
    PyObject     *o;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O", kwlist, &o)) {
        return NULL;
    }

    /* FIXME: Is this valid in python 3?  Change to Py_TYPE() instead? */
    if (o->ob_type == &silkPyIPv4AddrType ||
        o->ob_type == &silkPyIPv6AddrType)
    {
        /* IPv{4,6}Addr objects are immutable, so just incref*/
        Py_INCREF(o);
        return o;
    }
    if (silkPyIPAddr_Check(o)) {
        /* Unknown subclass of IPAddr, do a real copy */
        if (type == &silkPyIPAddrType) {
            if (skipaddrIsV6(&((silkPyIPAddr*)o)->addr)) {
                type = &silkPyIPv6AddrType;
            } else {
                type = &silkPyIPv4AddrType;
            }
        }
    } else if (IS_STRING(o)) {
        PyObject *bytes = bytes_from_string(o);
        const char *straddr;
        const char *c;

        if (bytes == NULL) {
            return NULL;
        }

        straddr = PyBytes_AS_STRING(bytes);
        c = strchr(straddr, ':');
        if (c) {
            type = &silkPyIPv6AddrType;
        } else {
            type = &silkPyIPv4AddrType;
        }
        Py_DECREF(bytes);
    } else if (IS_INT(o)) {
        /* The IPAddr(int) constructor is deprecated */
        int rv = PyErr_Warn(PyExc_DeprecationWarning,
                            ("IPAddr(int) is deprecated.  Use IPv4Addr(int) "
                             "or IPv6Addr(int) instead."));
        if (rv) {
            return NULL;
        }
        type = &silkPyIPv4AddrType;

    } else {
        return PyErr_Format(PyExc_TypeError, "Must be a string or IPAddr");
    }

    /* Allocate the object */
    self = (silkPyIPAddr*)type->tp_alloc(type, 0);
    if (self == NULL) {
        return NULL;
    }

    return (PyObject*)self;
}

static PyObject *silkPyIPAddr_octets(
    silkPyIPAddr   *self)
{
    PyObject *retval;
    PyObject *octet;
    int i;

#if SK_ENABLE_IPV6
    if (skipaddrIsV6(&self->addr)) {
        uint8_t v6[16];
        retval = PyTuple_New(16);
        if (retval == NULL) {
            return NULL;
        }
        skipaddrGetV6(&self->addr, v6);

        for (i = 0; i < 16; i++) {
            octet = PyInt_FromLong(v6[i]);
            if (octet == NULL) {
                Py_DECREF(retval);
                return NULL;
            }
            PyTuple_SET_ITEM(retval, i, octet);
        }
    } else
#endif
    {
        uint32_t v4 = skipaddrGetV4(&self->addr);
        retval = PyTuple_New(4);
        if (retval == NULL) {
            return NULL;
        }

        for (i = 3; i >= 0; i--) {
            octet = PyInt_FromLong(v4 & 0xff);
            if (octet == NULL) {
                Py_DECREF(retval);
                return NULL;
            }
            PyTuple_SET_ITEM(retval, i, octet);
            v4 >>= 8;
        }
    }

    return retval;
}

static PyObject *silkPyIPAddr_padded(
    silkPyIPAddr   *obj)
{
    char buf[SK_NUM2DOT_STRLEN];

    skipaddrString(buf, &obj->addr, SKIPADDR_ZEROPAD);
    return PyUnicode_FromString(buf);
}

static PyObject *silkPyIPAddr_repr(
    silkPyIPAddr   *obj)
{
    char buf[SK_NUM2DOT_STRLEN];
    PyTypeObject *type;
    PyObject *retval;

    type = (PyTypeObject*)PyObject_Type((PyObject*)obj);
    if (type == NULL) {
        return NULL;
    }

    skipaddrString(buf, &obj->addr, SKIPADDR_CANONICAL);

    retval = PyUnicode_FromFormat("%s('%s')", type->tp_name, buf);
    Py_DECREF(type);

    return retval;
}

static PyObject *silkPyIPAddr_richcompare(
    silkPyIPAddr   *self,
    PyObject       *obj,
    int             cmp)
{
    int rv;
    silkPyIPAddr *other;

    if (!silkPyIPAddr_Check(obj)) {
        PyErr_SetString(PyExc_TypeError, "Expected silk.IPAddr");
        return NULL;
    }
    other = (silkPyIPAddr*)obj;
    rv = skipaddrCompare(&self->addr, &other->addr);
    if (rv < 0) {
        return PyBool_FromLong(cmp == Py_LT || cmp == Py_LE || cmp == Py_NE);
    }
    if (rv > 0) {
        return PyBool_FromLong(cmp == Py_GT || cmp == Py_GE || cmp == Py_NE);
    }
    return PyBool_FromLong(cmp == Py_EQ || cmp == Py_LE || cmp == Py_GE);
}

static int silkPyIPAddr_setup(
    PyObject   *mod)
{
    /* Setup number methods */
    memset(&silkPyIPAddr_number_methods, 0,
           sizeof(silkPyIPAddr_number_methods));
    silkPyIPAddr_number_methods.nb_int = (unaryfunc)silkPyIPAddr_int;
#if PY_MAJOR_VERSION < 3
    silkPyIPAddr_number_methods.nb_long = (unaryfunc)silkPyIPAddr_int;
#endif

    /* Initialize type and add to module */
    if (PyType_Ready(&silkPyIPAddrType) < 0) {
        return -1;
    }
    return PyModule_AddObject(mod, "IPAddr", (PyObject*)&silkPyIPAddrType);
}

static PyObject *silkPyIPAddr_str(
    silkPyIPAddr   *obj)
{
    char buf[SK_NUM2DOT_STRLEN];

    skipaddrString(buf, &obj->addr, SKIPADDR_CANONICAL);
    return PyUnicode_FromString(buf);
}

static PyObject *silkPyIPAddr_to_ipv4(
    PyObject   *self)
{
#if SK_ENABLE_IPV6
    PyObject *obj;

    obj = PyObject_CallFunctionObjArgs((PyObject*)&silkPyIPv4AddrType,
                                       self, NULL);
    if (obj == NULL) {
        if (PyErr_ExceptionMatches(PyExc_ValueError)) {
            PyErr_Clear();
            Py_RETURN_NONE;
        }
    }

    return obj;
#else  /* SK_ENABLE_IPV6 */
    Py_INCREF(self);
    return self;
#endif  /* SK_ENABLE_IPV6 */
}

#if SK_ENABLE_IPV6
static PyObject *silkPyIPAddr_to_ipv6(
    PyObject   *self)
{
    return PyObject_CallFunctionObjArgs((PyObject*)&silkPyIPv6AddrType,
                                        self, NULL);
}
#endif  /* SK_ENABLE_IPV6 */

static int silkPyIPv4Addr_init(
    silkPyIPAddr   *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char  *kwlist[] = {"address", NULL};
    PyObject     *addr;
    int           rv;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O", kwlist, &addr)) {
        return -1;
    }

    if (addr == (PyObject*)self) {
        /* We were initialized by new */
        return 0;
    }

    if (IS_STRING(addr)) {
        PyObject *bytes;
        bytes = bytes_from_string(addr);
        if (bytes == NULL) {
            return -1;
        }
        rv = skStringParseIP(&self->addr, PyBytes_AS_STRING(bytes));
        Py_DECREF(bytes);
        if (rv != 0) {
            PyErr_SetString(PyExc_ValueError,
                            "String is not a valid IP address");
            return -1;
        }
        if (skipaddrIsV6(&self->addr)) {
            PyErr_SetString(PyExc_ValueError,
                            "String is not a valid IPv4 address");
            return -1;
        }
    } else if (IS_INT(addr)) {
        uint32_t value;
        PyObject *num;

        num = PyLong_FromLong(0);
        rv = PyObject_RichCompareBool(addr, num, Py_LT);
        Py_DECREF(num);
        if (rv) {
            /* Negative */
            PyErr_SetString(PyExc_ValueError,
                            "Illegal IPv4 address (negative)");
            return -1;
        }
        rv = PyObject_RichCompareBool(addr, GLOBALS->maxintipv4, Py_GT);
        if (rv) {
            PyErr_SetString(PyExc_ValueError,
                            "Illegal IPv4 address (integer too large)");
            return -1;
        }

        value = PyLong_AsUnsignedLong(addr);
        skipaddrSetV4(&self->addr, &value);

#if SK_ENABLE_IPV6
    } else if (silkPyIPv6Addr_Check(addr)) {
        /* Convert to v4 */
        silkPyIPAddr *v6addr = (silkPyIPAddr*)addr;

        if (skipaddrV6toV4(&v6addr->addr, &self->addr)) {
            PyErr_SetString(PyExc_ValueError,
                            "IP address not convertable to IPv4.");
            return -1;
        }
#endif  /* SK_ENABLE_IPV6 */

    } else if (silkPyIPv4Addr_Check(addr)) {
        /* Copy */
        skipaddrCopy(&self->addr, &((silkPyIPAddr*)addr)->addr);
    } else {
        PyErr_SetString(PyExc_TypeError, "Must be a string or integer");
        return -1;
    }

    return 0;
}

static PyObject *silkPyIPv4Addr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds)
{
    return silkPyIPvXAddr_new(&silkPyIPv4AddrType, type, args, kwds);
}

static int silkPyIPv6Addr_init(
    silkPyIPAddr   *self,
    PyObject       *args,
    PyObject       *kwds)
{
#if !SK_ENABLE_IPV6
    silkPyNotImplemented((PyObject*)self, args, kwds);
    return -1;
#else  /* if SK_ENABLE_IPV6 */
    static char  *kwlist[] = {"address", NULL};
    PyObject     *addr;
    int           rv;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O", kwlist, &addr)) {
        return -1;
    }

    if (addr == (PyObject*)self) {
        /* We were initialized by new */
        return 0;
    }

    if (IS_STRING(addr)) {
        PyObject *bytes = bytes_from_string(addr);
        if (bytes == NULL) {
            return -1;
        }
        rv = skStringParseIP(&self->addr, PyBytes_AS_STRING(bytes));
        Py_DECREF(bytes);
        if (rv != 0) {
            PyErr_SetString(PyExc_ValueError,
                            "String is not a valid IP address");
            return -1;
        }
        if (!skipaddrIsV6(&self->addr)) {
            PyErr_SetString(PyExc_ValueError,
                            "String is not a valid IPv6 address");
            return -1;
        }
    } else if (IS_INT(addr)) {
        uint8_t   v6[16];
        uint32_t *v6_32;
        PyObject *next;
        PyObject *shift;
        PyObject *num;
        int i;

        num = PyLong_FromLong(0);
        rv = PyObject_RichCompareBool(addr, num, Py_LT);
        Py_DECREF(num);
        if (rv) {
            /* Negative */
            PyErr_SetString(PyExc_ValueError,
                            "Illegal IPv6 address (negative)");
            return -1;
        }
        rv = PyObject_RichCompareBool(addr, GLOBALS->maxintipv6, Py_GT);
        if (rv) {
            PyErr_SetString(PyExc_ValueError,
                            "Illegal IPv6 address (integer too large)");
            return -1;
        }

        /* Set IP */
        shift = PyLong_FromLong(32);
        v6_32 = (uint32_t*)v6 + 3;

        next = addr;
        Py_INCREF(next);
        for (i = 0; i < 4; i++, v6_32--) {
            PyObject *tmp;

            tmp = PyNumber_And(next, GLOBALS->maxintipv4);
            *v6_32 = htonl(PyLong_AsUnsignedLong(tmp));
            Py_DECREF(tmp);
            tmp = next;
            next = PyNumber_Rshift(tmp, shift);
            Py_DECREF(tmp);
        }
        Py_DECREF(shift);
        skipaddrSetV6(&self->addr, v6);

    } else if (silkPyIPv4Addr_Check(addr)) {
        /* Convert to v6 */
        silkPyIPAddr *v4addr = (silkPyIPAddr*)addr;

        if (skipaddrIsV6(&v4addr->addr)) {
            skipaddrCopy(&self->addr, &v4addr->addr);
        } else {
            skipaddrV4toV6(&v4addr->addr, &self->addr);
        }

    } else if (silkPyIPv6Addr_Check(addr)) {
        /* Copy */
        skipaddrCopy(&self->addr, &((silkPyIPAddr*)addr)->addr);
    } else {
        PyErr_SetString(PyExc_TypeError, "Must be a string or integer");
        return -1;
    }

    return 0;
#endif  /* else of !SK_ENABLE_IPV6 */
}

#if SK_ENABLE_IPV6
static PyObject *silkPyIPv6Addr_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds)
{
    return silkPyIPvXAddr_new(&silkPyIPv6AddrType, type, args, kwds);
}
#endif  /* SK_ENABLE_IPV6 */

static PyObject *silkPyIPvXAddr_new(
    PyTypeObject   *basetype,
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds)
{
    static char  *kwlist[] = {"address", NULL};
    PyObject   *self;
    PyObject   *o;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O", kwlist, &o)) {
        return NULL;
    }

    /* FIXME: Is this valid in python 3?  Change to Py_TYPE() instead? */
    if (type == basetype && o->ob_type == basetype) {
        Py_INCREF(o);
        return o;
    }

    self = type->tp_alloc(type, 0);
    return self;
}


/*
 *************************************************************************
 *   IPWildcard
 *************************************************************************
 */

typedef struct silkPyIPWildcard_st {
    PyObject_HEAD
    skIPWildcard_t  wildcard;
    PyObject       *name;
} silkPyIPWildcard;

typedef struct silkPyIPWildcardIter_st {
    PyObject_HEAD
    silkPyIPWildcard       *wildcard;
    skIPWildcardIterator_t  iter;
} silkPyIPWildcardIter;

/* function prototypes */
static void silkPyIPWildcardIter_dealloc(
    silkPyIPWildcardIter   *self);
static PyObject *silkPyIPWildcardIter_iternext(
    silkPyIPWildcardIter   *self);
static int silkPyIPWildcard_contains(
    silkPyIPWildcard   *self,
    PyObject           *obj);
static void silkPyIPWildcard_dealloc(
    silkPyIPWildcard   *obj);
static PyObject *silkPyIPWildcard_is_ipv6(
    silkPyIPWildcard   UNUSED_NOv6(*self));
static PyObject *silkPyIPWildcard_isipv6_deprecated(
    silkPyIPWildcard   *self);
static PyObject *silkPyIPWildcard_iter(
    silkPyIPWildcard   *self);
static PyObject *silkPyIPWildcard_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyIPWildcard_repr(
    silkPyIPWildcard   *obj);
static PyObject *silkPyIPWildcard_str(
    silkPyIPWildcard   *obj);

/* define docs and methods */
static char silkPyIPWildcard_doc[] =
    ("IPWildcard(string) -> IP Wildcard address");

static PyMethodDef silkPyIPWildcard_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"isipv6", (PyCFunction)silkPyIPWildcard_isipv6_deprecated, METH_NOARGS,
     "wild.isipv6() -> bool -- return whether wild is an IPv6 wildcard"},
    {"is_ipv6", (PyCFunction)silkPyIPWildcard_is_ipv6, METH_NOARGS,
     "wild.is_ipv6() -> bool -- return whether wild is an IPv6 wildcard"},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PySequenceMethods silkPyIPWildcard_sequence_methods = {
    0,                          /* sq_length */
    0,                          /* sq_concat */
    0,                          /* sq_repeat */
    0,                          /* sq_item */
    0,                          /* sq_slice */
    0,                          /* sq_ass_item */
    0,                          /* sq_ass_slice */
    (objobjproc)silkPyIPWildcard_contains, /* sq_contains */
    0,                          /* sq_inplace_concat */
    0                           /* sq_inplace_repeat */
};

/* define the object types */
static PyTypeObject silkPyIPWildcardType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.IPWildcard",          /* tp_name */
    sizeof(silkPyIPWildcard),   /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyIPWildcard_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    (reprfunc)silkPyIPWildcard_repr, /* tp_repr */
    0,                          /* tp_as_number */
    &silkPyIPWildcard_sequence_methods, /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    (reprfunc)silkPyIPWildcard_str, /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
                                /* tp_flags */
    silkPyIPWildcard_doc,       /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    (getiterfunc)silkPyIPWildcard_iter, /* tp_iter */
    0,                          /* tp_iternext */
    silkPyIPWildcard_methods,   /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    silkPyIPWildcard_new,       /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyIPWildcardIterType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.IPWildcardIter", /* tp_name */
    sizeof(silkPyIPWildcardIter), /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyIPWildcardIter_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "IP Wildcard iterator object", /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    iter_iter,                  /* tp_iter */
    (iternextfunc)silkPyIPWildcardIter_iternext, /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyIPWildcard_Check(op)                      \
    PyObject_TypeCheck(op, &silkPyIPWildcardType)
#define silkPyIPWildcardIter_Check(op)                  \
    PyObject_TypeCheck(op, &silkPyIPWildcardIterType)

static void silkPyIPWildcardIter_dealloc(
    silkPyIPWildcardIter   *self)
{
    Py_XDECREF(self->wildcard);
    Py_TYPE(self)->tp_free((PyObject*)self);
}

static PyObject *silkPyIPWildcardIter_iternext(
    silkPyIPWildcardIter   *self)
{
    silkPyIPAddr       *addr;
    skIteratorStatus_t  rv;
    skipaddr_t          raw_addr;

    rv = skIPWildcardIteratorNext(&self->iter, &raw_addr);
    if (rv == SK_ITERATOR_NO_MORE_ENTRIES) {
        PyErr_SetNone(PyExc_StopIteration);
        return NULL;
    }
    addr = (silkPyIPAddr*)silkPyIPv4AddrType.tp_alloc(&silkPyIPv4AddrType, 0);
    if (addr == NULL) {
        return NULL;
    }
    addr->addr = raw_addr;

    return (PyObject*)addr;
}

static int silkPyIPWildcard_contains(
    silkPyIPWildcard   *self,
    PyObject           *obj)
{
    int           retval;
    silkPyIPAddr *silkaddr;

    if (IS_STRING(obj)) {
        obj = PyObject_CallFunctionObjArgs((PyObject*)&silkPyIPAddrType,
                                           obj, NULL);
        if (obj == NULL) {
            return -1;
        }
    } else if (silkPyIPAddr_Check(obj)) {
        Py_INCREF(obj);
    } else {
        PyErr_SetString(PyExc_TypeError, "Must be a string or silk.IPAddr");
        return -1;
    }

    silkaddr = (silkPyIPAddr*)obj;
    retval = skIPWildcardCheckIp(&self->wildcard, &silkaddr->addr);
    Py_DECREF(obj);

    return retval ? 1 : 0;
}

static void silkPyIPWildcard_dealloc(
    silkPyIPWildcard   *obj)
{
    Py_XDECREF(obj->name);
    Py_TYPE(obj)->tp_free((PyObject*)obj);
}

static PyObject *silkPyIPWildcard_is_ipv6(
    silkPyIPWildcard   UNUSED_NOv6(*self))
{
#if SK_ENABLE_IPV6
    return PyBool_FromLong(skIPWildcardIsV6(&self->wildcard));
#else
    Py_RETURN_FALSE;
#endif
}

static PyObject *silkPyIPWildcard_isipv6_deprecated(
    silkPyIPWildcard   *self)
{
    PyErr_Warn(PyExc_DeprecationWarning,
               ("IPWildcard.isipv6() is deprecated.  "
                "Use IPWildcard.is_ipv6() instead."));
    return silkPyIPWildcard_is_ipv6(self);
}

static PyObject *silkPyIPWildcard_iter(
    silkPyIPWildcard   *self)
{
    silkPyIPWildcardIter *iter;

    iter = (silkPyIPWildcardIter*)silkPyIPWildcardIterType.tp_alloc(
        &silkPyIPWildcardIterType, 0);
    if (iter) {
        ASSERT_RESULT(skIPWildcardIteratorBind(&iter->iter, &self->wildcard),
                      int, 0);
        Py_INCREF(self);
        iter->wildcard = self;
    }
    return (PyObject*)iter;
}

static PyObject *silkPyIPWildcard_new(
    PyTypeObject   *type,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"wildcard", NULL};
    silkPyIPWildcard *self;

    if (PyArg_ParseTupleAndKeywords(args, kwds, "O!", kwlist,
                                    &silkPyIPWildcardType, &self))
    {
        Py_INCREF(self);
        return (PyObject*)self;
    } else {
        PyErr_Clear();
    }

    self = (silkPyIPWildcard*)type->tp_alloc(type, 0);
    if (self != NULL) {
        int   len;
        const char  *wildcard;
        int          rv;

        if (!PyArg_ParseTupleAndKeywords(args, kwds, "s#", kwlist,
                                         &wildcard, &len))
        {
            Py_DECREF(self);
            return NULL;
        }
        rv = skStringParseIPWildcard(&self->wildcard, wildcard);
        if (rv != 0) {
            Py_DECREF(self);
            return PyErr_Format(PyExc_ValueError,
                                "Illegal IP wildcard: %s", wildcard);
        }
        self->name = PyUnicode_DecodeASCII(wildcard, len, "strict");
        if (self->name == NULL) {
            Py_DECREF(self);
            return NULL;
        }
    }

    return (PyObject*)self;
}

static PyObject *silkPyIPWildcard_repr(
    silkPyIPWildcard   *obj)
{
    PyObject *format;
    PyObject *arg;
    PyObject *retval;

    format = PyUnicode_FromString("silk.IPWildcard(\"%s\")");
    if (format == NULL) {
        return NULL;
    }
    arg = Py_BuildValue("(O)", obj->name);
    if (arg == NULL) {
        Py_DECREF(format);
        return NULL;
    }
    retval = PyUnicode_Format(format, arg);
    Py_DECREF(format);
    Py_DECREF(arg);

    return retval;
}

static PyObject *silkPyIPWildcard_str(
    silkPyIPWildcard   *obj)
{
    Py_INCREF(obj->name);
    return obj->name;
}


/*
 *************************************************************************
 *   IPSet
 *************************************************************************
 */

typedef struct silkPyIPSet_st {
    PyObject_HEAD
    skipset_t  *ipset;
} silkPyIPSet;

typedef struct silkPyIPSetIter_st {
    PyObject_HEAD
    silkPyIPSet *set;
    skipset_iterator_t iter;
    unsigned is_cidr : 1;
} silkPyIPSetIter;

/* function prototypes */
static void silkPyIPSetIter_dealloc(
    silkPyIPSetIter    *self);
static PyObject *silkPyIPSetIter_iternext(
    silkPyIPSetIter    *self);
static PyObject *silkPyIPSet_add(
    silkPyIPSet    *self,
    PyObject       *obj);
static PyObject *silkPyIPSet_cardinality(
    silkPyIPSet    *self);
static PyObject *silkPyIPSet_cidr_iter(
    silkPyIPSet    *self);
static PyObject *silkPyIPSet_clear(
    silkPyIPSet    *self);
static int silkPyIPSet_contains(
    silkPyIPSet    *self,
    PyObject       *obj);
static void silkPyIPSet_dealloc(
    silkPyIPSet    *obj);
static PyObject *silkPyIPSet_difference_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj);
static PyObject *silkPyIPSet_discard(
    silkPyIPSet    *self,
    PyObject       *obj);
static int silkPyIPSet_init(
    silkPyIPSet    *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyIPSet_intersection_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj);
static PyObject *silkPyIPSet_isdisjoint(
    silkPyIPSet    *self,
    PyObject       *obj);
static PyObject *silkPyIPSet_iter(
    silkPyIPSet    *self);
static Py_ssize_t silkPyIPSet_len(
    silkPyIPSet    *self);
static PyObject *silkPyIPSet_save(
    silkPyIPSet    *self,
    PyObject       *name);
static PyObject *silkPyIPSet_union_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj);

/* define docs and methods */
static char silkPyIPSet_doc[] =
    ("IPSetBase() -> empty IPset\n"
     "IPSetBase(filename) -> IPset from file");

static PyMethodDef silkPyIPSet_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"cardinality", (PyCFunction)silkPyIPSet_cardinality, METH_NOARGS,
     "ipset.cardinality() -> long -- number of IP Addresses in the IPSet"},
    {"intersection_update", (PyCFunction)silkPyIPSet_intersection_update,
     METH_O,
     ("Return the intersection of two IPSets as a new IPSet.\n"
      "\n"
      "(i.e. all elements that are in both IPSets.)")},
    {"update", (PyCFunction)silkPyIPSet_union_update,
     METH_O,
     ("Update the IPSet with the union of itself and another.")},
    {"difference_update", (PyCFunction)silkPyIPSet_difference_update,
     METH_O,
     ("Remove all elements of another IPSet from this IPSet.")},
    {"clear", (PyCFunction)silkPyIPSet_clear, METH_NOARGS,
     ("Remove all elements from this IPSet.")},
    {"save", (PyCFunction)silkPyIPSet_save, METH_O,
     "ipset.save(filename) -- Saves the set to a file."},
    {"cidr_iter", (PyCFunction)silkPyIPSet_cidr_iter, METH_NOARGS,
     "Return an iterator over IPAddr/prefix tuples."},
    {"add", (PyCFunction)silkPyIPSet_add, METH_O,
     ("Add an element to an IPSet.  The element can be an IP address, an\n"
      "IP wildcard, or the string representation of either.\n"
      "\n"
      "This has no effect for any element already present.")},
    {"discard", (PyCFunction)silkPyIPSet_discard, METH_O,
     ("Discard an element to an IPSet.  The element can be an IP address, an\n"
      "IP wildcard, or the string representation of either.\n"
      "\n"
      "This has no effect for any element not present in the IPset.")},
    {"isdisjoint", (PyCFunction)silkPyIPSet_isdisjoint, METH_O,
     "Return whether the IPSet has any elements in common with the argument"},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PySequenceMethods silkPyIPSet_sequence_methods = {
#if PY_VERSION_HEX < 0x02050000
    (inquiry)silkPyIPSet_len,   /* sq_length */
#else
    (lenfunc)silkPyIPSet_len,   /* sq_length */
#endif
    0,                          /* sq_concat */
    0,                          /* sq_repeat */
    0,                          /* sq_item */
    0,                          /* sq_slice */
    0,                          /* sq_ass_item */
    0,                          /* sq_ass_slice */
    (objobjproc)silkPyIPSet_contains, /* sq_contains */
    0,                          /* sq_inplace_concat */
    0                           /* sq_inplace_repeat */
};

/* define the object types */
static PyTypeObject silkPyIPSetType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.IPSetBase",    /* tp_name */
    sizeof(silkPyIPSet),        /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyIPSet_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    &silkPyIPSet_sequence_methods, /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyIPSet_doc,            /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    (getiterfunc)silkPyIPSet_iter, /* tp_iter */
    0,                          /* tp_iternext */
    silkPyIPSet_methods,        /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyIPSet_init, /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyIPSetIterType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.IPSetIter",    /* tp_name */
    sizeof(silkPyIPSetIter),    /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyIPSetIter_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "IP Set iterator object",   /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    iter_iter,                  /* tp_iter */
    (iternextfunc)silkPyIPSetIter_iternext, /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};


/* macro and function defintions */
#define silkPyIPSet_Check(op)                   \
    PyObject_TypeCheck(op, &silkPyIPSetType)
#define silkPyIPSetIter_Check(op)                       \
    PyObject_TypeCheck(op, &silkPyIPSetIterType)

static void silkPyIPSetIter_dealloc(
    silkPyIPSetIter    *self)
{
    Py_XDECREF(self->set);
    Py_TYPE(self)->tp_free((PyObject*)self);
}

static PyObject *silkPyIPSetIter_iternext(
    silkPyIPSetIter    *self)
{
    silkPyIPAddr       *addr;
    int                 rv;
    PyObject           *retval;
    skipaddr_t          raw_addr;
    uint32_t            raw_prefix;

    rv = skIPSetIteratorNext(&self->iter, &raw_addr, &raw_prefix);
    if (rv == SK_ITERATOR_NO_MORE_ENTRIES) {
        PyErr_SetNone(PyExc_StopIteration);
        return NULL;
    }

    addr = (silkPyIPAddr*)silkPyIPAddrType.tp_alloc(&silkPyIPAddrType, 0);
    if (addr == NULL) {
        return NULL;
    }
    addr->addr = raw_addr;

    if (!self->is_cidr) {
        retval = (PyObject*)addr;
    } else {
        PyObject       *pair;
        PyObject       *len;

        len = PyInt_FromLong(raw_prefix);
        if (len == NULL) {
            Py_DECREF(addr);
            return NULL;
        }
        pair = PyTuple_New(2);
        if (pair == NULL) {
            Py_DECREF(addr);
            Py_DECREF(len);
        }
        PyTuple_SET_ITEM(pair, 0, (PyObject*)addr);
        PyTuple_SET_ITEM(pair, 1, len);

        retval = pair;
    }

    return retval;
}

static PyObject *silkPyIPSet_add(
    silkPyIPSet    *self,
    PyObject       *obj)
{
    int           rv;

    if (silkPyIPAddr_Check(obj)) {
        silkPyIPAddr *addr = (silkPyIPAddr*)obj;

        rv = skIPSetInsertAddress(self->ipset, &addr->addr,
                                  (skipaddrIsV6(&addr->addr) ? 128 : 32));
    } else if silkPyIPWildcard_Check(obj) {
        silkPyIPWildcard *wild;

        assert(silkPyIPWildcard_Check(obj));

        wild = (silkPyIPWildcard*)obj;

        rv = skIPSetInsertIPWildcard(self->ipset, &wild->wildcard);
    } else {
        PyErr_SetString(PyExc_TypeError,
                        "Must be a silk.IPAddr or a silk.IPWildcard");
        return NULL;
    }

    if (rv == SKIPSET_ERR_ALLOC) {
        return PyErr_NoMemory();
    }
    if (rv == SKIPSET_ERR_IPV6) {
        PyErr_SetString(PyExc_ValueError,
                        "Must only include IPv4 addresses");
        return NULL;
    }
    assert(rv == SKIPSET_OK);

    Py_INCREF(self);
    return (PyObject*)self;
}

static PyObject *silkPyIPSet_cardinality(
    silkPyIPSet    *self)
{
    uint64_t count;
    double count_d;

    skIPSetClean(self->ipset);
    count = skIPSetCountIPs(self->ipset, &count_d);
    return PyLong_FromUnsignedLongLong(count);
}

static PyObject *silkPyIPSet_cidr_iter(
    silkPyIPSet    *self)
{
    silkPyIPSetIter *iter;

    iter = (silkPyIPSetIter*)silkPyIPSetIterType.tp_alloc(
        &silkPyIPSetIterType, 0);
    if (iter) {
        skIPSetClean(self->ipset);
        if (skIPSetIteratorBind(&iter->iter, self->ipset, 1,SK_IPV6POLICY_MIX))
        {
            Py_DECREF(iter);
            return PyErr_NoMemory();
        }
        Py_INCREF(self);
        iter->set = self;
        iter->is_cidr = 1;
    }
    return (PyObject*)iter;
}

static PyObject *silkPyIPSet_clear(
    silkPyIPSet    *self)
{
    skIPSetRemoveAll(self->ipset);

    Py_INCREF(self);
    return (PyObject*)self;
}

static int silkPyIPSet_contains(
    silkPyIPSet    *self,
    PyObject       *obj)
{
    int           retval;
    silkPyIPAddr *silkaddr;

    if (IS_STRING(obj)) {
        obj = PyObject_CallFunctionObjArgs((PyObject*)&silkPyIPAddrType,
                                           obj, NULL);
        if (obj == NULL) {
            return -1;
        }
    } else if (silkPyIPAddr_Check(obj)) {
        Py_INCREF(obj);
    } else {
        PyErr_SetString(PyExc_TypeError, "Must be a string or silk.IPAddr");
        return -1;
    }

    silkaddr = (silkPyIPAddr*)obj;
    retval = skIPSetCheckAddress(self->ipset, &silkaddr->addr);
    Py_DECREF(obj);

    return retval ? 1 : 0;
}

static void silkPyIPSet_dealloc(
    silkPyIPSet    *obj)
{
    if (obj->ipset) {
        skIPSetDestroy(&obj->ipset);
    }
    Py_TYPE(obj)->tp_free((PyObject*)obj);
}

static PyObject *silkPyIPSet_difference_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj)
{
    if (!silkPyIPSet_Check(obj)) {
        PyErr_SetString(PyExc_NotImplementedError,
                        "Argument must be a silk.IPSet");
        return NULL;
    }

    skIPSetClean(self->ipset);
    skIPSetClean(obj->ipset);
    skIPSetSubtract(self->ipset, obj->ipset);

    Py_INCREF(self);
    return (PyObject*)self;
}

static PyObject *silkPyIPSet_discard(
    silkPyIPSet    *self,
    PyObject       *obj)
{
    int rv;

    if (silkPyIPAddr_Check(obj)) {
        silkPyIPAddr *addr = (silkPyIPAddr*)obj;

        rv = skIPSetRemoveAddress(self->ipset, &addr->addr,
                                  (skipaddrIsV6(&addr->addr) ? 128 : 32));
    } else if silkPyIPWildcard_Check(obj) {
        silkPyIPWildcard *wild;

        assert(silkPyIPWildcard_Check(obj));

        wild = (silkPyIPWildcard*)obj;

        rv = skIPSetRemoveIPWildcard(self->ipset, &wild->wildcard);
    } else {
        PyErr_SetString(PyExc_TypeError,
                        "Must be a silk.IPAddr or a silk.IPWildcard");
        return NULL;
    }

    if (rv == SKIPSET_ERR_ALLOC) {
        return PyErr_NoMemory();
    }
    assert(rv == SKIPSET_OK);

    Py_INCREF(self);
    return (PyObject*)self;
}

static int silkPyIPSet_init(
    silkPyIPSet    *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"filename", NULL};
    char        *fname = NULL;
    int          rv;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|et", kwlist,
                                     Py_FileSystemDefaultEncoding, &fname))
    {
        return -1;
    }

    if (fname) {
        rv = skIPSetLoad(&self->ipset, fname);
        if (rv != 0) {
            PyErr_Format(PyExc_IOError, "Unable to read IPSet from %s: %s",
                         fname, skIPSetStrerror(rv));
            PyMem_Free(fname);
            return -1;
        }
        PyMem_Free(fname);
    } else {
        rv = skIPSetCreate(&self->ipset, 0);
        if (rv == SKIPSET_ERR_ALLOC) {
            PyErr_NoMemory();
            return -1;
        }
        assert(rv == SKIPSET_OK);
    }

    return 0;
}

static PyObject *silkPyIPSet_intersection_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj)
{
    if (!silkPyIPSet_Check(obj)) {
        PyErr_SetString(PyExc_NotImplementedError,
                        "Argument must be a silk.IPSet");
        return NULL;
    }

    skIPSetClean(self->ipset);
    skIPSetClean(obj->ipset);
    skIPSetIntersect(self->ipset, obj->ipset);

    Py_INCREF(self);
    return (PyObject*)self;
}

static PyObject *silkPyIPSet_isdisjoint(
    silkPyIPSet    *self,
    PyObject       *obj)
{
    int disjoint;

    if (silkPyIPSet_Check(obj)) {
        disjoint = !skIPSetCheckIPSet(
            self->ipset, ((silkPyIPSet*)obj)->ipset);
    } else if (silkPyIPWildcard_Check(obj)) {
        disjoint = !skIPSetCheckIPWildcard(
            self->ipset, &((silkPyIPWildcard*)obj)->wildcard);
    } else {
        PyErr_SetString(PyExc_TypeError, "Expected an IPSet or an IPWildcard");
        return NULL;
    }
    if (disjoint) {
        Py_RETURN_TRUE;
    }
    Py_RETURN_FALSE;
}

static PyObject *silkPyIPSet_iter(
    silkPyIPSet    *self)
{
    silkPyIPSetIter *iter;

    iter = (silkPyIPSetIter*)silkPyIPSetIterType.tp_alloc(
        &silkPyIPSetIterType, 0);
    if (iter) {
        skIPSetClean(self->ipset);
        if (skIPSetIteratorBind(&iter->iter, self->ipset, 0,SK_IPV6POLICY_MIX))
        {
            Py_DECREF(iter);
            return PyErr_NoMemory();
        }
        Py_INCREF(self);
        iter->set = self;
    }
    return (PyObject*)iter;
}

static Py_ssize_t silkPyIPSet_len(
    silkPyIPSet    *self)
{
    uint64_t count;
    double count_d;

    skIPSetClean(self->ipset);
    count = skIPSetCountIPs(self->ipset, &count_d);
    if (count > PY_SSIZE_T_MAX) {
        PyErr_SetString(PyExc_OverflowError, "IPSet too long for integer");
        return -1;
    }
    return (Py_ssize_t)count;
}

static PyObject *silkPyIPSet_save(
    silkPyIPSet    *self,
    PyObject       *name)
{
    int rv;
    PyObject *bytes;

    if (!IS_STRING(name)) {
        PyErr_SetString(PyExc_TypeError, "Filename required");
        return NULL;
    }
#if PY_MAJOR_VERSION < 3
    if (PyBytes_Check(name)) {
        bytes = name;
        Py_INCREF(bytes);
    } else
#endif
    {
        bytes = PyUnicode_AsEncodedString(name, Py_FileSystemDefaultEncoding,
                                          "strict");
        if (bytes == NULL) {
            return NULL;
        }
    }

    skIPSetClean(self->ipset);
    rv = skIPSetSave(self->ipset, PyBytes_AS_STRING(bytes));
    Py_DECREF(bytes);
    if (rv != SKIPSET_OK) {
        PyErr_SetString(PyExc_IOError, skIPSetStrerror(rv));
        return NULL;
    }

    Py_RETURN_NONE;
}

static PyObject *silkPyIPSet_union_update(
    silkPyIPSet    *self,
    silkPyIPSet    *obj)
{
    int rv;

    if (!silkPyIPSet_Check(obj)) {
        PyErr_SetString(PyExc_NotImplementedError,
                        "Argument must be a silk.IPSet");
        return NULL;
    }

    skIPSetClean(self->ipset);
    skIPSetClean(obj->ipset);
    rv = skIPSetUnion(self->ipset, obj->ipset);
    if (rv != 0) {
        return PyErr_NoMemory();
    }

    Py_INCREF(self);
    return (PyObject*)self;
}


/* *************************************************************************
 *   Prefix Map
 *************************************************************************
 */

typedef struct silkPyPmap_st {
    PyObject_HEAD
    skPrefixMap_t *map;
} silkPyPmap;

typedef struct silkPyPmapIter_st {
    PyObject_HEAD
    silkPyPmap *map;
    skPrefixMapIterator_t iter;
} silkPyPmapIter;

/* function prototypes */
static void silkPyPmapIter_dealloc(
    silkPyPmapIter *self);
static PyObject *silkPyPmapIter_iternext(
    silkPyPmapIter *self);
static void silkPyPmap_dealloc(
    silkPyPmap *obj);
static PyObject *silkPyPmap_get_content(
    silkPyPmap     *self,
    void    UNUSED(*cbdata));
static PyObject *silkPyPmap_get_name(
    silkPyPmap     *self,
    void    UNUSED(*cbdata));
static PyObject *silkPyPmap_get_num_values(
    silkPyPmap     *self,
    void    UNUSED(*cbdata));
static PyObject *silkPyPmap_get_value_string(
    silkPyPmap *self,
    PyObject   *value);
static int silkPyPmap_init(
    silkPyPmap *self,
    PyObject   *args,
    PyObject   *kwds);
static PyObject *silkPyPmap_iter(
    silkPyPmap *self);
static PyObject *silkPyPmap_subscript(
    silkPyPmap *self,
    PyObject   *sub);

/* define docs and methods */
static char silkPyPmap_doc[] =
    ("PMapBase(filename) -> Prefix map from file");

static PyMethodDef silkPyPmap_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"get_value_string", (PyCFunction)silkPyPmap_get_value_string, METH_O,
     ("Get the string associated with an integer value")},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PyGetSetDef silkPyPmap_getsetters[] = {
    {"content", (getter)silkPyPmap_get_content, NULL, "Content type", NULL},
    {"name", (getter)silkPyPmap_get_name, NULL, "Prefix map name", NULL},
    {"num_values", (getter)silkPyPmap_get_num_values, NULL,
     "Prefix map number of values", NULL},
    {NULL, NULL, NULL, NULL, NULL}
};

static PyMappingMethods silkPyPmap_mapping_methods = {
    0,
    (binaryfunc)silkPyPmap_subscript,
    0
};

/* define the object types */
static PyTypeObject silkPyPmapType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.PMapBase",     /* tp_name */
    sizeof(silkPyPmap),         /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyPmap_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    &silkPyPmap_mapping_methods, /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyPmap_doc,             /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    (getiterfunc)silkPyPmap_iter, /* tp_iter */
    0,                          /* tp_iternext */
    silkPyPmap_methods,         /* tp_methods */
    0,                          /* tp_members */
    silkPyPmap_getsetters,      /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyPmap_init,  /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyPmapIterType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.PMapBaseIter", /* tp_name */
    sizeof(silkPyPmapIterType), /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyPmapIter_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Prefix map iterator object", /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    iter_iter,                  /* tp_iter */
    (iternextfunc)silkPyPmapIter_iternext, /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyPmap_Check(op)                    \
    PyObject_TypeCheck(op, &silkPyPmapType)
#define silkPyPmapIter_Check(op)                \
    PyObject_TypeCheck(op, &silkPyPmapIterType)

static void silkPyPmapIter_dealloc(
    silkPyPmapIter *self)
{
    Py_XDECREF(self->map);
    Py_TYPE(self)->tp_free((PyObject*)self);
}

static PyObject *silkPyPmapIter_iternext(
    silkPyPmapIter *self)
{
    skIteratorStatus_t    rv;
    PyObject             *retval;
    PyObject             *startval = NULL;
    PyObject             *endval = NULL;
    union {
        skipaddr_t             addr;
        skPrefixMapProtoPort_t pp;
    } start, end;
    uint32_t              value;
    skPrefixMapContent_t  content;

    rv = skPrefixMapIteratorNext(&self->iter, &start, &end, &value);
    if (rv == SK_ITERATOR_NO_MORE_ENTRIES) {
        PyErr_SetNone(PyExc_StopIteration);
        return NULL;
    }

    content = skPrefixMapGetContentType(self->map->map);
    switch (content) {
      case SKPREFIXMAP_CONT_ADDR_V4:
      case SKPREFIXMAP_CONT_ADDR_V6:
        {
            PyTypeObject *type =
                (content == SKPREFIXMAP_CONT_ADDR_V4) ?
                &silkPyIPv4AddrType : &silkPyIPv6AddrType;

            startval = type->tp_alloc(type, 0);
            if (startval == NULL) {
                return NULL;
            }
            endval = type->tp_alloc(type, 0);
            if (endval == NULL) {
                Py_DECREF(startval);
                return NULL;
            }
            skipaddrCopy(&((silkPyIPAddr*)startval)->addr, &start.addr);
            skipaddrCopy(&((silkPyIPAddr*)endval)->addr, &end.addr);
        }
        break;
      case SKPREFIXMAP_CONT_PROTO_PORT:
        {
            startval = Py_BuildValue("BH", start.pp.proto, start.pp.port);
            if (startval == NULL) {
                return NULL;
            }
            endval = Py_BuildValue("BH", end.pp.proto, end.pp.port);
            if (endval == NULL) {
                Py_DECREF(startval);
                return NULL;
            }
        }
        break;
    }

    retval = Py_BuildValue("NNk", startval, endval, value);
    if (retval == NULL) {
        Py_DECREF(startval);
        Py_DECREF(endval);
    }

    return retval;
}

static void silkPyPmap_dealloc(
    silkPyPmap *obj)
{
    if (obj->map) {
        skPrefixMapDelete(obj->map);
    }
    Py_TYPE(obj)->tp_free((PyObject*)obj);
}

static PyObject *silkPyPmap_get_content(
    silkPyPmap     *self,
    void    UNUSED(*cbdata))
{
    return PyUnicode_FromString(skPrefixMapGetContentName(
                                    skPrefixMapGetContentType(self->map)));
}

static PyObject *silkPyPmap_get_name(
    silkPyPmap     *self,
    void    UNUSED(*cbdata))
{
    const char *name = skPrefixMapGetMapName(self->map);
    if (name == NULL) {
        Py_RETURN_NONE;
    }

    return PyUnicode_FromString(name);
}

static PyObject *silkPyPmap_get_num_values(
    silkPyPmap     *self,
    void    UNUSED(*cbdata))
{
    return PyInt_FromLong(skPrefixMapDictionaryGetWordCount(self->map));
}

static PyObject *silkPyPmap_get_value_string(
    silkPyPmap *self,
    PyObject   *value)
{
    uint32_t  val;
    uint32_t  size;
    char     *buf;
    int       rv;
    PyObject *retval;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return NULL;
    }

    val = PyLong_AsUnsignedLong(value);
    if (PyErr_Occurred()) {
        return NULL;
    }

    size = skPrefixMapDictionaryGetMaxWordSize(self->map) + 1;
    buf = (char*)malloc(size);
    if (buf == NULL) {
        return PyErr_NoMemory();
    }

    rv = skPrefixMapDictionaryGetEntry(self->map, val, buf, size);
    assert(rv < (int32_t)size);

    retval = PyUnicode_DecodeASCII(buf, rv, "strict");
    free(buf);

    return retval;
}

static int silkPyPmap_init(
    silkPyPmap *self,
    PyObject   *args,
    PyObject   *kwds)
{
    static char      *kwlist[] = {"filename", NULL};
    char             *fname;
    skPrefixMapErr_t  rv;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "et", kwlist,
                                     Py_FileSystemDefaultEncoding, &fname))
    {
        return -1;
    }

    rv = skPrefixMapLoad(&self->map, fname);
    if (rv != SKPREFIXMAP_OK) {
        PyMem_Free(fname);
        PyErr_Format(PyExc_IOError, "Unable to read prefix map from %s: %s",
                     fname, skPrefixMapStrerror(rv));
        return -1;
    }
    PyMem_Free(fname);

    return 0;
}

static PyObject *silkPyPmap_iter(
    silkPyPmap *self)
{
    int             rv;
    silkPyPmapIter *iter;

    iter = (silkPyPmapIter*)silkPyPmapIterType.tp_alloc(
        &silkPyPmapIterType, 0);
    if (iter) {
        rv = skPrefixMapIteratorBind(&iter->iter, self->map);
        if (rv != 0) {
            Py_DECREF(iter);
            PyErr_SetString(PyExc_RuntimeError,
                            "Failed to create prefix map iterator");
            return NULL;
        }
        Py_INCREF(self);
        iter->map = self;
    }
    return (PyObject*)iter;
}

static PyObject *silkPyPmap_subscript(
    silkPyPmap *self,
    PyObject   *sub)
{
    void *key = NULL;
    uint32_t value;
    skPrefixMapProtoPort_t protoport;
    skPrefixMapContent_t content;
    PyObject *tuple;
    int32_t i32;
    int rv;

    content = skPrefixMapGetContentType(self->map);

    switch (content) {
      case SKPREFIXMAP_CONT_ADDR_V4:
      case SKPREFIXMAP_CONT_ADDR_V6:
        if (!silkPyIPAddr_Check(sub)) {
            PyErr_SetString(PyExc_TypeError, "Expected an IPAddr");
            return NULL;
        }
        key = &((silkPyIPAddr*)sub)->addr;
        break;
      case SKPREFIXMAP_CONT_PROTO_PORT:
        if (!PySequence_Check(sub) || PySequence_Size(sub) != 2) {
            PyErr_SetString(PyExc_TypeError, "Expected a (proto, port) pair");
            return NULL;
        }
        tuple = PySequence_Tuple(sub);
        if (tuple == NULL) {
            return NULL;
        }
        rv = PyArg_ParseTuple(tuple, "bi;Expected a (proto, port) pair",
                              &protoport.proto, &i32);
        Py_DECREF(tuple);
        if (!rv) {
            return NULL;
        }
        if (i32 < 0 || i32 > 0xFFFF) {
            PyErr_SetString(PyExc_OverflowError, "Port is out of bounds");
            return NULL;
        }
        protoport.port = i32;
        key = &protoport;
        break;
    }

    value = skPrefixMapGet(self->map, key);

    return PyLong_FromUnsignedLong(value);
}


/*
 *************************************************************************
 *   Bag
 *************************************************************************
 */

typedef struct silkPyBag_st {
    PyObject_HEAD
    skBag_t *bag;
    unsigned is_ipaddr : 1;
} silkPyBag;

typedef struct silkPyBagIter_st {
    PyObject_HEAD
    silkPyBag       *bag;
    skBagIterator_t *iter;
    unsigned         ipaddr : 1;
} silkPyBagIter;

/* function prototypes */
static void silkPyBagIter_dealloc(
    silkPyBagIter  *self);
static PyObject *silkPyBagIter_iternext(
    silkPyBagIter  *self);
static PyObject *silkPyBag__get_custom_type(
    PyObject    UNUSED(*self));
static PyObject *silkPyBag__get_ipv4_type(
    PyObject    UNUSED(*self));
static PyObject *silkPyBag__get_ipv6_type(
    PyObject    UNUSED(*self));
static int silkPyBag_ass_subscript(
    silkPyBag  *self,
    PyObject   *sub,
    PyObject   *value);
static PyObject *silkPyBag_clear(
    silkPyBag  *self);
static Py_ssize_t silkPyBag_count(
    silkPyBag  *self);
static void silkPyBag_dealloc(
    silkPyBag  *obj);
static PyObject *silkPyBag_decr(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds);
static PyObject *silkPyBag_field_types(
    PyObject    UNUSED(*self));
static PyObject *silkPyBag_get_info(
    silkPyBag  *self);
static PyObject *silkPyBag_iadd(
    silkPyBag  *self,
    silkPyBag  *other);
#if 0
static skBagErr_t silkPyBag_iadd_bounds(
    const skBagTypedKey_t       UNUSED(*key),
    skBagTypedCounter_t                *in_out_counter,
    const skBagTypedCounter_t   UNUSED(*in_counter),
    void                        UNUSED(*cb_data));
#endif
static PyObject *silkPyBag_incr(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds);
static int silkPyBag_init(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds);
static PyObject *silkPyBag_iter(
    silkPyBag  *self);
static PyObject *silkPyBag_iter_helper(
    silkPyBag  *self,
    int         sorted);
static int silkPyBag_modify(
    silkPyBag      *self,
    PyObject       *sub,
    PyObject       *value,
    silkBagModFn    fn);
static PyObject *silkPyBag_save(
    silkPyBag  *self,
    PyObject   *name);
static PyObject *silkPyBag_set_info(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds);
static int silkPyBag_setup(
    PyObject   *mod);
static PyObject *silkPyBag_sorted_iter(
    silkPyBag  *self);
static PyObject *silkPyBag_subscript(
    silkPyBag  *self,
    PyObject   *sub);
static PyObject *silkPyBag_type_merge(
    PyObject    UNUSED(*self),
    PyObject           *args);

/* define docs and methods */
static char silkPyBag_doc[] =
    ("BagBase(filename) -> Bag from file");

static PyNumberMethods silkPyBag_number_methods;

static PyMappingMethods silkPyBag_mapping_methods = {
#if PY_VERSION_HEX < 0x02050000
    (inquiry)silkPyBag_count,   /* mp_length */
#else
    (lenfunc)silkPyBag_count,   /* mp_length */
#endif
    (binaryfunc)silkPyBag_subscript,       /* mp_subscript */
    (objobjargproc)silkPyBag_ass_subscript /* mp_ass_subscript */
};

static PyMethodDef silkPyBag_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"incr", (PyCFunction)silkPyBag_incr, METH_KEYWORDS | METH_VARARGS,
     ("bag.incr(key, value) -- increments bag[key] by value")},
    {"decr", (PyCFunction)silkPyBag_decr, METH_KEYWORDS | METH_VARARGS,
     ("bag.decr(key, value) -- decrements bag[key] by value")},
    {"save", (PyCFunction)silkPyBag_save, METH_O,
     "bag.save(filename) -- saves the bag to a file"},
    {"clear", (PyCFunction)silkPyBag_clear, METH_NOARGS,
     "bag.clear() -- empties the bag"},
    {"sorted_iter", (PyCFunction)silkPyBag_sorted_iter, METH_NOARGS,
     ("bag.sorted_iter() -- returns an iterator whose values are "
      "sorted by key")},
    {"get_info", (PyCFunction)silkPyBag_get_info, METH_NOARGS,
     "get_info() -- returns (key_type, key_len, counter_type, counter_len)"},
    {"set_info", (PyCFunction)silkPyBag_set_info,
     METH_KEYWORDS | METH_VARARGS,
     ("set_info([key_type][, key_len][, counter_type][, counter_len]) --\n\t"
      "returns the result of bag.get_info()")},
    {"field_types", (PyCFunction)silkPyBag_field_types,
     METH_NOARGS | METH_STATIC,
     "field_types() -> Tuple of valid field types for Bag keys and counters"},
    {"type_merge", (PyCFunction)silkPyBag_type_merge,
     METH_VARARGS | METH_STATIC,
     "type_merge(keytype_a, keytype_b) -> key_type of merged output"},
    {"_get_custom_type", (PyCFunction)silkPyBag__get_custom_type,
     METH_NOARGS | METH_STATIC, NULL},
    {"_get_ipv4_type", (PyCFunction)silkPyBag__get_ipv4_type,
     METH_NOARGS | METH_STATIC, NULL},
    {"_get_ipv6_type", (PyCFunction)silkPyBag__get_ipv6_type,
     METH_NOARGS | METH_STATIC, NULL},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};


/* define the object types */
static PyTypeObject silkPyBagType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.BagBase",      /* tp_name */
    sizeof(silkPyBag),          /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyBag_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    &silkPyBag_number_methods,  /* tp_as_number */
    0,                          /* tp_as_sequence */
    &silkPyBag_mapping_methods, /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
#if PY_MAJOR_VERSION < 3
    Py_TPFLAGS_CHECKTYPES |
#endif
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyBag_doc,              /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    (getiterfunc)silkPyBag_iter, /* tp_iter */
    0,                          /* tp_iternext */
    silkPyBag_methods,          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyBag_init,   /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyBagIterType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.BagBaseIter",  /* tp_name */
    sizeof(silkPyBagIterType),  /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyBagIter_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Bag iterator object",      /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    iter_iter,                  /* tp_iter */
    (iternextfunc)silkPyBagIter_iternext, /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    0,                          /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyBag_Check(op)                     \
    PyObject_TypeCheck(op, &silkPyBagType)
#define silkPyBagIter_Check(op)                 \
    PyObject_TypeCheck(op, &silkPyBagIterType)

#define IS_IPV4_KEY(k)                          \
    ((k) == SKBAG_FIELD_SIPv4                   \
     || (k) == SKBAG_FIELD_DIPv4                \
     || (k) == SKBAG_FIELD_NHIPv4               \
     || (k) == SKBAG_FIELD_ANY_IPv4)

#define IS_IPV6_KEY(k)                          \
    ((k) == SKBAG_FIELD_SIPv6                   \
     || (k) == SKBAG_FIELD_DIPv6                \
     || (k) == SKBAG_FIELD_NHIPv6               \
     || (k) == SKBAG_FIELD_ANY_IPv6)

#define IS_IP_KEY(k) (IS_IPV4_KEY(k) || IS_IPV6_KEY(k))

static void silkPyBagIter_dealloc(
    silkPyBagIter  *self)
{
    Py_XDECREF(self->bag);
    skBagIteratorDestroy(self->iter);
    Py_TYPE(self)->tp_free((PyObject*)self);
}

static PyObject *silkPyBagIter_iternext(
    silkPyBagIter  *self)
{
    skBagErr_t           rv;
    PyObject            *retkey;
    skBagTypedKey_t      key;
    skBagTypedCounter_t  counter;

    counter.type = SKBAG_COUNTER_U64;
    key.type = self->ipaddr ? SKBAG_KEY_IPADDR : SKBAG_KEY_U32;

    rv = skBagIteratorNextTyped(self->iter, &key, &counter);
    if (rv == SKBAG_ERR_KEY_NOT_FOUND) {
        PyErr_SetNone(PyExc_StopIteration);
        return NULL;
    }
    if (rv == SKBAG_ERR_MODIFIED) {
        PyErr_SetString(PyExc_RuntimeError,
                        "Underlying Bag changed during iteration");
        return NULL;
    }
    if (self->ipaddr) {
        PyTypeObject *type = (skipaddrIsV6(&key.val.addr)
                              ? &silkPyIPv6AddrType : &silkPyIPv4AddrType);
        retkey = type->tp_alloc(type, 0);
        if (retkey == NULL) {
            return NULL;
        }
        skipaddrCopy(&((silkPyIPAddr*)retkey)->addr, &key.val.addr);
    } else {
        retkey = PyLong_FromUnsignedLong(key.val.u32);
        if (retkey == NULL) {
            return NULL;
        }
    }

    return Py_BuildValue("OK", retkey, counter.val.u64);
}

static PyObject *silkPyBag__get_custom_type(
    PyObject    UNUSED(*self))
{
    char buf[SKBAG_MAX_FIELD_BUFLEN];

    skBagFieldTypeAsString(SKBAG_FIELD_CUSTOM, buf, sizeof(buf));
    return PyUnicode_FromString(buf);
}

static PyObject *silkPyBag__get_ipv4_type(
    PyObject    UNUSED(*self))
{
    char buf[SKBAG_MAX_FIELD_BUFLEN];

    skBagFieldTypeAsString(SKBAG_FIELD_ANY_IPv4, buf, sizeof(buf));
    return PyUnicode_FromString(buf);
}

static PyObject *silkPyBag__get_ipv6_type(
    PyObject    UNUSED(*self))
{
    char buf[SKBAG_MAX_FIELD_BUFLEN];

    skBagFieldTypeAsString(SKBAG_FIELD_ANY_IPv6, buf, sizeof(buf));
    return PyUnicode_FromString(buf);
}

static int silkPyBag_ass_subscript(
    silkPyBag  *self,
    PyObject   *sub,
    PyObject   *value)
{
    /* skBagCounterSet will ignore the extra NULL passed to it */
    return silkPyBag_modify(self, sub, value, (silkBagModFn)skBagCounterSet);
}

static PyObject *silkPyBag_clear(
    silkPyBag  *self)
{
    skBagErr_t rv;
    skBagFieldType_t key, value;
    size_t keylen, valuelen;
    skBag_t *bag;

    key = skBagKeyFieldType(self->bag);
    keylen = skBagKeyFieldLength(self->bag);
    value = skBagCounterFieldType(self->bag);
    valuelen = skBagCounterFieldLength(self->bag);

    rv = skBagCreateTyped(&bag, key, value, keylen, valuelen);
    if (rv == SKBAG_ERR_MEMORY) {
        return PyErr_NoMemory();
    }
    assert(rv == SKBAG_OK);
    skBagAutoConvertDisable(bag);

    skBagDestroy(&self->bag);
    self->bag = bag;

    Py_RETURN_NONE;
}

static Py_ssize_t silkPyBag_count(
    silkPyBag  *self)
{
    uint64_t count = skBagCountKeys(self->bag);
    return (Py_ssize_t)count;
}

static void silkPyBag_dealloc(
    silkPyBag  *obj)
{
    if (obj->bag) {
        skBagDestroy(&obj->bag);
    }
    Py_TYPE(obj)->tp_free((PyObject*)obj);
}

static PyObject *silkPyBag_decr(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds)
{
    static char *kwlist[] = {"key", "value", NULL};
    PyObject *sub;
    PyObject *value;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO", kwlist, &sub, &value)) {
        return NULL;
    }

    if (silkPyBag_modify(self, sub, value, skBagCounterSubtract)) {
        return NULL;
    }

    Py_RETURN_NONE;
}

static PyObject *silkPyBag_field_types(
    PyObject    UNUSED(*self))
{
    char buf[SKBAG_MAX_FIELD_BUFLEN];
    skBagFieldTypeIterator_t iter;
    Py_ssize_t count;
    PyObject *retval;

    /* First, count thee types */
    skBagFieldTypeIteratorBind(&iter);
    count = 0;
    while (skBagFieldTypeIteratorNext(&iter, NULL, NULL, NULL, 0) == SKBAG_OK)
    {
        count++;
    }

    /* Create the tuple */
    retval = PyTuple_New(count);
    if (retval == NULL) {
        return NULL;
    }

    /* Then fill in the tuple */
    skBagFieldTypeIteratorReset(&iter);
    count = 0;
    while (skBagFieldTypeIteratorNext(&iter, NULL, NULL, buf, sizeof(buf))
           == SKBAG_OK)
    {
        PyObject *name = PyUnicode_InternFromString(buf);
        if (name == NULL) {
            Py_DECREF(retval);
            return NULL;
        }
        PyTuple_SET_ITEM(retval, count, name);
        count++;
    }

    return retval;
}

static PyObject *silkPyBag_get_info(
    silkPyBag  *self)
{
    char buf[80];
    unsigned int key_len;
    unsigned int counter_len;
    PyObject *key_name;
    PyObject *counter_name;

    skBagKeyFieldName(self->bag, buf, sizeof(buf));
    key_len = skBagKeyFieldLength(self->bag);
    key_name = PyUnicode_FromString(buf);
    if (key_name == NULL) {
        return NULL;
    }
    skBagCounterFieldName(self->bag, buf, sizeof(buf));
    counter_len = skBagCounterFieldLength(self->bag);
    counter_name = PyUnicode_FromString(buf);
    if (counter_name == NULL) {
        Py_DECREF(key_name);
        return NULL;
    }
    return Py_BuildValue("{sN sI sN sI}",
                         "key_type", key_name, "key_len", key_len,
                         "counter_type", counter_name,
                         "counter_len", counter_len);
}

static PyObject *silkPyBag_iadd(
    silkPyBag  *self,
    silkPyBag  *other)
{
    skBagErr_t rv;

    if (!silkPyBag_Check(other)) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    rv = skBagAddBag(self->bag, other->bag, NULL, NULL);
    switch (rv) {
      case SKBAG_OK:
        break;
      case SKBAG_ERR_MEMORY:
        PyErr_NoMemory();
        return NULL;
      case SKBAG_ERR_OP_BOUNDS:
        PyErr_SetString(PyExc_OverflowError, skBagStrerror(rv));
        return NULL;
      case SKBAG_ERR_KEY_RANGE:
        PyErr_SetString(PyExc_ValueError, skBagStrerror(rv));
        return NULL;
      case SKBAG_ERR_INPUT:
      case SKBAG_ERR_KEY_NOT_FOUND:
        /* Fall through */
      default:
        skAbortBadCase(rv);
    }
    self->is_ipaddr = (skBagKeyFieldLength(self->bag) == 16
                       || IS_IP_KEY(skBagKeyFieldType(self->bag)));

    Py_INCREF(self);
    return (PyObject*)self;
}

#if 0
/* Clamp bag values for iadd to SKBAG_COUNTER_MAX without errors. */
static skBagErr_t silkPyBag_iadd_bounds(
    const skBagTypedKey_t       UNUSED(*key),
    skBagTypedCounter_t                *in_out_counter,
    const skBagTypedCounter_t   UNUSED(*in_counter),
    void                        UNUSED(*cb_data))
{
    in_out_counter->type = SKBAG_COUNTER_U64;
    in_out_counter->val.u64 = SKBAG_COUNTER_MAX;
    return SKBAG_OK;
}
#endif

static PyObject *silkPyBag_incr(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds)
{
    static char *kwlist[] = {"key", "value", NULL};
    PyObject *sub;
    PyObject *value;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO", kwlist, &sub, &value)) {
        return NULL;
    }

    if (silkPyBag_modify(self, sub, value, skBagCounterAdd)) {
        return NULL;
    }

    Py_RETURN_NONE;
}

static int silkPyBag_init(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds)
{
    static char *kwlist[] = {"copy", "filename", "key_type", "counter_type",
                             "key_len", "counter_len", NULL};
    char         *fname        = NULL;
    silkPyBag    *copy         = NULL;
    char         *key          = NULL;
    char         *counter      = NULL;
    unsigned int  key_size     = 0;
    unsigned int  counter_size = 0;
    skBagErr_t    rv;

    if (!PyArg_ParseTupleAndKeywords(
            args, kwds, "|O!etssII", kwlist, &silkPyBagType,
            (PyObject*)&copy, Py_FileSystemDefaultEncoding, &fname,
            &key, &counter, &key_size, &counter_size))
    {
        return -1;
    }

    if ((copy && fname)
        || (copy && (key || counter))
        || (fname && (key || counter)))
    {
        PyErr_SetString(PyExc_ValueError, "Illegal argument combination");
        return -1;
    }

    if (fname) {
        rv = skBagLoad(&self->bag, fname);
        if (rv != SKBAG_OK) {
            PyErr_Format(PyExc_IOError, "Unable to read Bag from %s: %s",
                         fname, skBagStrerror(rv));
            PyMem_Free(fname);
            return -1;
        }
        PyMem_Free(fname);
        self->is_ipaddr = skBagKeyFieldLength(self->bag) == 16
                          || IS_IP_KEY(skBagKeyFieldType(self->bag));
    } else if (copy) {
        rv = skBagCopy(&self->bag, copy->bag);
        self->is_ipaddr = copy->is_ipaddr;
    } else {
        skBagFieldType_t key_type, counter_type;
        if (!key) {
            key_type = SKBAG_FIELD_CUSTOM;
        } else {
            rv = skBagFieldTypeLookup(key, &key_type, NULL);
            if (rv != SKBAG_OK) {
                PyErr_Format(PyExc_ValueError,
                             "'%s' is not a valid key type", key);
                return -1;
            }
        }
        if (key_type == SKBAG_FIELD_CUSTOM && key_size == 0) {
            key_size = 4;
        }
        if (!counter) {
            counter_type = SKBAG_FIELD_CUSTOM;
        } else {
            rv = skBagFieldTypeLookup(counter, &counter_type, NULL);
            if (rv != SKBAG_OK) {
                PyErr_Format(PyExc_ValueError,
                             "'%s' is not a valid counter type", counter);
                return -1;
            }
        }
        if (counter_type == SKBAG_FIELD_CUSTOM && counter_size == 0) {
            counter_size = 8;
        }

        rv = skBagCreateTyped(&self->bag, key_type, counter_type,
                              key_size, counter_size);
        if (rv == SKBAG_ERR_INPUT) {
            PyErr_Format(PyExc_ValueError,
                         "Illegal arguments to Bag constructor");
            return -1;
        }
        skBagAutoConvertDisable(self->bag);
        self->is_ipaddr = (counter_size == 16 || IS_IP_KEY(key_type));
    }

    if (rv == SKBAG_ERR_MEMORY) {
        PyErr_NoMemory();
        return -1;
    }
    assert(rv == SKBAG_OK);

    return 0;
}

static PyObject *silkPyBag_iter(
    silkPyBag  *self)
{
    return silkPyBag_iter_helper(self, 0);
}

static PyObject *silkPyBag_iter_helper(
    silkPyBag  *self,
    int         sorted)
{
    skBagErr_t     rv;
    silkPyBagIter *iter;

    iter = (silkPyBagIter*)silkPyBagIterType.tp_alloc(
        &silkPyBagIterType, 0);
    if (iter) {
        if (sorted) {
            rv = skBagIteratorCreate(self->bag, &iter->iter);
        } else {
            rv = skBagIteratorCreateUnsorted(self->bag, &iter->iter);
        }
        if (rv == SKBAG_ERR_MEMORY) {
            Py_DECREF(iter);
            return PyErr_NoMemory();
        }
        if (rv != SKBAG_OK) {
            Py_DECREF(iter);
            PyErr_SetString(PyExc_RuntimeError,
                            "Failed to create bag iterator");
            return NULL;
        }
        Py_INCREF(self);
        iter->bag = self;
        iter->ipaddr = self->is_ipaddr;
    }
    return (PyObject*)iter;
}

static int silkPyBag_modify(
    silkPyBag      *self,
    PyObject       *sub,
    PyObject       *value,
    silkBagModFn    fn)
{
    skBagTypedCounter_t bagvalue;
    skBagTypedKey_t     key;
    skBagErr_t          rv;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer value");
        return -1;
    }
    bagvalue.val.u64 = LONG_AS_UNSIGNED_LONGLONG(value);
    if (PyErr_Occurred()) {
        return -1;
    }
    bagvalue.type = SKBAG_COUNTER_U64;

    if (IS_INT(sub)) {
        if (self->is_ipaddr) {
            PyErr_SetString(PyExc_TypeError, "Expected an IPAddr index");
            return -1;
        }
        /* long long is 64-bits on 32 and 64-bit arches, so use it for
         * consistency. */
        key.val.u64 = LONG_AS_UNSIGNED_LONGLONG(sub);
        if (PyErr_Occurred()) {
            if (PyErr_ExceptionMatches(PyExc_OverflowError)) {
                PyErr_Clear();
                PyErr_SetString(PyExc_IndexError, "Index out of range");
            }
            return -1;
        }
        if (key.val.u64 > 0xffffffff) {
            PyErr_SetString(PyExc_IndexError, "Index out of range");
            return -1;
        }
        key.val.u32 = key.val.u64;
        key.type = SKBAG_KEY_U32;
    } else if (silkPyIPAddr_Check(sub)) {
        silkPyIPAddr *addr;
        if (!self->is_ipaddr) {
            PyErr_SetString(PyExc_TypeError, "Expected an integer index");
            return -1;
        }
        addr = (silkPyIPAddr*)sub;
        skipaddrCopy(&key.val.addr, &addr->addr);
        key.type = SKBAG_KEY_IPADDR;
    } else {
        PyErr_SetString(PyExc_TypeError, "Expected an integer or IP address");
        return -1;
    }

    rv = fn(self->bag, &key, &bagvalue, NULL);
    switch (rv) {
      case SKBAG_OK:
        break;
      case SKBAG_ERR_INPUT:
      case SKBAG_ERR_KEY_RANGE:
        PyErr_SetString(PyExc_IndexError, "Address out of range");
        return -1;
      case SKBAG_ERR_MEMORY:
        PyErr_NoMemory();
        return -1;
      case SKBAG_ERR_OP_BOUNDS:
        PyErr_SetString(PyExc_OverflowError, skBagStrerror(rv));
        return -1;
      case SKBAG_ERR_KEY_NOT_FOUND:
        /* Fall through */
      default:
        skAbortBadCase(rv);
    }

    return 0;
}

static PyObject *silkPyBag_save(
    silkPyBag  *self,
    PyObject   *name)
{
    skBagErr_t rv;
    PyObject *bytes;

    if (!IS_STRING(name)) {
        PyErr_SetString(PyExc_TypeError, "Filename required");
        return NULL;
    }
#if PY_MAJOR_VERSION < 3
    if (PyBytes_Check(name)) {
        bytes = name;
        Py_INCREF(bytes);
    } else
#endif
    {
        bytes = PyUnicode_AsEncodedString(name, Py_FileSystemDefaultEncoding,
                                          "strict");
        if (bytes == NULL) {
            return NULL;
        }
    }

    rv = skBagSave(self->bag, PyBytes_AS_STRING(bytes));
    Py_DECREF(bytes);
    if (rv != SKBAG_OK) {
        PyErr_SetString(PyExc_IOError, skBagStrerror(rv));
        return NULL;
    }

    Py_RETURN_NONE;
}

static PyObject *silkPyBag_set_info(
    silkPyBag  *self,
    PyObject   *args,
    PyObject   *kwds)
{
    static char *kwlist[] = {"key_type", "key_len",
                             "counter_type", "counter_len", NULL};
    size_t key_len = SKBAG_OCTETS_NO_CHANGE;
    size_t counter_len = SKBAG_OCTETS_NO_CHANGE;
    unsigned int key_len_tmp = UINT_MAX;
    unsigned int counter_len_tmp = UINT_MAX;
    char *key_name = NULL;
    char *counter_name = NULL;
    skBagFieldType_t key_type;
    skBagFieldType_t counter_type;
    skBagErr_t err;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|sIsI", kwlist,
                                     &key_name, &key_len_tmp,
                                     &counter_name, &counter_len_tmp))
    {
        return NULL;
    }

    /* key_len_tmp and counter_len_tmp are unsigned ints, since there
     * were no conversion functions to size_t until Python 2.6 */
    if (key_len_tmp != UINT_MAX) {
        key_len = key_len_tmp;
    }
    if (counter_len_tmp != UINT_MAX) {
        counter_len = counter_len_tmp;
    }
    if (key_name) {
        err = skBagFieldTypeLookup(key_name, &key_type, NULL);
        if (err != SKBAG_OK) {
            assert(err == SKBAG_ERR_INPUT);
            return PyErr_Format(PyExc_ValueError,
                                "'%s' is not a valid key type", key_name);
        }
    } else {
        key_type = skBagKeyFieldType(self->bag);
    }

    if (counter_name) {
        err = skBagFieldTypeLookup(counter_name, &counter_type, NULL);
        if (err != SKBAG_OK) {
            assert(err == SKBAG_ERR_INPUT);
            return PyErr_Format(PyExc_ValueError,
                                "'%s' is not a valid counter type",
                                counter_name);
        }
    } else {
        counter_type = skBagCounterFieldType(self->bag);
    }

    err = skBagModify(self->bag, key_type, counter_type,
                      key_len, counter_len);
    if (err != SKBAG_OK) {
        PyErr_SetString(PyExc_ValueError,
                        "Illegal value was passed to Bag.set_info");
        return NULL;
    }

    self->is_ipaddr = (counter_len == 16 || IS_IP_KEY(key_type));

    return silkPyBag_get_info(self);
}

static int silkPyBag_setup(
    PyObject   *mod)
{
    /* Setup number methods */
    memset(&silkPyBag_number_methods, 0, sizeof(silkPyBag_number_methods));
    silkPyBag_number_methods.nb_inplace_add = (binaryfunc)silkPyBag_iadd;

    /* Initialize type and add to module */
    silkPyBagType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyBagType) < 0) {
        return -1;
    }
    return PyModule_AddObject(mod, "BagBase", (PyObject*)&silkPyBagType);
}

static PyObject *silkPyBag_sorted_iter(
    silkPyBag  *self)
{
    return silkPyBag_iter_helper(self, 1);
}

static PyObject *silkPyBag_subscript(
    silkPyBag  *self,
    PyObject   *sub)
{
    skBagTypedKey_t     key;
    skBagTypedCounter_t value;
    skBagErr_t          rv;

    if (IS_INT(sub)) {
        if (self->is_ipaddr) {
            PyErr_SetString(PyExc_TypeError, "Expected an IPAddr index");
            return NULL;
        }
        /* long long is 64-bits on 32 and 64-bit arches, so use it for
         * consistency. */
        key.val.u64 = LONG_AS_UNSIGNED_LONGLONG(sub);
        if (PyErr_Occurred()) {
            if (PyErr_ExceptionMatches(PyExc_OverflowError)) {
                PyErr_Clear();
                PyErr_SetString(PyExc_IndexError, "Index out of range");
            }
            return NULL;
        }
        if (key.val.u64 > 0xffffffff) {
            PyErr_SetString(PyExc_IndexError, "Index out of range");
            return NULL;
        }
        key.val.u32 = key.val.u64;
        key.type = SKBAG_KEY_U32;
    } else if (silkPyIPAddr_Check(sub)) {
        silkPyIPAddr *addr;
        if (!self->is_ipaddr) {
            PyErr_SetString(PyExc_TypeError, "Expected an integer index");
            return NULL;
        }
        addr = (silkPyIPAddr*)sub;
        skipaddrCopy(&key.val.addr, &addr->addr);
        key.type = SKBAG_KEY_IPADDR;
    } else {
        PyErr_SetString(PyExc_TypeError, "Expected an integer or IP address");
        return NULL;
    }

    value.type = SKBAG_COUNTER_U64;
    rv = skBagCounterGet(self->bag, &key, &value);

    assert(rv != SKBAG_ERR_KEY_NOT_FOUND);
    if (rv == SKBAG_ERR_KEY_RANGE) {
        PyErr_SetString(PyExc_IndexError, "Index out of range");
        return NULL;
    }
    if (rv != SKBAG_OK) {
        PyErr_SetString(PyExc_ValueError, skBagStrerror(rv));
        return NULL;
    }
    assert(value.type == SKBAG_COUNTER_U64);

    return PyLong_FromUnsignedLongLong(value.val.u64);
}

static PyObject *silkPyBag_type_merge(
    PyObject    UNUSED(*self),
    PyObject           *args)
{
    char *a, *b;
    skBagFieldType_t a_type, b_type, c_type;
    skBagErr_t rv;
    char buf[SKBAG_MAX_FIELD_BUFLEN];

    if (!PyArg_ParseTuple(args, "ss", &a, &b)) {
        return NULL;
    }
    rv = skBagFieldTypeLookup(a, &a_type, NULL);
    if (rv != SKBAG_OK) {
        PyErr_Format(PyExc_ValueError, "'%s' is not a valid key type", a);
    }
    rv = skBagFieldTypeLookup(b, &b_type, NULL);
    if (rv != SKBAG_OK) {
        PyErr_Format(PyExc_ValueError, "'%s' is not a valid key type", b);
    }
    c_type = skBagFieldTypeMerge(a_type, b_type);
    skBagFieldTypeAsString(c_type, buf, sizeof(buf));
    return PyUnicode_FromString(buf);
}


/*
 *************************************************************************
 *   TCP Flags
 *************************************************************************
 */

typedef struct silkPyTCPFlags_st {
    PyObject_HEAD
    uint8_t val;
} silkPyTCPFlags;

/* function prototypes */
static PyObject *silkPyTCPFlags_and(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b);
static PyObject *silkPyTCPFlags_getflag(
    silkPyTCPFlags *obj,
    void           *bit);
static PyObject *silkPyTCPFlags_getflag_deprecated(
    silkPyTCPFlags *obj,
    void           *bit);
static long silkPyTCPFlags_hash(
    silkPyTCPFlags *obj);
static int silkPyTCPFlags_init(
    silkPyTCPFlags *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyTCPFlags_int(
    silkPyTCPFlags *obj);
static silkPyTCPFlags *silkPyTCPFlags_invert(
    silkPyTCPFlags *obj);
static PyObject *silkPyTCPFlags_matches(
    silkPyTCPFlags *self,
    PyObject       *arg);
static PyObject *silkPyTCPFlags_new(
    PyTypeObject       *type,
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds));
static int silkPyTCPFlags_nonzero(
    silkPyTCPFlags *a);
static PyObject *silkPyTCPFlags_or(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b);
static PyObject *silkPyTCPFlags_padded(
    silkPyTCPFlags *obj);
static PyObject *silkPyTCPFlags_repr(
    silkPyTCPFlags *obj);
static PyObject *silkPyTCPFlags_richcompare(
    silkPyTCPFlags *self,
    PyObject       *obj,
    int             cmp);
static int silkPyTCPFlags_setup(
    PyObject   *mod);
static PyObject *silkPyTCPFlags_str(
    silkPyTCPFlags *obj);
static PyObject *silkPyTCPFlags_xor(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b);

/* define docs and methods */
static char silkPyTCPFlags_doc[] =
    ("TCPFlags(string)   -> TCPFlags based on flag string\n"
     "TCPFlags(int)      -> TCPFlags based on integer representation\n"
     "TCPFlags(TCPFlags) -> Copy of TCPFlags");

static PyNumberMethods silkPyTCPFlags_number_methods;

static PyMethodDef silkPyTCPFlags_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"matches", (PyCFunction)silkPyTCPFlags_matches, METH_O,
     "Return whether the flags match the high/mask flagstring"},
    {"padded", (PyCFunction)silkPyTCPFlags_padded, METH_NOARGS,
     "Returns the flags string padded with spaces, so flags line up"},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static uint8_t flags_array[] = {
    FIN_FLAG, SYN_FLAG, RST_FLAG, PSH_FLAG,
    ACK_FLAG, URG_FLAG, ECE_FLAG, CWR_FLAG};

static PyGetSetDef silkPyTCPFlags_getsetters[] = {
    {"fin", (getter)silkPyTCPFlags_getflag, NULL,
     "FIN flag", (void*)&flags_array[0]},
    {"syn", (getter)silkPyTCPFlags_getflag, NULL,
     "SYN flag", (void*)&flags_array[1]},
    {"rst", (getter)silkPyTCPFlags_getflag, NULL,
     "RST flag", (void*)&flags_array[2]},
    {"psh", (getter)silkPyTCPFlags_getflag, NULL,
     "PSH flag", (void*)&flags_array[3]},
    {"ack", (getter)silkPyTCPFlags_getflag, NULL,
     "ACK flag", (void*)&flags_array[4]},
    {"urg", (getter)silkPyTCPFlags_getflag, NULL,
     "URG flag", (void*)&flags_array[5]},
    {"ece", (getter)silkPyTCPFlags_getflag, NULL,
     "ECE flag", (void*)&flags_array[6]},
    {"cwr", (getter)silkPyTCPFlags_getflag, NULL,
     "CWR flag", (void*)&flags_array[7]},

    {"FIN", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "FIN flag", (void*)&flags_array[0]},
    {"SYN", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "SYN flag", (void*)&flags_array[1]},
    {"RST", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "RST flag", (void*)&flags_array[2]},
    {"PSH", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "PSH flag", (void*)&flags_array[3]},
    {"ACK", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "ACK flag", (void*)&flags_array[4]},
    {"URG", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "URG flag", (void*)&flags_array[5]},
    {"ECE", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "ECE flag", (void*)&flags_array[6]},
    {"CWR", (getter)silkPyTCPFlags_getflag_deprecated, NULL,
     "CWR flag", (void*)&flags_array[7]},
    {NULL, NULL, NULL, NULL, NULL}
};

/* define the object types */
static PyTypeObject silkPyTCPFlagsType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.TCPFlags",            /* tp_name */
    sizeof(silkPyTCPFlags),     /* tp_basicsize */
    0,                          /* tp_itemsize */
    obj_dealloc,                /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    (reprfunc)silkPyTCPFlags_repr, /* tp_repr */
    &silkPyTCPFlags_number_methods, /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    (hashfunc)silkPyTCPFlags_hash, /* tp_hash  */
    0,                          /* tp_call */
    (reprfunc)silkPyTCPFlags_str, /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
#if PY_MAJOR_VERSION < 3
    Py_TPFLAGS_CHECKTYPES | Py_TPFLAGS_HAVE_RICHCOMPARE |
#endif
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    silkPyTCPFlags_doc,         /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    (richcmpfunc)silkPyTCPFlags_richcompare, /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    silkPyTCPFlags_methods,     /* tp_methods */
    0,                          /* tp_members */
    silkPyTCPFlags_getsetters,  /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyTCPFlags_init, /* tp_init */
    0,                          /* tp_alloc */
    silkPyTCPFlags_new,         /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyTCPFlags_Check(op)                \
    PyObject_TypeCheck(op, &silkPyTCPFlagsType)

static PyObject *silkPyTCPFlags_and(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b)
{
    PyObject *new_obj;

    if (!silkPyTCPFlags_Check(b)) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    new_obj = silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType, 0);
    if (new_obj != NULL) {
        ((silkPyTCPFlags*)new_obj)->val = a->val & b->val;
    }
    return new_obj;
}

static PyObject *silkPyTCPFlags_getflag(
    silkPyTCPFlags *obj,
    void           *bit)
{
    return PyBool_FromLong(obj->val & *(uint8_t*)bit);
}

static PyObject *silkPyTCPFlags_getflag_deprecated(
    silkPyTCPFlags *obj,
    void           *bit)
{
    PyErr_Warn(PyExc_DeprecationWarning,
               ("Use of upper-case flag check attributes for "
                "TCPFlags is deprecated"));
    return PyBool_FromLong(obj->val & *(uint8_t*)bit);
}

static long silkPyTCPFlags_hash(
    silkPyTCPFlags *obj)
{
    return obj->val;
}

static int silkPyTCPFlags_init(
    silkPyTCPFlags *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"value", NULL};
    PyObject *val;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "O", kwlist, &val)) {
        return -1;
    }

    if (silkPyTCPFlags_Check(val)) {
        silkPyTCPFlags *oflags = (silkPyTCPFlags*)val;
        self->val = oflags->val;
    } else if (IS_INT(val)) {
        long intval = PyLong_AsLong(val);
        if (intval < 0 || intval > (long)UINT8_MAX) {
            PyErr_Format(PyExc_OverflowError,
                         "Illegal TCP flag value: %ld", intval);
            return -1;
        }
        self->val = intval;
    } else if (IS_STRING(val)) {
        PyObject *bytes = bytes_from_string(val);
        char *strval;

        if (bytes == NULL) {
            return -1;
        }
        strval = PyBytes_AS_STRING(bytes);
        if (skStringParseTCPFlags(&self->val, strval)) {
            PyErr_Format(PyExc_ValueError,
                         "Illegal TCP flag value: %s", strval);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
    } else {
        obj_error("Illegal value: %s", val);
        return -1;
    }

    return 0;
}

static PyObject *silkPyTCPFlags_int(
    silkPyTCPFlags *obj)
{
    return PyInt_FromLong(obj->val);
}

static silkPyTCPFlags *silkPyTCPFlags_invert(
    silkPyTCPFlags *obj)
{
    silkPyTCPFlags *new_obj =
        (silkPyTCPFlags*)silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType, 0);

    if (new_obj != NULL) {
        new_obj->val = ~obj->val;
    }
    return new_obj;
}

static PyObject *silkPyTCPFlags_matches(
    silkPyTCPFlags *self,
    PyObject       *arg)
{
    char *repr;
    PyObject *bytes;
    uint8_t high, mask;
    int rv;

    if (!IS_STRING(arg)) {
        PyErr_SetString(PyExc_TypeError, "Expected string");
        return NULL;
    }

    bytes = bytes_from_string(arg);
    repr = PyBytes_AS_STRING(bytes);
    rv = skStringParseTCPFlagsHighMask(&high, &mask, repr);
    Py_DECREF(bytes);
    if (rv == SKUTILS_ERR_SHORT) {
        mask = high;
    } else if (rv != 0) {
        PyErr_SetString(PyExc_ValueError, "Illegal flag/mask");
        return NULL;
    }

    return PyBool_FromLong((self->val & mask) == high);
}

static PyObject *silkPyTCPFlags_new(
    PyTypeObject       *type,
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds))
{
    silkPyTCPFlags *self;

    self = (silkPyTCPFlags*)type->tp_alloc(type, 0);

    if (self != NULL) {
        self->val = 0;
    }

    return (PyObject*)self;
}

static int silkPyTCPFlags_nonzero(
    silkPyTCPFlags *a)
{
    return a->val ? 1 : 0;
}

static PyObject *silkPyTCPFlags_or(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b)
{
    PyObject *new_obj;

    if (!silkPyTCPFlags_Check(b)) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    new_obj = silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType, 0);
    if (new_obj != NULL) {
        ((silkPyTCPFlags*)new_obj)->val = a->val | b->val;
    }
    return new_obj;
}

static PyObject *silkPyTCPFlags_padded(
    silkPyTCPFlags *obj)
{
    char flags[SK_TCPFLAGS_STRLEN];

    tcpflags_string_r(obj->val, flags);

    return PyUnicode_FromString(flags);
}

static PyObject *silkPyTCPFlags_repr(
    silkPyTCPFlags *obj)
{
    char flags[SK_TCPFLAGS_STRLEN];

    tcpflags_string_r(obj->val, flags);

    return PyUnicode_FromFormat("silk.TCPFlags('%s')", flags);
}

static PyObject *silkPyTCPFlags_richcompare(
    silkPyTCPFlags *self,
    PyObject       *obj,
    int             cmp)
{
    if (cmp != Py_EQ && cmp != Py_NE) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    if (!silkPyTCPFlags_Check(obj)) {
        PyErr_SetString(PyExc_TypeError, "Expected silk.TCPFlags");
        return NULL;
    }

    if (self->val == ((silkPyTCPFlags*)obj)->val) {
        if (cmp == Py_EQ) {
            Py_RETURN_TRUE;
        } else {
            Py_RETURN_FALSE;
        }
    }
    if (cmp == Py_NE) {
        Py_RETURN_TRUE;
    }

    Py_RETURN_FALSE;
}

static int silkPyTCPFlags_setup(
    PyObject   *mod)
{
    /* Setup number methods */
    memset(&silkPyTCPFlags_number_methods, 0,
           sizeof(silkPyTCPFlags_number_methods));
#if PY_MAJOR_VERSION >= 3
    silkPyTCPFlags_number_methods.nb_bool =
        (inquiry)silkPyTCPFlags_nonzero;
#else
    silkPyTCPFlags_number_methods.nb_nonzero =
        (inquiry)silkPyTCPFlags_nonzero;
#endif
    silkPyTCPFlags_number_methods.nb_invert =
        (unaryfunc)silkPyTCPFlags_invert;
    silkPyTCPFlags_number_methods.nb_and = (binaryfunc)silkPyTCPFlags_and;
    silkPyTCPFlags_number_methods.nb_xor = (binaryfunc)silkPyTCPFlags_xor;
    silkPyTCPFlags_number_methods.nb_or = (binaryfunc)silkPyTCPFlags_or;
    silkPyTCPFlags_number_methods.nb_int = (unaryfunc)silkPyTCPFlags_int;

    /* Initialize type and add to module */
    if (PyType_Ready(&silkPyTCPFlagsType) < 0) {
        return -1;
    }
    return PyModule_AddObject(mod, "TCPFlags",
                              (PyObject*)&silkPyTCPFlagsType);
}

static PyObject *silkPyTCPFlags_str(
    silkPyTCPFlags *obj)
{
    char flags[SK_TCPFLAGS_STRLEN];
    char unpadded[SK_TCPFLAGS_STRLEN];
    char *f, *u;

    tcpflags_string_r(obj->val, flags);
    for (f = flags, u = unpadded; *f; f++) {
        if (*f != ' ') {
            *u++ = *f;
        }
    }
    *u = '\0';

    return PyUnicode_FromString(unpadded);
}

static PyObject *silkPyTCPFlags_xor(
    silkPyTCPFlags *a,
    silkPyTCPFlags *b)
{
    PyObject *new_obj;

    if (!silkPyTCPFlags_Check(b)) {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    new_obj = silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType, 0);
    if (new_obj != NULL) {
        ((silkPyTCPFlags*)new_obj)->val = a->val ^ b->val;
    }
    return new_obj;
}


/*
 *************************************************************************
 *   RWRec
 *************************************************************************
 */

typedef struct silkPyRawRWRec_st {
    PyObject_HEAD
    rwRec rec;
} silkPyRawRWRec;

typedef struct silkPyRWRec_st {
    PyObject_HEAD
    silkPyRawRWRec *raw;
} silkPyRWRec;

/* function prototypes */
static PyObject *silkPyRWRec_application_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_application_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_bytes_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_bytes_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_classname_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_classtype_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_classtype_id_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_classtype_id_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static int silkPyRWRec_classtype_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static void silkPyRWRec_dealloc(
    silkPyRWRec    *obj);
static PyObject *silkPyRWRec_dip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_dip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_dport_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_dport_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_duration_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_duration_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_duration_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static int silkPyRWRec_duration_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_etime_epoch_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_etime_epoch_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_etime_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_etime_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_finnoack_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_finnoack_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_icmpcode_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_icmpcode_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_icmptype_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_icmptype_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_new(
    PyTypeObject *type,
    PyObject     *args,
    PyObject     *kwds);
static int silkPyRWRec_init(
    silkPyRWRec    *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyRWRec_initial_tcpflags_get(
    silkPyRWRec    *obj,
    PyObject       *deprecated);
static int silkPyRWRec_initial_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    PyObject       *deprecated);
static PyObject *silkPyRWRec_input_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_input_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_is_web(
    silkPyRWRec    *obj);
static PyObject *silkPyRWRec_nhip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_nhip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_output_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_output_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_packets_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_packets_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_protocol_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_protocol_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_richcompare(
    silkPyRWRec    *self,
    PyObject       *obj,
    int             cmp);
static PyObject *silkPyRWRec_sensor_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_sensor_id_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_sensor_id_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static int silkPyRWRec_sensor_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_session_tcpflags_get(
    silkPyRWRec    *obj,
    PyObject       *deprecated);
static int silkPyRWRec_session_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    PyObject       *deprecated);
static PyObject *silkPyRWRec_sip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_sip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_sport_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_sport_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_stime_epoch_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_stime_epoch_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_stime_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_stime_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_tcpflags_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_timeout_killed_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_timeout_killed_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_timeout_started_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_timeout_started_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_typename_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static PyObject *silkPyRWRec_uniform_packets_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure));
static int silkPyRWRec_uniform_packets_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure));
static int silkPyRawRWRec_init(
    silkPyRawRWRec *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyRawRWRec_new(
    PyTypeObject       *type,
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds));

/* define docs and methods */
static PyMethodDef silkPyRWRec_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"is_web", (PyCFunction)silkPyRWRec_is_web, METH_NOARGS,
     "Returns whether IPv6 can be stored in a WWW file format"},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PyGetSetDef silkPyRWRec_getseters[] = {
    {"application",
     (getter)silkPyRWRec_application_get, (setter)silkPyRWRec_application_set,
     "\"service\" port set by the collector", NULL},
    {"bytes",
     (getter)silkPyRWRec_bytes_get,       (setter)silkPyRWRec_bytes_set,
     "Count of bytes", NULL},
    {"classname",
     (getter)silkPyRWRec_classname_get,   NULL,
     "class name (read-only)", NULL},
    {"classtype",
     (getter)silkPyRWRec_classtype_get,   (setter)silkPyRWRec_classtype_set,
     "class name, type name pair", NULL},
    {"classtype_id",
     (getter)silkPyRWRec_classtype_id_get,(setter)silkPyRWRec_classtype_id_set,
     "class, type pair ID", NULL},
    {"dip",
     (getter)silkPyRWRec_dip_get,         (setter)silkPyRWRec_dip_set,
     "destination IP", NULL},
    {"dport",
     (getter)silkPyRWRec_dport_get,       (setter)silkPyRWRec_dport_set,
     "Destination port", NULL},
    {"duration",
     (getter)silkPyRWRec_duration_get,    (setter)silkPyRWRec_duration_set,
     "duration of flow as datetime.timedelta", NULL},
    {"duration_secs",
     (getter)silkPyRWRec_duration_secs_get,
     (setter)silkPyRWRec_duration_secs_set,
     "duration of flow in seconds", NULL},
    {"etime",
     (getter)silkPyRWRec_etime_get,       (setter)silkPyRWRec_etime_set,
     "end time of flow as datetime.timedelta", NULL},
    {"etime_epoch_secs",
     (getter)silkPyRWRec_etime_epoch_secs_get,
     (setter)silkPyRWRec_etime_epoch_secs_set,
     "end time of flow as a number of seconds since the epoch time", NULL},
    {"finnoack",
     (getter)silkPyRWRec_finnoack_get,    (setter)silkPyRWRec_finnoack_set,
     "FIN followed by not ACK", NULL},
    {"icmpcode",
     (getter)silkPyRWRec_icmpcode_get,    (setter)silkPyRWRec_icmpcode_set,
     "ICMP code", NULL},
    {"icmptype",
     (getter)silkPyRWRec_icmptype_get,    (setter)silkPyRWRec_icmptype_set,
     "ICMP type", NULL},
    {"initflags",
     (getter)silkPyRWRec_initial_tcpflags_get,
     (setter)silkPyRWRec_initial_tcpflags_set,
     "TCP flags of first packet", Py_True}, /* Deprecated */
    {"initial_tcpflags",
     (getter)silkPyRWRec_initial_tcpflags_get,
     (setter)silkPyRWRec_initial_tcpflags_set,
     "TCP flags of first packet", NULL},
    {"input",
     (getter)silkPyRWRec_input_get,       (setter)silkPyRWRec_input_set,
     "router incoming SNMP interface", NULL},
    {"nhip",
     (getter)silkPyRWRec_nhip_get,        (setter)silkPyRWRec_nhip_set,
     "router next hop IP", NULL},
    {"output",
     (getter)silkPyRWRec_output_get,      (setter)silkPyRWRec_output_set,
     "router outgoing SNMP interface", NULL},
    {"packets",
     (getter)silkPyRWRec_packets_get,     (setter)silkPyRWRec_packets_set,
     "count of packets", NULL},
    {"protocol",
     (getter)silkPyRWRec_protocol_get,    (setter)silkPyRWRec_protocol_set,
     "IP protocol", NULL},
    {"restflags",
     (getter)silkPyRWRec_session_tcpflags_get,
     (setter)silkPyRWRec_session_tcpflags_set,
     "TCP flags on non-initial packets", Py_True}, /* Deprecated */
    {"sensor",
     (getter)silkPyRWRec_sensor_get,      (setter)silkPyRWRec_sensor_set,
     "sensor name", NULL},
    {"sensor_id",
     (getter)silkPyRWRec_sensor_id_get,   (setter)silkPyRWRec_sensor_id_set,
     "sensor ID", NULL},
    {"session_tcpflags",
     (getter)silkPyRWRec_session_tcpflags_get,
     (setter)silkPyRWRec_session_tcpflags_set,
     "TCP flags on non-initial packets", NULL},
    {"sip",
     (getter)silkPyRWRec_sip_get,         (setter)silkPyRWRec_sip_set,
     "source IP", NULL},
    {"sport",
     (getter)silkPyRWRec_sport_get,       (setter)silkPyRWRec_sport_set,
     "source port", NULL},
    {"stime",
     (getter)silkPyRWRec_stime_get,       (setter)silkPyRWRec_stime_set,
     "start time of flow as datetime.datetime", NULL},
    {"stime_epoch_secs",
     (getter)silkPyRWRec_stime_epoch_secs_get,
     (setter)silkPyRWRec_stime_epoch_secs_set,
     "start time of flow as a number of seconds since the epoch time", NULL},
    {"tcpflags",
     (getter)silkPyRWRec_tcpflags_get,    (setter)silkPyRWRec_tcpflags_set,
     "OR of all tcpflags", NULL},
    {"timeout_killed",
     (getter)silkPyRWRec_timeout_killed_get,
     (setter)silkPyRWRec_timeout_killed_set,
     "flow ended prematurely due to timeout by the collector", NULL},
    {"timeout_started",
     (getter)silkPyRWRec_timeout_started_get,
     (setter)silkPyRWRec_timeout_started_set,
     "flow is a continuation of a flow timed-out by the collector", NULL},
    {"typename",
     (getter)silkPyRWRec_typename_get,    NULL,
     "type name (read-only)", NULL},
    {"uniform_packets",
     (getter)silkPyRWRec_uniform_packets_get,
     (setter)silkPyRWRec_uniform_packets_set,
     "flow contained only packets of uniform size", NULL},
    {NULL, NULL, NULL, NULL, NULL}
};

/* define the object types */
static PyTypeObject silkPyRawRWRecType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.RWRawRec",     /* tp_name */
    sizeof(silkPyRawRWRec),     /* tp_basicsize */
    0,                          /* tp_itemsize */
    obj_dealloc,                /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Raw RW Record",            /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyRawRWRec_init, /* tp_init */
    0,                          /* tp_alloc */
    silkPyRawRWRec_new,         /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

static PyTypeObject silkPyRWRecType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.RWRecBase",    /* tp_name */
    sizeof(silkPyRWRec),        /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyRWRec_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
#if PY_MAJOR_VERSION < 3
    Py_TPFLAGS_HAVE_RICHCOMPARE |
#endif
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Base RW Record",           /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    (richcmpfunc)silkPyRWRec_richcompare, /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    silkPyRWRec_methods,        /* tp_methods */
    0,                          /* tp_members */
    silkPyRWRec_getseters,      /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyRWRec_init, /* tp_init */
    0,                          /* tp_alloc */
    silkPyRWRec_new,            /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyRawRWRec_Check(op)                \
    PyObject_TypeCheck(op, &silkPyRawRWRecType)
#define silkPyRWRec_Check(op)                   \
    PyObject_TypeCheck(op, &silkPyRWRecType)

static PyObject *silkPyRWRec_application_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetApplication(&obj->raw->rec));
}

static int silkPyRWRec_application_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The application value must be a 16-bit integer");
        return -1;
    }

    rwRecSetApplication(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_bytes_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyLong_FromUnsignedLong(rwRecGetBytes(&obj->raw->rec));
}

static int silkPyRWRec_bytes_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    unsigned long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsUnsignedLong(value);
    if (PyErr_Occurred() || val > UINT32_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The bytes value must be a 32-bit integer");
        return -1;
    }

    rwRecSetBytes(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_classname_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    char          class_name[SK_MAX_STRLEN_FLOWTYPE+1];
    flowtypeID_t  flowtype = rwRecGetFlowType(&obj->raw->rec);

    CHECK_SITE(NULL);

    sksiteFlowtypeGetClass(class_name, sizeof(class_name), flowtype);

    return PyUnicode_InternFromString(class_name);
}

static PyObject *silkPyRWRec_classtype_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    char          class_name[SK_MAX_STRLEN_FLOWTYPE+1];
    char          type_name[SK_MAX_STRLEN_FLOWTYPE+1];
    flowtypeID_t  flowtype = rwRecGetFlowType(&obj->raw->rec);
    PyObject     *pair     = PyTuple_New(2);

    if (pair == NULL) {
        return NULL;
    }

    CHECK_SITE(NULL);

    sksiteFlowtypeGetClass(class_name, sizeof(class_name), flowtype);
    sksiteFlowtypeGetType(type_name, sizeof(type_name), flowtype);

    PyTuple_SET_ITEM(pair, 0, PyUnicode_InternFromString(class_name));
    PyTuple_SET_ITEM(pair, 1, PyUnicode_InternFromString(type_name));

    return pair;
}

static PyObject *silkPyRWRec_classtype_id_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetFlowType(&obj->raw->rec));
}

static int silkPyRWRec_classtype_id_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The classtype_id value must be an 8-bit integer");
        return -1;
    }

    rwRecSetFlowType(&obj->raw->rec, val);
    return 0;
}

static int silkPyRWRec_classtype_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    char *class_name, *type_name;
    flowtypeID_t flowtype;


    if (!PyArg_ParseTuple(value, "ss", &class_name, &type_name)) {
        return -1;
    }

    CHECK_SITE(-1);

    flowtype = sksiteFlowtypeLookupByClassType(class_name, type_name);

    if (flowtype == SK_INVALID_FLOWTYPE) {
        PyErr_SetString(PyExc_ValueError, "Invalid (class_name, type) pair");
        return -1;
    }

    rwRecSetFlowType(&obj->raw->rec, flowtype);
    return 0;
}

static void silkPyRWRec_dealloc(
    silkPyRWRec    *obj)
{
    Py_XDECREF((PyObject*)obj->raw);
    Py_TYPE(obj)->tp_free(obj);
}

static PyObject *silkPyRWRec_dip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    silkPyIPAddr *addr;
    PyTypeObject *type;

    if (rwRecIsIPv6(&obj->raw->rec)) {
        type = &silkPyIPv6AddrType;
    } else {
        type = &silkPyIPv4AddrType;
    }

    addr = PyObject_New(silkPyIPAddr, type);
    if (addr != NULL) {
        rwRecMemGetDIP(&obj->raw->rec, &addr->addr);
    }

    return (PyObject*)addr;
}

static int silkPyRWRec_dip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    if (IS_STRING(value)) {
        skipaddr_t  addr;
        PyObject   *bytes;
        char       *repr;
        int         rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv = skStringParseIP(&addr, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal IP address: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecMemSetDIP(&obj->raw->rec, &addr);
        return 0;
    }

    if (silkPyIPAddr_Check(value)) {
        silkPyIPAddr *addr = (silkPyIPAddr*)value;
        rwRecMemSetDIP(&obj->raw->rec, &addr->addr);
        return 0;
    }

    PyErr_SetString(PyExc_TypeError, "The dip must be a valid IP address");
    return -1;
}

static PyObject *silkPyRWRec_dport_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetDPort(&obj->raw->rec));
}

static int silkPyRWRec_dport_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The dport value must be a 16-bit integer");
        return -1;
    }

    rwRecSetDPort(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_duration_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyObject_CallFunction(GLOBALS->timedelta, "IIII", 0, 0, 0,
                                 rwRecGetElapsed(&obj->raw->rec));
}

static PyObject *silkPyRWRec_duration_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    double elapsed = ((double)rwRecGetElapsed(&obj->raw->rec)) / 1.0e3;

    return PyFloat_FromDouble(elapsed);
}

static int silkPyRWRec_duration_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    const char *errstr = "The duration_secs value must be a positive number";
    PyObject *pyfloat_val;
    PyObject *pyint_val;
    int64_t long_val;

    if (!PyNumber_Check(value)) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }

    pyfloat_val = PyNumber_Multiply(value, GLOBALS->thousand);
    if (pyfloat_val == NULL) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }
    pyint_val = PyNumber_Long(pyfloat_val);
    Py_DECREF(pyfloat_val);
    if (pyint_val == NULL) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }
    long_val = PyLong_AsLongLong(pyint_val);
    Py_DECREF(pyint_val);
    if (long_val < 0) {
        PyErr_SetString(PyExc_OverflowError, errstr);
        return -1;
    }

    if (long_val > UINT32_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The duration_secs value must be less than "
                        "4294967.295");
        return -1;
    }

    rwRecSetElapsed(&obj->raw->rec, long_val);
    return 0;
}

static int silkPyRWRec_duration_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    PyObject *days;
    PyObject *secs;
    PyObject *usecs;
    uint32_t millisecs;

    if (!PyDelta_Check(value)) {
        PyErr_SetString(PyExc_TypeError,
                        "The duration value must be a datetime.timedelta");
        return -1;
    }
    if (PyObject_RichCompareBool(value, GLOBALS->minelapsed, Py_LT) ||
        PyObject_RichCompareBool(value, GLOBALS->maxelapsed, Py_GT))
    {
        PyErr_SetString(PyExc_OverflowError,
                        ("The duration must be in the range [0,4294967295] "
                         "milliseconds"));
        return -1;
    }
    days = PyObject_GetAttrString(value, "days");
    secs = PyObject_GetAttrString(value, "seconds");
    usecs = PyObject_GetAttrString(value, "microseconds");
    millisecs = PyLong_AsLong(days) * 1000 * 3600 * 24 +
                PyLong_AsLong(secs) * 1000 +
                PyLong_AsLong(usecs) / 1000;
    Py_DECREF(secs);
    Py_DECREF(usecs);

    rwRecSetElapsed(&obj->raw->rec, millisecs);
    return 0;
}

static PyObject *silkPyRWRec_etime_epoch_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    double duration = (double)(rwRecGetStartTime(&obj->raw->rec) +
                               rwRecGetElapsed(&obj->raw->rec)) / 1.0e3;

    return PyFloat_FromDouble(duration);
}

static int silkPyRWRec_etime_epoch_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    PyObject *s_time, *dur;
    int       retval;

    s_time = silkPyRWRec_stime_epoch_secs_get(obj, NULL);
    if (s_time == NULL) {
        return -1;
    }

    dur = PyNumber_Subtract(value, s_time);
    Py_DECREF(s_time);
    if (dur == NULL) {
        return -1;
    }

    retval = silkPyRWRec_duration_secs_set(obj, dur, NULL);
    Py_DECREF(dur);

    return retval;
}

static PyObject *silkPyRWRec_etime_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    PyObject *s_time, *dur;
    PyObject *retval;

    s_time = silkPyRWRec_stime_get(obj, NULL);
    if (s_time == NULL) {
        return NULL;
    }
    dur = silkPyRWRec_duration_get(obj, NULL);
    if (dur == NULL) {
        Py_DECREF(s_time);
        return NULL;
    }

    retval = PyNumber_Add(s_time, dur);

    Py_DECREF(s_time);

    return retval;
}

static int silkPyRWRec_etime_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    PyObject *s_time, *dur;
    int       retval;

    s_time = silkPyRWRec_stime_get(obj, NULL);
    if (s_time == NULL) {
        return -1;
    }

    dur = PyNumber_Subtract(value, s_time);
    Py_DECREF(s_time);
    if (dur == NULL) {
        return -1;
    }

    retval = silkPyRWRec_duration_set(obj, dur, NULL);
    Py_DECREF(dur);

    return retval;
}

static PyObject *silkPyRWRec_finnoack_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    uint8_t state;

    state = rwRecGetTcpState(&obj->raw->rec);
    if (!(state & SK_TCPSTATE_EXPANDED)) {
        Py_RETURN_NONE;
    }
    return PyBool_FromLong(state & SK_TCPSTATE_FIN_FOLLOWED_NOT_ACK);
}

static int silkPyRWRec_finnoack_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    int     rv;
    uint8_t state;

    rv = PyObject_IsTrue(value);
    if (rv == -1) {
        return -1;
    }
    state = rwRecGetTcpState(&obj->raw->rec) | SK_TCPSTATE_EXPANDED;
    if (rv) {
        state |= SK_TCPSTATE_FIN_FOLLOWED_NOT_ACK;
    } else {
        state &= ~SK_TCPSTATE_FIN_FOLLOWED_NOT_ACK;
    }
    rwRecSetTcpState(&obj->raw->rec, state);

    return 0;
}

static PyObject *silkPyRWRec_icmpcode_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetIcmpCode(&obj->raw->rec));
}

static int silkPyRWRec_icmpcode_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The icmpcode value must be a 8-bit integer");
        return -1;
    }

    rwRecSetIcmpCode(&obj->raw->rec, (uint8_t)val);
    return 0;
}

static PyObject *silkPyRWRec_icmptype_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetIcmpType(&obj->raw->rec));
}

static int silkPyRWRec_icmptype_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The icmptype value must be a 8-bit integer");
        return -1;
    }

    rwRecSetIcmpType(&obj->raw->rec, (uint8_t)val);
    return 0;
}

static PyObject *silkPyRWRec_new(
    PyTypeObject    *type,
    PyObject UNUSED(*args),
    PyObject UNUSED(*kwds))
{
    silkPyRWRec *self;
    PyObject *newrawrec = GLOBALS->newrawrec;

    self = (silkPyRWRec *)type->tp_alloc(type, 0);
    if (self != NULL) {
        self->raw = (silkPyRawRWRec *)newrawrec;
        Py_INCREF(newrawrec);
    }
    return (PyObject *)self;
}

static int silkPyRWRec_init(
    silkPyRWRec    *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"clone", "copy", NULL};
    silkPyRawRWRec *clne = NULL;
    silkPyRWRec    *copy  = NULL;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O!O!", kwlist,
                                     &silkPyRawRWRecType, (PyObject **)&clne,
                                     &silkPyRWRecType, (PyObject **)&copy))
    {
        return -1;
    }

    if (clne && copy) {
        PyErr_SetString(PyExc_RuntimeError, "Cannot clone and copy");
        return -1;
    }

    Py_XDECREF((PyObject*)self->raw);
    if (clne) {
        Py_INCREF(clne);
        self->raw = clne;
    } else if (copy) {
        self->raw = (silkPyRawRWRec*)PyObject_CallFunctionObjArgs(
            (PyObject*)&silkPyRawRWRecType, copy->raw, NULL);
    } else {
        self->raw = (silkPyRawRWRec*)PyObject_CallFunctionObjArgs(
            (PyObject*)&silkPyRawRWRecType, NULL);
    }

    if (self->raw == NULL) {
        return -1;
    }

    return 0;
}

static PyObject *silkPyRWRec_initial_tcpflags_get(
    silkPyRWRec    *obj,
    PyObject       *deprecated)
{
    silkPyTCPFlags *flags;

    if (deprecated == Py_True) {
        int rv = PyErr_Warn(PyExc_DeprecationWarning,
                            ("'initflags' is deprecated in favor of"
                             " 'initial_tcpflags'."));
        if (rv) {
            return NULL;
        }
    }

    if (!(rwRecGetTcpState(&obj->raw->rec) & SK_TCPSTATE_EXPANDED)) {
        Py_RETURN_NONE;
    }
    flags = (silkPyTCPFlags*)silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType,
                                                          0);
    if (flags != NULL) {
        flags->val = rwRecGetInitFlags(&obj->raw->rec);
    }

    return (PyObject*)flags;
}

static int silkPyRWRec_initial_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    PyObject       *deprecated)
{
    long    val;
    uint8_t state;

    if (deprecated == Py_True) {
        int rv = PyErr_Warn(PyExc_DeprecationWarning,
                            ("'initflags' is deprecated in favor of"
                             " 'initial_tcpflags'."));
        if (rv) {
            return -1;
        }
    }

    state = rwRecGetTcpState(&obj->raw->rec) | SK_TCPSTATE_EXPANDED;

    if (IS_STRING(value)) {
        uint8_t  flagval;
        PyObject *bytes;
        char    *repr;
        int      rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv = skStringParseTCPFlags(&flagval, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal TCP flags: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecSetTcpState(&obj->raw->rec, state);
        rwRecSetInitFlags(&obj->raw->rec, flagval);
        return 0;
    }

    if (silkPyTCPFlags_Check(value)) {
        silkPyTCPFlags *flags = (silkPyTCPFlags*)value;
        rwRecSetTcpState(&obj->raw->rec, state);
        rwRecSetInitFlags(&obj->raw->rec, flags->val);
        return 0;
    }

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        obj_error("Illegal TCP flags value: %s", value);
        return -1;
    }

    rwRecSetTcpState(&obj->raw->rec, state);
    rwRecSetInitFlags(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_input_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return  PyInt_FromLong(rwRecGetInput(&obj->raw->rec));
}

static int silkPyRWRec_input_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The input value must be a 16-bit integer");
        return -1;
    }

    rwRecSetInput(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_is_web(
    silkPyRWRec    *obj)
{
    uint16_t sport = rwRecGetSPort(&obj->raw->rec);
    uint16_t dport = rwRecGetDPort(&obj->raw->rec);
    return PyBool_FromLong(
        rwRecGetProto(&obj->raw->rec) == 6 &&
        (sport == 80 || dport == 80 ||
         sport == 443 || dport == 443 ||
         sport == 8080 || dport == 8080));
}

static PyObject *silkPyRWRec_nhip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    silkPyIPAddr *addr;
    PyTypeObject *type;

    if (rwRecIsIPv6(&obj->raw->rec)) {
        type = &silkPyIPv6AddrType;
    } else {
        type = &silkPyIPv4AddrType;
    }

    addr = PyObject_New(silkPyIPAddr, type);
    if (addr != NULL) {
        rwRecMemGetNhIP(&obj->raw->rec, &addr->addr);
    }
    return (PyObject*)addr;
}

static int silkPyRWRec_nhip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    if (IS_STRING(value)) {
        skipaddr_t  addr;
        PyObject   *bytes;
        char       *repr;
        int         rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv   = skStringParseIP(&addr, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal IP address: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecMemSetNhIP(&obj->raw->rec, &addr);
        return 0;
    }

    if (silkPyIPAddr_Check(value)) {
        silkPyIPAddr *addr = (silkPyIPAddr*)value;
        rwRecMemSetNhIP(&obj->raw->rec, &addr->addr);
        return 0;
    }

    PyErr_SetString(PyExc_TypeError, "The nhip must be a valid IP address");
    return -1;
}

static PyObject *silkPyRWRec_output_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetOutput(&obj->raw->rec));
}

static int silkPyRWRec_output_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The output value must be a 16-bit integer");
        return -1;
    }

    rwRecSetOutput(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_packets_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyLong_FromUnsignedLong(rwRecGetPkts(&obj->raw->rec));
}

static int silkPyRWRec_packets_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    unsigned long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsUnsignedLong(value);
    if (PyErr_Occurred() || val > UINT32_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The packets value must be a 32-bit integer");
        return -1;
    }

    rwRecSetPkts(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_protocol_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetProto(&obj->raw->rec));
}

static int silkPyRWRec_protocol_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The protocol value must be an 8-bit integer");
        return -1;
    }

    rwRecSetProto(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_richcompare(
    silkPyRWRec    *self,
    PyObject       *obj,
    int             cmp)
{
    int rv;

    if ((cmp != Py_EQ && cmp != Py_NE) ||
        !silkPyRWRec_Check(obj))
    {
        Py_INCREF(Py_NotImplemented);
        return Py_NotImplemented;
    }

    rv = memcmp(&self->raw->rec, &((silkPyRWRec*)obj)->raw->rec,
                sizeof(self->raw->rec));
    rv = (rv == 0) ? 1 : 0;
    if (cmp == Py_NE) {
        rv = !rv;
    }

    return PyBool_FromLong(rv);
}

static PyObject *silkPyRWRec_sensor_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    char name[SK_MAX_STRLEN_SENSOR+1];

    CHECK_SITE(NULL);

    sksiteSensorGetName(name, sizeof(name), rwRecGetSensor(&obj->raw->rec));
    return PyUnicode_InternFromString(name);
}

static PyObject *silkPyRWRec_sensor_id_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetSensor(&obj->raw->rec));
}

static int silkPyRWRec_sensor_id_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The sensor_id value must be a 16-bit integer");
        return -1;
    }

    rwRecSetSensor(&obj->raw->rec, val);
    return 0;
}

static int silkPyRWRec_sensor_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    char *repr;
    sensorID_t sensor;
    PyObject *bytes;

    bytes = bytes_from_string(value);
    if (bytes == NULL) {
        PyErr_SetString(PyExc_TypeError, "The sensor value must be a string");
        return -1;
    }
    repr = PyBytes_AS_STRING(bytes);

    if (init_site(NULL)) {
        Py_DECREF(bytes);
        return -1;
    }

    sensor = sksiteSensorLookup(repr);
    Py_DECREF(bytes);
    if (sensor == SK_INVALID_SENSOR) {
        PyErr_SetString(PyExc_ValueError, "Invalid sensor name");
        return -1;
    }

    rwRecSetSensor(&obj->raw->rec, sensor);
    return 0;
}

static PyObject *silkPyRWRec_session_tcpflags_get(
    silkPyRWRec    *obj,
    PyObject       *deprecated)
{
    silkPyTCPFlags *flags;

    if (deprecated == Py_True) {
        int rv = PyErr_Warn(PyExc_DeprecationWarning,
                            ("'restflags' is deprecated in favor of"
                             " 'session_tcpflags'."));
        if (rv) {
            return NULL;
        }
    }

    if (!(rwRecGetTcpState(&obj->raw->rec) & SK_TCPSTATE_EXPANDED)) {
        Py_RETURN_NONE;
    }
    flags = (silkPyTCPFlags*)silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType,
                                                          0);
    if (flags != NULL) {
        flags->val = rwRecGetRestFlags(&obj->raw->rec);
    }

    return (PyObject*)flags;
}

static int silkPyRWRec_session_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    PyObject       *deprecated)
{
    long    val;
    uint8_t state;

    if (deprecated == Py_True) {
        int rv = PyErr_Warn(PyExc_DeprecationWarning,
                            ("'restflags' is deprecated in favor of"
                             " 'session_tcpflags'."));
        if (rv) {
            return -1;
        }
    }

    state = rwRecGetTcpState(&obj->raw->rec) | SK_TCPSTATE_EXPANDED;

    if (IS_STRING(value)) {
        uint8_t     flagval;
        PyObject   *bytes;
        char       *repr;
        int         rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv   = skStringParseTCPFlags(&flagval, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal TCP flags: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecSetTcpState(&obj->raw->rec, state);
        rwRecSetRestFlags(&obj->raw->rec, flagval);
        return 0;
    }

    if (silkPyTCPFlags_Check(value)) {
        silkPyTCPFlags *flags = (silkPyTCPFlags*)value;
        rwRecSetTcpState(&obj->raw->rec, state);
        rwRecSetRestFlags(&obj->raw->rec, flags->val);
        return 0;
    }

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        obj_error("Illegal TCP flags value: %s", value);
        return -1;
    }

    rwRecSetTcpState(&obj->raw->rec, state);
    rwRecSetRestFlags(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_sip_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    silkPyIPAddr *addr;
    PyTypeObject *type;

    if (rwRecIsIPv6(&obj->raw->rec)) {
        type = &silkPyIPv6AddrType;
    } else {
        type = &silkPyIPv4AddrType;
    }

    addr = PyObject_New(silkPyIPAddr, type);
    if (addr != NULL) {
        rwRecMemGetSIP(&obj->raw->rec, &addr->addr);
    }
    return (PyObject*)addr;
}

static int silkPyRWRec_sip_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    if (IS_STRING(value)) {
        skipaddr_t  addr;
        PyObject   *bytes;
        char       *repr;
        int         rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv   = skStringParseIP(&addr, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal IP address: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecMemSetSIP(&obj->raw->rec, &addr);
        return 0;
    }

    if (silkPyIPAddr_Check(value)) {
        silkPyIPAddr *addr = (silkPyIPAddr*)value;
        rwRecMemSetSIP(&obj->raw->rec, &addr->addr);
        return 0;
    }

    PyErr_SetString(PyExc_TypeError, "The sip must be a valid IP address");
    return -1;
}

static PyObject *silkPyRWRec_sport_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(rwRecGetSPort(&obj->raw->rec));
}

static int silkPyRWRec_sport_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT16_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "The sport value must be a 16-bit integer");
        return -1;
    }

    rwRecSetSPort(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_stime_epoch_secs_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    double s_time = ((double)rwRecGetStartTime(&obj->raw->rec)) / 1.0e3;

    return PyFloat_FromDouble(s_time);
}

static int silkPyRWRec_stime_epoch_secs_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    const char *errstr = ("The stime_epoch_secs value must be a "
                          "positive number");
    PyObject *pyfloat_val;
    PyObject *pyint_val;
    int64_t long_val;

    if (!PyNumber_Check(value)) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }

    pyfloat_val = PyNumber_Multiply(value, GLOBALS->thousand);
    if (pyfloat_val == NULL) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }
    pyint_val = PyNumber_Long(pyfloat_val);
    Py_DECREF(pyfloat_val);
    if (pyint_val == NULL) {
        PyErr_SetString(PyExc_TypeError, errstr);
        return -1;
    }
    long_val = PyLong_AsLongLong(pyint_val);
    Py_DECREF(pyint_val);
    if (long_val < 0) {
        PyErr_SetString(PyExc_OverflowError, errstr);
        return -1;
    }

    rwRecSetStartTime(&obj->raw->rec, long_val);
    return 0;
}

static PyObject *silkPyRWRec_stime_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    PyObject *seconds, *millisecs, *final;
    imaxdiv_t d = imaxdiv(rwRecGetStartTime(&obj->raw->rec), 1000);

    seconds = PyObject_CallMethod(GLOBALS->datetime,
                                  "utcfromtimestamp", "L", d.quot);
    millisecs = PyObject_CallFunction(GLOBALS->timedelta,
                                      "IIIL", 0, 0, 0, d.rem);
    final = PyNumber_Add(seconds, millisecs);
    Py_DECREF(seconds);
    Py_DECREF(millisecs);
    return final;
}

static int silkPyRWRec_stime_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    sktime_t t;
    int rv;

    rv = silkPyDatetimeToSktime(&t, value);

    if (rv == 0) {
        rwRecSetStartTime(&obj->raw->rec, t);
    }
    return rv;
}

static PyObject *silkPyRWRec_tcpflags_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    silkPyTCPFlags *flags;

    flags = (silkPyTCPFlags*)silkPyTCPFlagsType.tp_alloc(&silkPyTCPFlagsType,
                                                          0);
    if (flags != NULL) {
        flags->val = rwRecGetFlags(&obj->raw->rec);
    }

    return (PyObject*)flags;
}

static int silkPyRWRec_tcpflags_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    long val;

    if (IS_STRING(value)) {
        uint8_t   flagval;
        PyObject *bytes;
        char     *repr;
        int       rv;

        bytes = bytes_from_string(value);
        if (bytes == NULL) {
            return -1;
        }
        repr = PyBytes_AS_STRING(bytes);
        rv = skStringParseTCPFlags(&flagval, repr);
        if (rv != 0) {
            PyErr_Format(PyExc_ValueError, "Illegal TCP flags: %s", repr);
            Py_DECREF(bytes);
            return -1;
        }
        Py_DECREF(bytes);
        rwRecSetFlags(&obj->raw->rec, flagval);
        return 0;
    }

    if (silkPyTCPFlags_Check(value)) {
        silkPyTCPFlags *flags = (silkPyTCPFlags*)value;
        rwRecSetFlags(&obj->raw->rec, flags->val);
        return 0;
    }

    if (!IS_INT(value)) {
        PyErr_SetString(PyExc_TypeError, "Expected an integer");
        return -1;
    }

    val = PyLong_AsLong(value);
    if (PyErr_Occurred() || val < 0 || val > (long)UINT8_MAX) {
        obj_error("Illegal TCP flags value: %s", value);
        return -1;
    }

    rwRecSetFlags(&obj->raw->rec, val);
    return 0;
}

static PyObject *silkPyRWRec_timeout_killed_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    uint8_t state;

    state = rwRecGetTcpState(&obj->raw->rec);
    if (!(state & SK_TCPSTATE_EXPANDED)) {
        Py_RETURN_NONE;
    }
    return PyBool_FromLong(state & SK_TCPSTATE_TIMEOUT_KILLED);
}

static int silkPyRWRec_timeout_killed_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    int     rv;
    uint8_t state;

    rv = PyObject_IsTrue(value);
    if (rv == -1) {
        return -1;
    }
    state = rwRecGetTcpState(&obj->raw->rec) | SK_TCPSTATE_EXPANDED;
    if (rv) {
        state |= SK_TCPSTATE_TIMEOUT_KILLED;
    } else {
        state &= ~SK_TCPSTATE_TIMEOUT_KILLED;
    }
    rwRecSetTcpState(&obj->raw->rec, state);

    return 0;
}

static PyObject *silkPyRWRec_timeout_started_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    uint8_t state;

    state = rwRecGetTcpState(&obj->raw->rec);
    if (!(state & SK_TCPSTATE_EXPANDED)) {
        Py_RETURN_NONE;
    }
    return PyBool_FromLong(state & SK_TCPSTATE_TIMEOUT_STARTED);
}

static int silkPyRWRec_timeout_started_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    int     rv;
    uint8_t state;

    rv = PyObject_IsTrue(value);
    if (rv == -1) {
        return -1;
    }
    state = rwRecGetTcpState(&obj->raw->rec) | SK_TCPSTATE_EXPANDED;
    if (rv) {
        state |= SK_TCPSTATE_TIMEOUT_STARTED;
    } else {
        state &= ~SK_TCPSTATE_TIMEOUT_STARTED;
    }
    rwRecSetTcpState(&obj->raw->rec, state);

    return 0;
}

static PyObject *silkPyRWRec_typename_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    char          type_name[SK_MAX_STRLEN_FLOWTYPE+1];
    flowtypeID_t  flowtype = rwRecGetFlowType(&obj->raw->rec);

    CHECK_SITE(NULL);

    sksiteFlowtypeGetType(type_name, sizeof(type_name), flowtype);
    return PyUnicode_InternFromString(type_name);
}

static PyObject *silkPyRWRec_uniform_packets_get(
    silkPyRWRec    *obj,
    void    UNUSED(*closure))
{
    uint8_t state;

    state = rwRecGetTcpState(&obj->raw->rec);
    return PyBool_FromLong(state & SK_TCPSTATE_UNIFORM_PACKET_SIZE);
}

static int silkPyRWRec_uniform_packets_set(
    silkPyRWRec    *obj,
    PyObject       *value,
    void    UNUSED(*closure))
{
    int     rv;
    uint8_t state;

    rv = PyObject_IsTrue(value);
    if (rv == -1) {
        return -1;
    }
    state = rwRecGetTcpState(&obj->raw->rec);
    if (rv) {
        state |= SK_TCPSTATE_UNIFORM_PACKET_SIZE;
    } else {
        state &= ~SK_TCPSTATE_UNIFORM_PACKET_SIZE;
    }
    rwRecSetTcpState(&obj->raw->rec, state);

    return 0;
}

static int silkPyRawRWRec_init(
    silkPyRawRWRec *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"rec", NULL};
    PyObject *copy = NULL;

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O!", kwlist,
                                     &silkPyRawRWRecType, &copy))
    {
        return -1;
    }

    if (copy) {
        RWREC_COPY(&self->rec, &((silkPyRawRWRec*)copy)->rec);
    }
    return 0;
}

static PyObject *silkPyRawRWRec_new(
    PyTypeObject       *type,
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds))
{
    silkPyRawRWRec *self;

    self = (silkPyRawRWRec*)type->tp_alloc(type, 0);

    if (self != NULL) {
        RWREC_CLEAR(&self->rec);
    }

    return (PyObject*)self;
}


/*
 *************************************************************************
 *   SiLK File
 *************************************************************************
 */

typedef struct silkPySilkFile_st {
    PyObject_HEAD
    skstream_t *io;
} silkPySilkFile;

/* function prototypes */
static PyObject *silkPySilkFile_close(
    silkPySilkFile *obj);
static void silkPySilkFile_dealloc(
    silkPySilkFile *obj);
static PyObject *silkPySilkFile_get_mode(
    silkPySilkFile *obj,
    void    UNUSED(*closure));
static PyObject *silkPySilkFile_get_name(
    silkPySilkFile *obj,
    void    UNUSED(*closure));
static int silkPySilkFile_init(
    silkPySilkFile *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPySilkFile_invocations(
    silkPySilkFile *obj);
static PyObject *silkPySilkFile_notes(
    silkPySilkFile *obj);
static PyObject *silkPySilkFile_read(
    silkPySilkFile *obj);
static PyObject *silkPySilkFile_write(
    silkPySilkFile *obj,
    PyObject       *rec);

/* define docs and methods */
static PyMethodDef silkPySilkFile_methods[] = {
    {"__reduce__", (PyCFunction)reduce_error, METH_NOARGS, ""},
    {"read", (PyCFunction)silkPySilkFile_read, METH_NOARGS,
     "Read a RWRec from a RW File"},
    {"write", (PyCFunction)silkPySilkFile_write, METH_O,
     "Write a RWRec to a RW File"},
    {"close", (PyCFunction)silkPySilkFile_close, METH_NOARGS,
     "Read a RWRec from a RW File"},
    {"notes", (PyCFunction)silkPySilkFile_notes, METH_NOARGS,
     "Get the file's annotations"},
    {"invocations", (PyCFunction)silkPySilkFile_invocations, METH_NOARGS,
     "Get the file's invocations"},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PyGetSetDef silkPySilkFile_getseters[] = {
    {"name", (getter)silkPySilkFile_get_name, NULL,
     "name of file associated with SilkFile", NULL},
    {"mode", (getter)silkPySilkFile_get_mode, NULL,
     "mode associated with SilkFile", NULL},
    {NULL, NULL, NULL, NULL, NULL}    /* Sentinel */
};

/* define the object types */
static PyTypeObject silkPySilkFileType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.SilkFileBase", /* tp_name */
    sizeof(silkPySilkFile),         /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPySilkFile_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Base Silk File",           /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    0,                          /* tp_iter */
    0,                          /* tp_iternext */
    silkPySilkFile_methods,     /* tp_methods */
    0,                          /* tp_members */
    silkPySilkFile_getseters,   /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPySilkFile_init, /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPySilkFile_Check(op) \
    PyObject_TypeCheck(op, &silkPySilkFileType)

static PyObject *throw_ioerror(
    silkPySilkFile *obj,
    int             errcode)
{
    skStreamPrintLastErr(obj->io, errcode, error_printf);
    PyErr_SetString(PyExc_IOError, error_buffer);
    return NULL;
}

static PyObject *silkPySilkFile_close(
    silkPySilkFile *obj)
{
    int rv;

    rv = skStreamClose(obj->io);
    if (rv == 0) {
        Py_RETURN_NONE;
    }

    return throw_ioerror(obj, rv);
}

static void silkPySilkFile_dealloc(
    silkPySilkFile *obj)
{
    if (obj->io) {
        skStreamDestroy(&obj->io);
    }
    Py_TYPE(obj)->tp_free(obj);
}

static PyObject *silkPySilkFile_get_mode(
    silkPySilkFile *obj,
    void    UNUSED(*closure))
{
    return PyInt_FromLong(skStreamGetMode(obj->io));
}

static PyObject *silkPySilkFile_get_name(
    silkPySilkFile *obj,
    void    UNUSED(*closure))
{
    const char *name = skStreamGetPathname(obj->io);
    if (name) {
        return PyUnicode_FromString(name);
    }
    Py_RETURN_NONE;
}

static int silkPySilkFile_init(
    silkPySilkFile *self,
    PyObject       *args,
    PyObject       *kwds)
{
    char             *filename;
    int               mode;
    int               format      = NOT_SET;
    int               policy      = NOT_SET;
    int               compr       = NOT_SET;
    int               file_des    = NOT_SET;
    PyObject         *annotations = NULL;
    PyObject         *invocations = NULL;
    sk_file_header_t *hdr;
    int               rv;

    static char *kwlist[] = {"filename", "mode", "compression",
                             "format", "policy", "invocations",
                             "notes", "_fileno", NULL};

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "si|iiiO!O!i", kwlist,
                                     &filename, &mode, &compr,
                                     &format, &policy,
                                     &PyList_Type, &invocations,
                                     &PyList_Type, &annotations,
                                     &file_des))
    {
        return -1;
    }

    if (mode != SK_IO_READ && mode != SK_IO_WRITE && mode != SK_IO_APPEND) {
        PyErr_SetString(PyExc_ValueError, "Illegal mode");
        Py_DECREF(self);
        return -1;
    }
    if (self->io) {
        skStreamDestroy(&self->io);
    }
    rv = skStreamCreate(&self->io, (skstream_mode_t)mode,SK_CONTENT_SILK_FLOW);
    if (rv != 0) {
        throw_ioerror(self, rv);
        return -1;
    }

    rv = skStreamBind(self->io, filename);
    if (rv != 0) {
        throw_ioerror(self, rv);
        return -1;
    }

    hdr = skStreamGetSilkHeader(self->io);

    if (policy != NOT_SET) {
        rv = skStreamSetIPv6Policy(self->io, (sk_ipv6policy_t)policy);
        if (rv != 0) {
            throw_ioerror(self, rv);
            return -1;
        }
    }

    if (compr != NOT_SET) {
        if (mode != SK_IO_WRITE) {
            PyErr_SetString(PyExc_ValueError,
                            "Cannot set compression unless in WRITE mode");
            return -1;
        }
        rv = skHeaderSetCompressionMethod(hdr, compr);
        if (rv != 0) {
            throw_ioerror(self, rv);
            return -1;
        }
    }

    if (format != NOT_SET) {
        if (mode != SK_IO_WRITE) {
            PyErr_SetString(PyExc_ValueError,
                            "Cannot set file format unless in WRITE mode");
            return -1;
        }
        rv = skHeaderSetFileFormat(hdr, format);
        if (rv != 0) {
            throw_ioerror(self, rv);
            return -1;
        }
    }

    if (annotations != NULL) {
        if (mode != SK_IO_WRITE) {
            PyErr_SetString(PyExc_ValueError,
                            "Cannot set file format unless in WRITE mode");
            return -1;
        }
        if (hdr != NULL) {
            ssize_t len = PyList_GET_SIZE(annotations);
            ssize_t i;

            for (i = 0; i < len; i++) {
                PyObject *bytes;
                PyObject *item = PyList_GET_ITEM(annotations, i);

                if (!IS_STRING(item)) {
                    PyErr_SetString(PyExc_TypeError,
                                    "Annotation was not a string");
                    return -1;
                }
                bytes = bytes_from_string(item);
                if (bytes == NULL) {
                    return -1;
                }

                rv = skHeaderAddAnnotation(hdr, PyBytes_AS_STRING(bytes));
                Py_DECREF(bytes);
                if (rv != 0) {
                    throw_ioerror(self, rv);
                }
            }
        }
    }

    if (invocations != NULL) {
        if (mode != SK_IO_WRITE) {
            PyErr_SetString(PyExc_ValueError,
                            "Cannot set file format unless in WRITE mode");
            return -1;
        }
        if (hdr != NULL) {
            ssize_t len = PyList_GET_SIZE(invocations);
            ssize_t i;

            for (i = 0; i < len; i++) {
                PyObject *item = PyList_GET_ITEM(invocations, i);
                char     *value;
                PyObject *bytes;

                if (!IS_STRING(item)) {
                    PyErr_SetString(PyExc_TypeError,
                                    "Invocation was not a string");
                    return -1;
                }

                bytes = bytes_from_string(item);
                if (bytes == NULL) {
                    return -1;
                }
                value = PyBytes_AS_STRING(bytes);
                rv = skHeaderAddInvocation(hdr, 0, 1, &value);
                Py_DECREF(bytes);
                if (rv != 0) {
                    throw_ioerror(self, rv);
                }
            }
        }
    }

    if (file_des == NOT_SET) {
        rv = skStreamOpen(self->io);
    } else {
        rv = skStreamFDOpen(self->io, file_des);
    }
    if (rv != 0) {
        throw_ioerror(self, rv);
        return -1;
    }

    if (mode == SK_IO_WRITE) {
        rv = skStreamWriteSilkHeader(self->io);
        if (rv != 0) {
            throw_ioerror(self, rv);
            return -1;
        }
    } else {
        rv = skStreamReadSilkHeader(self->io, NULL);
        if (rv != 0) {
            throw_ioerror(self, rv);
            return -1;
        }
    }

    return 0;
}

static PyObject *silkPySilkFile_invocations(
    silkPySilkFile *obj)
{
    sk_file_header_t     *hdr;
    sk_header_entry_t    *entry;
    sk_hentry_iterator_t  iter;
    PyObject             *list;
    PyObject             *invoc;
    int                   rv;

    list = PyList_New(0);
    if (list == NULL) {
        return NULL;
    }
    hdr = skStreamGetSilkHeader(obj->io);
    if (hdr != NULL) {
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_INVOCATION_ID);
        while ((entry = skHeaderIteratorNext(&iter)) != NULL) {
            invoc = PyUnicode_FromString(
                ((sk_hentry_invocation_t*)entry)->command_line);
            if (invoc == NULL) {
                Py_DECREF(list);
                return NULL;
            }
            rv = PyList_Append(list, invoc);
            Py_DECREF(invoc);
            if (rv != 0) {
                Py_DECREF(list);
                return NULL;
            }
        }
    }

    return list;
}

static PyObject *silkPySilkFile_notes(
    silkPySilkFile *obj)
{
    sk_file_header_t     *hdr;
    sk_header_entry_t    *entry;
    sk_hentry_iterator_t  iter;
    PyObject             *list;
    PyObject             *annot;
    int                   rv;

    list = PyList_New(0);
    if (list == NULL) {
        return NULL;
    }
    hdr = skStreamGetSilkHeader(obj->io);
    if (hdr != NULL) {
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_ANNOTATION_ID);
        while ((entry = skHeaderIteratorNext(&iter)) != NULL) {
            annot = PyUnicode_FromString(
                ((sk_hentry_annotation_t*)entry)->annotation);
            if (annot == NULL) {
                Py_DECREF(list);
                return NULL;
            }
            rv = PyList_Append(list, annot);
            Py_DECREF(annot);
            if (rv != 0) {
                Py_DECREF(list);
                return NULL;
            }
        }
    }

    return list;
}

static PyObject *silkPySilkFile_read(
    silkPySilkFile *obj)
{
    PyObject *pyrec;
    int rv;

    pyrec = silkPyRawRWRecType.tp_alloc(&silkPyRawRWRecType, 0);
    if (pyrec == NULL) {
        return NULL;
    }

    rv = skStreamReadRecord(obj->io, &((silkPyRawRWRec*)pyrec)->rec);
    if (rv != 0) {
        Py_DECREF(pyrec);
        if (rv == SKSTREAM_ERR_EOF) {
            Py_RETURN_NONE;
        }
        return throw_ioerror(obj, rv);
    }

    return pyrec;
}

static PyObject *silkPySilkFile_write(
    silkPySilkFile *obj,
    PyObject       *rec)
{
    int rv;

    if (!silkPyRWRec_Check(rec)) {
        PyErr_SetString(PyExc_TypeError, "Argument not a RWRec");
        return NULL;
    }

    rv = skStreamWriteRecord(obj->io, &((silkPyRWRec*)rec)->raw->rec);
    if (rv == 0) {
        Py_RETURN_NONE;
    }

    return throw_ioerror(obj, rv);
}


/*
 *************************************************************************
 *   RepoIter
 *************************************************************************
 */

typedef struct silkPyRepoIter_st {
    PyObject_HEAD
    sksite_repo_iter_t *iter;
} silkPyRepoIter;

/* function prototypes */
static void silkPyRepoIter_dealloc(
    silkPyRepoIter *self);
static int silkPyRepoIter_init(
    silkPyRepoIter *self,
    PyObject       *args,
    PyObject       *kwds);
static PyObject *silkPyRepoIter_iternext(
    silkPyRepoIter *self);

/* define the object types */
static PyTypeObject silkPyRepoIterType ={
    PyVarObject_HEAD_INIT(NULL, 0)
    "silk.pysilk.RepoIter",    /* tp_name */
    sizeof(silkPyRepoIterType), /* tp_basicsize */
    0,                          /* tp_itemsize */
    (destructor)silkPyRepoIter_dealloc, /* tp_dealloc */
    0,                          /* tp_print */
    0,                          /* tp_getattr */
    0,                          /* tp_setattr */
    0,                          /* tp_compare */
    0,                          /* tp_repr */
    0,                          /* tp_as_number */
    0,                          /* tp_as_sequence */
    0,                          /* tp_as_mapping */
    0,                          /* tp_hash  */
    0,                          /* tp_call */
    0,                          /* tp_str */
    0,                          /* tp_getattro */
    0,                          /* tp_setattro */
    0,                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "SiLK repo file iterator object", /* tp_doc */
    0,                          /* tp_traverse */
    0,                          /* tp_clear */
    0,                          /* tp_richcompare */
    0,                          /* tp_weaklistoffset */
    iter_iter,                  /* tp_iter */
    (iternextfunc)silkPyRepoIter_iternext, /* tp_iternext */
    0,                          /* tp_methods */
    0,                          /* tp_members */
    0,                          /* tp_getset */
    0,                          /* tp_base */
    0,                          /* tp_dict */
    0,                          /* tp_descr_get */
    0,                          /* tp_descr_set */
    0,                          /* tp_dictoffset */
    (initproc)silkPyRepoIter_init, /* tp_init */
    0,                          /* tp_alloc */
    0,                          /* tp_new */
    0,                          /* tp_free */
    0,                          /* tp_is_gc */
    0,                          /* tp_bases */
    0,                          /* tp_mro */
    0,                          /* tp_cache */
    0,                          /* tp_subclasses */
    0,                          /* tp_weaklist */
    0                           /* tp_del */
#if PY_VERSION_HEX >= 0x02060000
    ,0                          /* tp_version_tag */
#endif
};

/* macro and function defintions */
#define silkPyRepoIter_Check(op)                \
    PyObject_TypeCheck(op, &silkPyRepoIterType)

static void silkPyRepoIter_dealloc(
    silkPyRepoIter *self)
{
    sksiteRepoIteratorDestroy(&self->iter);
    Py_TYPE(self)->tp_free((PyObject*)self);
}

static int silkPyRepoIter_init(
    silkPyRepoIter *self,
    PyObject       *args,
    PyObject       *kwds)
{
    static char *kwlist[] = {"start", "end", "flowtypes",
                             "sensors", "missing", NULL};
    PyObject *start     = NULL;
    PyObject *end       = NULL;
    PyObject *flowtypes = NULL;
    PyObject *sensors   = NULL;
    PyObject *missing   = NULL;
    PyObject *fast      = NULL;
    sk_vector_t *ft_vec     = NULL;
    sk_vector_t *sensor_vec = NULL;
    sktime_t starttime, endtime;
    flowtype_iter_t ft_iter;
    flowtypeID_t ft;
    uint32_t flags;
    int sprec;
    int eprec;
    int rv;

    CHECK_SITE(-1);

    ft_vec = skVectorNew(sizeof(flowtypeID_t));
    if (ft_vec == NULL) {
        PyErr_NoMemory();
        return -1;
    }

    if (!PyArg_ParseTupleAndKeywords(args, kwds, "OO|OOO", kwlist,
                                     &start, &end, &flowtypes,
                                     &sensors, &missing))
    {
        goto error;
    }

    /* Calculate starttime */
    if (PyDate_Check(start)) {
        rv = silkPyDatetimeToSktime(&starttime, start);
        if (rv != 0) {
            goto error;
        }
        sprec = PyDateTime_Check(start) ? 4 : 3;
    } else if (IS_STRING(start)) {
        PyObject *bytes = bytes_from_string(start);
        if (bytes == NULL) {
            goto error;
        }
        rv = skStringParseDatetime(&starttime, PyBytes_AS_STRING(bytes),
                                   &sprec);
        Py_DECREF(bytes);
        if (rv != 0) {
            PyErr_SetString(PyExc_ValueError, skStringParseStrerror(rv));
            goto error;
        }
    } else {
        PyErr_SetString(PyExc_TypeError,
                        "start must be a string or a "
                        "datetime.date[time] object");
        goto error;
    }
    starttime -= starttime % 3600000;

    /* Calculate endtime */
    if (end == NULL || end == Py_None) {
        end = NULL;
    } else if (PyDate_Check(end)) {
        rv = silkPyDatetimeToSktime(&endtime, end);
        if (rv != 0) {
            goto error;
        }
        eprec = PyDateTime_Check(end) ? 4 : 3;
    } else if (IS_STRING(end)) {
        PyObject *bytes = bytes_from_string(end);
        if (bytes == NULL) {
            goto error;
        }
        rv = skStringParseDatetime(&endtime, PyBytes_AS_STRING(bytes),
                                   &eprec);
        Py_DECREF(bytes);
        if (rv != 0) {
            PyErr_SetString(PyExc_ValueError, skStringParseStrerror(rv));
            goto error;
        }
    } else {
        PyErr_SetString(PyExc_TypeError,
                        "end must be a string or a "
                        "datetime.date[time] object");
        goto error;
    }

    /* End-time mashup */
    if (end) {
        if (sprec == 3) {
            /* when no starting hour given, we look at the full days,
             * regardless of the precision of the ending time; go to
             * the last hour of the ending day. */
            if (skDatetimeCeiling(&endtime, &endtime, sprec)) {
                PyErr_SetString(PyExc_ValueError,
                                "Could not determine end time");
                goto error;
            }
            endtime -= endtime % 3600000;
        } else if (eprec < 4) {
            /* starting time has an hour but ending time does not; use
             * same hour for ending time */
#if  SK_ENABLE_LOCALTIME
            time_t t;
            struct tm work_tm;
            int work_hour;

            /* get starting hour */
            t = starttime / 1000;
            localtime_r(&t, &work_tm);
            work_hour = work_tm.tm_hour;
            /* break apart end time */
            t = endtime / 1000;
            localtime_r(&t, &work_tm);
            /* set end hour to start hour and re-combine */
            work_tm.tm_hour = work_hour;
            t = mktime(&work_tm);
            endtime = sktimeCreate((t - (t % 3600)), 0);
#else
            endtime = endtime - (endtime % 86400000) + (starttime % 86400000);
#endif
        } else {
            endtime -= endtime % 3600000;
        }
    } else if (sprec >= 4) {
        /* no ending time was given and the starting time contains an
         * hour; we only look at that single hour */
        endtime = starttime;
    } else {
        /* no ending time was given and the starting time was to the
         * day; look at that entire day */
        if (skDatetimeCeiling(&endtime, &starttime, sprec)) {
            PyErr_SetString(PyExc_ValueError,
                            "Could not determine end time");
            goto error;
        }
        endtime -= endtime % 3600000;
    }

    if (starttime > endtime) {
        PyErr_SetString(PyExc_ValueError,
                        "start must be less or equal to end");
        goto error;
    }

    /* Calculate flowtypes */
    if (flowtypes == NULL || flowtypes == Py_None) {
        sksiteFlowtypeIterator(&ft_iter);
        while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
            rv = skVectorAppendValue(ft_vec, &ft);
            if (rv != 0) {
                PyErr_NoMemory();
                goto error;
            }
        }
    } else if (PySequence_Check(flowtypes)) {
        Py_ssize_t len;
        PyObject **items;
        fast = PySequence_Fast(flowtypes, "flowtypes must be a sequence");
        if (fast == NULL) {
            goto error;
        }
        len = PySequence_Fast_GET_SIZE(fast);
        items = PySequence_Fast_ITEMS(fast);
        for (/*empty*/; len; len--, items++) {
            const char *class_name, *type_name;
            if (!PyArg_ParseTuple(*items, "ss", &class_name, &type_name)) {
                goto error;
            }
            ft = sksiteFlowtypeLookupByClassType(class_name, type_name);
            if (ft == SK_INVALID_FLOWTYPE) {
                PyErr_SetString(PyExc_ValueError,
                                "Invalid (class, type) pair");
                return -1;
            }
            rv = skVectorAppendValue(ft_vec, &ft);
            if (rv != 0) {
                PyErr_NoMemory();
                goto error;
            }
        }
        Py_CLEAR(fast);
    } else {
        PyErr_SetString(PyExc_TypeError,
                        "flowtypes should be a sequence (class, type) pairs");
        goto error;
    }

    /* Calculate sensors */
    if (sensors && PySequence_Check(sensors)) {
        Py_ssize_t len;
        PyObject **items;
        sensorID_t sensor;

        sensor_vec = skVectorNew(sizeof(sensorID_t));
        if (sensor_vec == NULL) {
            PyErr_NoMemory();
            goto error;
        }

        fast = PySequence_Fast(sensors, "sensors must be a sequence");
        if (fast == NULL) {
            goto error;
        }
        len = PySequence_Fast_GET_SIZE(fast);
        items = PySequence_Fast_ITEMS(fast);
        for (/*empty*/; len; len--, items++) {
            PyObject *bytes;
            if (!IS_STRING(*items)) {
                PyErr_SetString(PyExc_TypeError,
                                "sensors must be strings");
                goto error;
            }
            bytes = bytes_from_string(*items);
            if (bytes == NULL) {
                goto error;
            }
            sensor = sksiteSensorLookup(PyBytes_AS_STRING(bytes));
            if (sensor == SK_INVALID_SENSOR) {
                PyErr_SetString(PyExc_ValueError, "Invalid sensor name");
                goto error;
            }
            rv = skVectorAppendValue(sensor_vec, &sensor);
            if (rv != 0) {
                PyErr_NoMemory();
                goto error;
            }
        }
        Py_CLEAR(fast);
    } else if (sensors != NULL && sensors != Py_None) {
        PyErr_SetString(PyExc_TypeError,
                        "sensors should be a sequence of strings");
        goto error;
    }

    if (missing && PyObject_IsTrue(missing)) {
        flags = RETURN_MISSING;
    } else {
        flags = 0;
    }

    rv = sksiteRepoIteratorCreate(
        &self->iter, ft_vec, sensor_vec, starttime, endtime, flags);
    if (rv != 0) {
        PyErr_NoMemory();
        goto error;
    }

    rv = 0;

  cleanup:
    Py_XDECREF(fast);
    if (ft_vec) {
        skVectorDestroy(ft_vec);
    }
    if (sensor_vec) {
        skVectorDestroy(sensor_vec);
    }

    return rv;

  error:
    rv = -1;
    goto cleanup;
}

static PyObject *silkPyRepoIter_iternext(
    silkPyRepoIter *self)
{
    char path[PATH_MAX];
    int missing;
    int rv;

    rv = sksiteRepoIteratorNextPath(self->iter, path, sizeof(path), &missing);
    if (rv == SK_ITERATOR_NO_MORE_ENTRIES) {
        PyErr_SetNone(PyExc_StopIteration);
        return NULL;
    }

    return Py_BuildValue("sO", path, missing ? Py_False : Py_True);
}


/*
 *************************************************************************
 *   Misc Globals
 *************************************************************************
 */

/* function prototypes */
static PyObject *silk_class_info(void);
static PyObject *silk_flowtype_info(void);
static PyObject *silk_get_rootdir(void);
static PyObject *silk_get_siteconf(void);
static PyObject *silk_have_site_config(void);
static PyObject *silk_init_country_codes(
    PyObject    UNUSED(*self),
    PyObject           *args);
static int silk_init_set_envvar(
    const char *value,
    const char *envvar);
static PyObject *silk_init_site(
    PyObject    UNUSED(*self),
    PyObject           *args,
    PyObject           *kwds);
static PyObject *silk_initial_tcpflags_enabled(void);
static PyObject *silk_ipset_supports_ipv6(void);
static PyObject *silk_ipv6_enabled(void);
static silkPyRawRWRec *silk_raw_rwrec_copy(
    PyObject    UNUSED(*self),
    PyObject           *c_rec);
static PyObject *silk_sensor_info(void);
static PyObject *silk_set_rootdir(
    PyObject    UNUSED(*self),
    PyObject           *args);
static PyObject *silk_version(void);

static PyMethodDef silk_methods[] = {
    {"ipv6_enabled", (PyCFunction)silk_ipv6_enabled,
     METH_NOARGS,
     ("Return whether IPv6 was enabled at compile-time")},
    {"ipset_supports_ipv6", (PyCFunction)silk_ipset_supports_ipv6,
     METH_NOARGS,
     ("Return whether IPv6-support for IPsets was enabled at compile-time")},
    {"initial_tcpflags_enabled", (PyCFunction)silk_initial_tcpflags_enabled,
     METH_NOARGS,
     ("Return whether initial tcpflagewere enabled at compile-time")},
    {"init_site", (PyCFunction)silk_init_site,
     METH_VARARGS | METH_KEYWORDS,
     ("init_site([siteconf][, rootdir])\n"
      "Initialize the silk site using the given siteconf file name\n"
      "and rootdir.\n"
      "If the siteconf is not supplied, the value of the environment\n"
      "variable " SILK_CONFIG_FILE_ENVAR " will be used if available.\n"
      "Otherwise, PySiLK will look for a file named 'silk.conf' in the\n"
      "following directories: the rootdir, the directories\n"
      "'$SILK_PATH/share/silk/' and '$SILK_PATH/share/'; and the\n"
      "'share/silk/' and 'share/' directories parallel to the\n"
      "application's directory.\n"
      "If the rootdir is not supplied, SiLK's default value will be\n"
      "used.\n"
      "Will throw an exception if the site is already initialized.")},
    {"have_site_config", (PyCFunction)silk_have_site_config,
     METH_NOARGS,
     ("Return whether the site configuration file has been loaded")},
    {"get_site_config", (PyCFunction)silk_get_siteconf,
     METH_NOARGS,
     ("Return the current site configuration file; None if not set")},
    {"set_data_rootdir", (PyCFunction)silk_set_rootdir,
     METH_VARARGS,
     ("Change the data root directory to the given path")},
    {"get_data_rootdir", (PyCFunction)silk_get_rootdir,
     METH_NOARGS,
     ("Return the current root directory")},
    {"sensor_info", (PyCFunction)silk_sensor_info,
     METH_NOARGS,
     ("Returns a list of information for configured sensors")},
    {"class_info", (PyCFunction)silk_class_info,
     METH_NOARGS,
     ("Return a list of information for configured classes")},
    {"flowtype_info", (PyCFunction)silk_flowtype_info,
     METH_NOARGS,
     ("Return a list of information for configured flowtypes")},
    {"silk_version", (PyCFunction)silk_version,
     METH_NOARGS,
     ("Return the version of SiLK that PySilk was linked against")},
    {"init_country_codes", (PyCFunction)silk_init_country_codes,
     METH_VARARGS,
     ("Initialize the country codes from the given file (can be left blank)")},
    {"_raw_rwrec_copy", (PyCFunction)silk_raw_rwrec_copy,
     METH_O,
     ("Create a RawRWRec from a C rwrec PyCObject, copying the value")},
    {NULL, NULL, 0, NULL}       /* Sentinel */
};

static PyObject *silk_class_info(void)
{
    CHECK_SITE(NULL);

    Py_INCREF(GLOBALS->classes);
    return GLOBALS->classes;
}

static PyObject *silk_flowtype_info(void)
{
    CHECK_SITE(NULL);

    Py_INCREF(GLOBALS->flowtypes);
    return GLOBALS->flowtypes;
}

static PyObject *silk_get_rootdir(void)
{
    char rootdir[PATH_MAX];

    sksiteGetRootDir(rootdir, sizeof(rootdir));
    return PyUnicode_InternFromString(rootdir);
}

static PyObject *silk_get_siteconf(void)
{
    char siteconf[PATH_MAX];

    sksiteGetConfigPath(siteconf, sizeof(siteconf));
    return PyUnicode_InternFromString(siteconf);
}

static PyObject *silk_have_site_config(void)
{
    Py_INCREF(GLOBALS->havesite);
    return GLOBALS->havesite;
}

static PyObject *silk_init_country_codes(
    PyObject    UNUSED(*self),
    PyObject           *args)
{
    int   rv;
    char *filename = NULL;

    rv = PyArg_ParseTuple(args, "|et",
                          Py_FileSystemDefaultEncoding, &filename);
    if (!rv) {
        return NULL;
    }

    skCountryTeardown();
    rv = skCountrySetup(filename, error_printf);
    PyMem_Free(filename);
    if (rv != 0) {
        PyErr_SetString(PyExc_RuntimeError, error_buffer);
        return NULL;
    }

    Py_RETURN_NONE;
}

static int silk_init_set_envvar(
    const char *value,
    const char *envvar)
{
    static char env_buf[101 + PATH_MAX];
    PyObject *os;
    int rv;

    if (value == NULL) {
        return 0;
    }

    /* setenv() does not exist on Solaris 9, so we use putenv()
     * instead */
    rv = snprintf(env_buf, sizeof(env_buf),
                  "%s=%s", envvar, value);
    if (rv >= (int)sizeof(env_buf) || putenv(env_buf) != 0) {
        PyErr_SetString(PyExc_RuntimeError,
                        ("Could not set " SILK_CONFIG_FILE_ENVAR));
        return -1;
    }

    /* Attempt to add the environment variable to Python's
     * environment list as well */
    os = PyImport_ImportModule("os");
    if (os != NULL) {
        PyObject *env = PyObject_GetAttrString(os, "environ");
        if (env != NULL) {
            PyObject *s = PyUnicode_Decode(value,
                                           strlen(value),
                                           Py_FileSystemDefaultEncoding,
                                           "strict");
            if (s != NULL) {
                PyMapping_SetItemString(env, (char*)envvar, s);
                Py_DECREF(s);
            }
            Py_DECREF(env);
        }
        Py_DECREF(os);
    }

    return 0;
}

static PyObject *silk_init_site(
    PyObject    UNUSED(*self),
    PyObject           *args,
    PyObject           *kwds)
{
    static char *kwlist[] = {"siteconf", "rootdir", NULL};
    char *siteconf = NULL;
    char *rootdir  = NULL;
    int   rv;

    rv = PyArg_ParseTupleAndKeywords(args, kwds, "|etet", kwlist,
                                     Py_FileSystemDefaultEncoding, &siteconf,
                                     Py_FileSystemDefaultEncoding, &rootdir);
    if (!rv) {
        goto error;
    }

    if (GLOBALS->site_configured) {
        PyErr_SetString(PyExc_RuntimeError, "Site already initialized");
        goto error;
    }

    if (siteconf) {
        rv = sksiteSetConfigPath(siteconf);
        assert(rv == 0);
    }

    if (rootdir) {
        if (!skDirExists(rootdir)) {
            PyErr_Format(PyExc_IOError,
                         "The directory %s does not exist", rootdir);
            goto error;
        }
        rv = sksiteSetRootDir(rootdir);
        if (rv != 0) {
            PyErr_SetString(PyExc_ValueError, "Illegal root directory");
            goto error;
        }
    }

    if (init_site(siteconf) != 0) {
        goto error;
    }

    /* These are needed for subprocess calls to silk */
    if (silk_init_set_envvar(siteconf, SILK_CONFIG_FILE_ENVAR)) {
        goto error;
    }
    if (silk_init_set_envvar(rootdir, SILK_DATA_ROOTDIR_ENVAR)) {
        goto error;
    }

    Py_INCREF(GLOBALS->havesite);
    return GLOBALS->havesite;

  error:
    PyMem_Free(siteconf);
    PyMem_Free(rootdir);

    return NULL;
}

static PyObject *silk_initial_tcpflags_enabled(void)
{
    Py_RETURN_TRUE;
}

static PyObject *silk_ipset_supports_ipv6(void)
{
#if SK_ENABLE_IPV6 && SK_ENABLE_SILK3_IPSETS
    Py_RETURN_TRUE;
#else
    Py_RETURN_FALSE;
#endif
}

static PyObject *silk_ipv6_enabled(void)
{
#if SK_ENABLE_IPV6
    Py_RETURN_TRUE;
#else
    Py_RETURN_FALSE;
#endif
}

static silkPyRawRWRec *silk_raw_rwrec_copy(
    PyObject    UNUSED(*self),
    PyObject           *c_rec)
{
    silkPyRawRWRec *pyrec;
    rwRec       *rec;

    if (!COBJ_CHECK(c_rec)) {
        PyErr_SetString(PyExc_TypeError, "Illegal argument type");
        return NULL;
    }

    pyrec = (silkPyRawRWRec*)silkPyRawRWRecType.tp_alloc(
        &silkPyRawRWRecType, 0);
    if (pyrec != NULL) {
        rec = (rwRec*)COBJ_PTR(c_rec);
        if (rec != NULL) {
            RWREC_COPY(&pyrec->rec, rec);
        }
    }

    return pyrec;
}

static PyObject *silk_sensor_info(void)
{
    CHECK_SITE(NULL);

    Py_INCREF(GLOBALS->sensors);
    return GLOBALS->sensors;
}

static PyObject *silk_set_rootdir(
    PyObject    UNUSED(*self),
    PyObject           *args)
{
    int   rv;
    char *rootdir = NULL;

    CHECK_SITE(NULL);

    rv = PyArg_ParseTuple(args, "|et",
                          Py_FileSystemDefaultEncoding, &rootdir);
    if (!rv) {
        return NULL;
    }
    if (!skDirExists(rootdir)) {
        PyErr_Format(PyExc_IOError,
                     "The directory %s does not exist", rootdir);
        PyMem_Free(rootdir);
        return NULL;
    }
    rv = sksiteSetRootDir(rootdir);
    PyMem_Free(rootdir);
    if (rv != 0) {
        PyErr_SetString(PyExc_ValueError, "Illegal root directory");
        return NULL;
    }

    Py_RETURN_NONE;
}

static PyObject *silk_version(void)
{
    return PyUnicode_InternFromString(SK_PACKAGE_VERSION);
}


/*
 *************************************************************************
 *   SUPPORT FUNCTIONS
 *************************************************************************
 */

static PyObject *any_obj_error(
    PyObject   *exc,
    const char *format,
    PyObject   *obj)
{
    PyObject *pformat = PyUnicode_FromString(format);
    PyObject *msg = PyUnicode_Format(pformat, obj);
    PyErr_SetObject(exc, msg);
    Py_DECREF(pformat);
    Py_DECREF(msg);
    return NULL;
}

#ifndef TEST_PRINTF_FORMATS
static int error_printf(
    const char *fmt,
    ...)
{
    int rv;

    va_list args;
    va_start(args, fmt);
    rv = vsnprintf(error_buffer, sizeof(error_buffer), fmt, args);
    error_buffer[sizeof(error_buffer) - 1] = '\0';
    va_end(args);

    return rv;
}
#endif  /* TEST_PRINTF_FORMATS */

static int init_classes(void)
{
    class_iter_t  iter;
    classID_t     class_id;
    char          name[SK_MAX_STRLEN_FLOWTYPE+1];
    PyObject     *list;
    PyObject     *dict;
    PyObject     *classes;
    PyObject     *val;
    PyObject     *class_data;
    int           rv;

    classes = PyDict_New();
    if (classes == NULL) {
        goto error;
    }
    class_id = sksiteClassGetDefault();
    if (class_id == SK_INVALID_CLASS) {
        val = Py_None;
        Py_INCREF(val);
    } else {
        val = PyInt_FromLong(sksiteClassGetDefault());
        if (val == NULL) {
            goto error;
        }
    }
    rv = PyDict_SetItemString(classes, "default", val);
    Py_DECREF(val);
    if (rv != 0) {
        goto error;
    }

    class_data = PyDict_New();
    if (class_data == NULL) {
        goto error;
    }
    rv = PyDict_SetItemString(classes, "data", class_data);
    Py_DECREF(class_data);
    if (rv != 0) {
        goto error;
    }

    sksiteClassIterator(&iter);
    while (sksiteClassIteratorNext(&iter, &class_id)) {
        sensor_iter_t    sensor_iter;
        sensorID_t       sensor;
        flowtype_iter_t  flowtype_iter;
        flowtypeID_t     flowtype;

        dict = PyDict_New();
        if (dict == NULL) {
            goto error;
        }
        val = PyInt_FromLong(class_id);
        if (val == NULL) {
            Py_DECREF(dict);
            goto error;
        }
        rv = PyDict_SetItem(class_data, val, dict);
        Py_DECREF(dict);
        if (rv != 0) {
            Py_DECREF(val);
            goto error;
        }
        rv = PyDict_SetItemString(dict, "id", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        sksiteClassGetName(name, sizeof(name), class_id);
        val = PyUnicode_InternFromString(name);
        if (val == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "name", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        list = PyList_New(0);
        if (list == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "sensors", list);
        Py_DECREF(list);
        if (rv != 0) {
            goto error;
        }
        sksiteClassSensorIterator(class_id, &sensor_iter);
        while (sksiteSensorIteratorNext(&sensor_iter, &sensor)) {
            val = PyInt_FromLong(sensor);
            if (val == NULL) {
                goto error;
            }
            rv = PyList_Append(list, val);
            Py_DECREF(val);
            if (rv != 0) {
                goto error;
            }
        }

        list = PyList_New(0);
        if (list == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "flowtypes", list);
        Py_DECREF(list);
        if (rv != 0) {
            goto error;
        }
        sksiteClassFlowtypeIterator(class_id, &flowtype_iter);
        while (sksiteFlowtypeIteratorNext(&flowtype_iter, &flowtype)) {
            val = PyInt_FromLong(flowtype);
            if (val == NULL) {
                goto error;
            }
            rv = PyList_Append(list, val);
            Py_DECREF(val);
            if (rv != 0) {
                goto error;
            }
        }

        list = PyList_New(0);
        if (list == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "default_flowtypes", list);
        Py_DECREF(list);
        if (rv != 0) {
            goto error;
        }
        sksiteClassDefaultFlowtypeIterator(class_id, &flowtype_iter);
        while (sksiteFlowtypeIteratorNext(&flowtype_iter, &flowtype)) {
            val = PyInt_FromLong(flowtype);
            if (val == NULL) {
                goto error;
            }
            rv = PyList_Append(list, val);
            Py_DECREF(val);
            if (rv != 0) {
                goto error;
            }
        }
    }

    GLOBALS->classes = classes;
    return 0;

  error:
    Py_XDECREF(classes);

    return -1;
}

static int init_flowtypes(void)
{
    flowtype_iter_t  flowtype_iter;
    flowtypeID_t     flowtype;
    char             name[SK_MAX_STRLEN_SENSOR+1];
    PyObject        *dict;
    PyObject        *flowtypes;
    int              rv;

    flowtypes = PyDict_New();
    if (flowtypes == NULL) {
        goto error;
    }

    sksiteFlowtypeIterator(&flowtype_iter);
    while (sksiteFlowtypeIteratorNext(&flowtype_iter, &flowtype)) {
        classID_t     class_id;
        PyObject     *val;

        dict = PyDict_New();
        if (dict == NULL) {
            goto error;
        }
        val = PyInt_FromLong(flowtype);
        if (val == NULL) {
            Py_DECREF(dict);
            goto error;
        }
        rv = PyDict_SetItem(flowtypes, val, dict);
        Py_DECREF(dict);
        if (rv != 0) {
            Py_DECREF(val);
            goto error;
        }
        rv = PyDict_SetItemString(dict, "id", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        sksiteFlowtypeGetName(name, sizeof(name), flowtype);
        val = PyUnicode_InternFromString(name);
        if (val == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "name", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        sksiteFlowtypeGetType(name, sizeof(name), flowtype);
        val = PyUnicode_InternFromString(name);
        if (val == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "type", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        class_id = sksiteFlowtypeGetClassID(flowtype);
        val = PyInt_FromLong(class_id);
        if (val == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "class", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }
    }

    GLOBALS->flowtypes = flowtypes;
    return 0;

  error:
    Py_XDECREF(flowtypes);

    return -1;
}

static int init_sensors(void)
{
    sensor_iter_t  sensor_iter;
    sensorID_t     sensor;
    char           name[SK_MAX_STRLEN_SENSOR+1];
    PyObject      *list;
    PyObject      *dict;
    PyObject      *sensors;
    int            rv;

    sensors = PyDict_New();
    if (sensors == NULL) {
        goto error;
    }

    sksiteSensorIterator(&sensor_iter);
    while (sksiteSensorIteratorNext(&sensor_iter, &sensor)) {
        class_iter_t  class_iter;
        classID_t     class_id;
        PyObject     *val;
        const char   *desc;

        dict = PyDict_New();
        if (dict == NULL) {
            goto error;
        }
        val = PyInt_FromLong(sensor);
        if (val == NULL) {
            Py_DECREF(dict);
            goto error;
        }
        rv = PyDict_SetItem(sensors, val, dict);
        Py_DECREF(dict);
        if (rv != 0) {
            Py_DECREF(val);
            goto error;
        }
        rv = PyDict_SetItemString(dict, "id", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        sksiteSensorGetName(name, sizeof(name), sensor);
        val = PyUnicode_InternFromString(name);
        if (val == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "name", val);
        Py_DECREF(val);
        if (rv != 0) {
            goto error;
        }

        desc = sksiteSensorGetDescription(sensor);
        if (desc) {
            val = PyUnicode_FromString(desc);
            if (val == NULL) {
                goto error;
            }
            rv = PyDict_SetItemString(dict, "description", val);
            Py_DECREF(val);
            if (rv != 0) {
                goto error;
            }
        }

        list = PyList_New(0);
        if (list == NULL) {
            goto error;
        }
        rv = PyDict_SetItemString(dict, "classes", list);
        Py_DECREF(list);
        if (rv != 0) {
            goto error;
        }
        sksiteSensorClassIterator(sensor, &class_iter);
        while (sksiteClassIteratorNext(&class_iter, &class_id)) {
            val = PyInt_FromLong(class_id);
            if (val == NULL) {
                goto error;
            }
            rv = PyList_Append(list, val);
            Py_DECREF(val);
            if (rv != 0) {
                goto error;
            }
        }
    }

    GLOBALS->sensors = sensors;
    return 0;

  error:
    Py_XDECREF(sensors);

    return -1;
}

static int init_silkfile_module(
    PyObject   *mod)
{
    int rv;

    PyModule_AddIntConstant(mod, "IGNORE", SK_IPV6POLICY_IGNORE);
    PyModule_AddIntConstant(mod, "ASV4", SK_IPV6POLICY_ASV4);
    PyModule_AddIntConstant(mod, "MIX", SK_IPV6POLICY_MIX);
    PyModule_AddIntConstant(mod, "FORCE", SK_IPV6POLICY_FORCE);
    PyModule_AddIntConstant(mod, "ONLY", SK_IPV6POLICY_ONLY);

    PyModule_AddIntConstant(mod, "READ", SK_IO_READ);
    PyModule_AddIntConstant(mod, "WRITE", SK_IO_WRITE);
    PyModule_AddIntConstant(mod, "APPEND", SK_IO_APPEND);

    PyModule_AddIntConstant(mod, "DEFAULT", NOT_SET);
    PyModule_AddIntConstant(mod, "NO_COMPRESSION", SK_COMPMETHOD_NONE);
    PyModule_AddIntConstant(mod, "ZLIB", SK_COMPMETHOD_ZLIB);
    PyModule_AddIntConstant(mod, "LZO1X", SK_COMPMETHOD_LZO1X);

    PyModule_AddObject(mod, "BAG_COUNTER_MAX",
                       PyLong_FromUnsignedLongLong(SKBAG_COUNTER_MAX));

    silkPySilkFileType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPySilkFileType) < 0) {
        return -1;
    }
    rv = PyModule_AddObject(mod, "SilkFileBase",
                            (PyObject*)&silkPySilkFileType);
    if (rv != 0) {
        return -1;
    }

    return 0;
}

static int init_site(
    const char *site_file)
{
    int rv;
    int siterv;
    int retval = 0;

    if (GLOBALS->site_configured) {
        return 0;
    }

    siterv = sksiteConfigure(0);
    if (siterv == 0) {
        GLOBALS->havesite = Py_True;
    } else if (siterv == -2) {
        GLOBALS->havesite = Py_False;
        if (site_file) {
            Py_INCREF(GLOBALS->havesite);
            PyErr_Format(PyExc_IOError,
                         "could not read site file %s", site_file);
            retval = -1;
        }
    } else {
        GLOBALS->havesite = Py_False;
        PyErr_SetString(PyExc_RuntimeError,
                        "error parsing site configuration file");
        retval = -1;
    }
    Py_INCREF(GLOBALS->havesite);
    rv = PyModule_AddObject(GLOBALS->silkmod, "_havesite", GLOBALS->havesite);
    if (rv != 0) {
        return -1;
    }

    rv = init_sensors();
    if (rv != 0) {
        return -1;
    }
    rv = PyModule_AddObject(GLOBALS->silkmod, "_sensors", GLOBALS->sensors);
    if (rv != 0) {
        return -1;
    }

    rv = init_classes();
    if (rv != 0) {
        return -1;
    }
    rv = PyModule_AddObject(GLOBALS->silkmod, "_classes", GLOBALS->classes);
    if (rv != 0) {
        return -1;
    }

    rv = init_flowtypes();
    if (rv != 0) {
        return -1;
    }
    rv = PyModule_AddObject(GLOBALS->silkmod, "_flowtypes",
                            GLOBALS->flowtypes);
    if (rv != 0) {
        return -1;
    }

    if (siterv == 0) {
        GLOBALS->site_configured = 1;
    }

    return retval;
}

static PyObject *iter_iter(
    PyObject   *self)
{
    Py_INCREF(self);
    return self;
}

static void obj_dealloc(
    PyObject   *obj)
{
    Py_TYPE(obj)->tp_free(obj);
}

static PyObject *obj_error(
    const char *format,
    PyObject   *obj)
{
    return any_obj_error(PyExc_ValueError, format, obj);
}

static PyObject *reduce_error(
    PyObject   *obj)
{
    if (Py_TYPE(obj) && Py_TYPE(obj)->tp_name) {
        PyErr_Format(PyExc_TypeError, "can't pickle %s objects",
                     Py_TYPE(obj)->tp_name);
    } else {
        PyErr_SetString(PyExc_TypeError, "This object cannot be pickled");
    }
    return NULL;
}

static int silkPyDatetimeToSktime(
    sktime_t   *silktime,
    PyObject   *datetime)
{
    PyObject *delta = NULL;
    PyObject *days = NULL;
    PyObject *secs = NULL;
    PyObject *usecs = NULL;
    int64_t millisecs;
    int retval = -1;

    if (!PyDateTime_Check(datetime)) {
        if (PyDate_Check(datetime)) {
            datetime = PyDateTime_FromDateAndTime(
                PyDateTime_GET_YEAR(datetime),
                PyDateTime_GET_MONTH(datetime),
                PyDateTime_GET_DAY(datetime),
                0, 0, 0, 0);
            if (datetime == NULL) {
                return -1;
            }
        } else {
            PyErr_SetString(PyExc_TypeError, "Expected a datetime.date");
            return -1;
        }
    } else {
        Py_INCREF(datetime);
    }
    if (PyObject_RichCompareBool(datetime, GLOBALS->zerotime, Py_LT)) {
        PyErr_SetString(PyExc_OverflowError, "Minimum time is Jan 1, 1970");
        Py_DECREF(datetime);
        return -1;
    }
    delta = PyNumber_Subtract(datetime, GLOBALS->zerotime);
    Py_DECREF(datetime);
    days = PyObject_GetAttrString(delta, "days");
    secs = PyObject_GetAttrString(delta, "seconds");
    usecs = PyObject_GetAttrString(delta, "microseconds");
    millisecs = (int64_t)PyLong_AsLong(days) * 1000 * 24 * 3600 +
                (int64_t)PyLong_AsLong(secs) * 1000 +
                PyLong_AsLong(usecs) / 1000;
    if (!PyErr_Occurred()) {
        *silktime = millisecs;
        retval = 0;
    }
    Py_XDECREF(delta);
    Py_XDECREF(days);
    Py_XDECREF(secs);
    Py_XDECREF(usecs);

    return retval;
}

#if !SK_ENABLE_IPV6
static PyObject *silkPyNotImplemented(
    PyObject    UNUSED(*self),
    PyObject    UNUSED(*args),
    PyObject    UNUSED(*kwds))
{
    return PyErr_Format(PyExc_NotImplementedError,
                        "SiLK was not built with IPv6 support.");
}
#endif  /* !SK_ENABLE_IPV6 */


static PyObject *initpysilkbase(
    char*   name)
{
#if PY_MAJOR_VERSION >= 3
    /* Module information for Python 3.0 */
    static struct PyModuleDef pysilk_module_static = {
        PyModuleDef_HEAD_INIT,
        NULL,                       /* m_name */
        NULL,                       /* m_doc */
        sizeof(silkpy_globals_t),   /* m_size */
        silk_methods,               /* m_methods */
        NULL,                       /* m_reaload (unused) */
        NULL,                       /* m_traverse */
        NULL,                       /* m_clear */
        NULL                        /* m_free */
    };
#endif

    PyObject         *silkmod;
    PyObject         *tmp;
    silkpy_globals_t *globals;

    PyDateTime_IMPORT;

#if PY_MAJOR_VERSION >= 3
    pysilk_module = &pysilk_module_static;
    pysilk_module->m_name = name;
    silkmod = PyModule_Create(pysilk_module);
    if (silkmod == NULL) {
        skAppPrintErr("Could not create module silk");
        goto err;
    }
    globals = (silkpy_globals_t*)PyModule_GetState(silkmod);
#else
    /* Globals are in the static global variable */
    globals = &silkpy_globals_static;
    silkmod = Py_InitModule3(name, silk_methods, "SiLK extension module");
    if (silkmod == NULL) {
        skAppPrintErr("Could not create module silk");
        goto err;
    }
#endif  /* #else of #if PY_MAJOR_VERSION >= 3 */

    memset(globals, 0, sizeof(*globals));
    globals->silkmod = silkmod;
    globals->havesite = Py_False;
    Py_INCREF(globals->havesite);

    if (init_silkfile_module(silkmod)) {
        goto err;
    }

    if (silkPyIPAddr_setup(silkmod) != 0) {
        goto err;
    }

    if (PyType_Ready(&silkPyIPv4AddrType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPv4Addr",
                                     (PyObject*)&silkPyIPv4AddrType),
                  int, 0);

    if (PyType_Ready(&silkPyIPv6AddrType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPv6Addr",
                                     (PyObject*)&silkPyIPv6AddrType),
                  int, 0);

    if (PyType_Ready(&silkPyIPWildcardType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPWildcard",
                                     (PyObject*)&silkPyIPWildcardType),
                  int, 0);

    silkPyIPWildcardIterType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyIPWildcardIterType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPWildcardIter",
                                     (PyObject*)&silkPyIPWildcardIterType),
                  int, 0);

    silkPyIPSetType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyIPSetType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPSetBase",
                                     (PyObject*)&silkPyIPSetType),
                  int, 0);

    silkPyIPSetIterType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyIPSetIterType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "IPSetIter",
                                     (PyObject*)&silkPyIPSetIterType),
                  int, 0);

    silkPyPmapType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyPmapType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "PMapBase",
                                     (PyObject*)&silkPyPmapType),
                  int, 0);

    silkPyPmapIterType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyPmapIterType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "PMapBaseIter",
                                     (PyObject*)&silkPyPmapIterType),
                  int, 0);

    if (silkPyBag_setup(silkmod) != 0) {
        goto err;
    }

    silkPyBagIterType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyBagIterType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "BagBaseIter",
                                     (PyObject*)&silkPyBagIterType),
                  int, 0);

    silkPyRepoIterType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyRepoIterType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "RepoIter",
                                     (PyObject*)&silkPyRepoIterType),
                  int, 0);

    if (silkPyTCPFlags_setup(silkmod)) {
        goto err;
    }

    if (PyType_Ready(&silkPyRawRWRecType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "RawRWRec",
                                     (PyObject*)&silkPyRawRWRecType),
                  int, 0);

    silkPyRWRecType.tp_new = PyType_GenericNew;
    if (PyType_Ready(&silkPyRWRecType) < 0) {
        goto err;
    }
    ASSERT_RESULT(PyModule_AddObject(silkmod, "RWRecBase",
                                     (PyObject*)&silkPyRWRecType),
                  int, 0);

    tmp = PyImport_ImportModule("datetime");
    if (tmp == NULL) {
        skAppPrintErr("Failed to import datetime module");
        goto err;
    }

    globals->timedelta = PyObject_GetAttrString(tmp, "timedelta");
    assert(globals->timedelta != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_timedelta", globals->timedelta),
                  int, 0);

    globals->datetime = PyObject_GetAttrString(tmp, "datetime");
    assert(globals->datetime != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_datetime", globals->datetime),
                  int, 0);

    Py_DECREF(tmp);

    globals->maxelapsed = PyObject_CallFunction(globals->timedelta,
                                                "iiik", 0, 0, 0, UINT32_MAX);
    assert(globals->maxelapsed != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_maxelapsed",
                                     globals->maxelapsed),
                  int, 0);

    globals->minelapsed = PyObject_CallObject(globals->timedelta, NULL);
    assert(globals->minelapsed != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_minelapsed",
                                     globals->minelapsed),
                  int, 0);

    globals->zerotime = PyObject_CallFunction(globals->datetime, "iii",
                                              1970, 1, 1);
    assert(globals->zerotime != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_zerotime", globals->zerotime),
                  int, 0);

    globals->thousand = PyFloat_FromDouble(1000.0);
    assert(globals->thousand != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_thousand", globals->thousand),
                  int, 0);

    globals->maxintipv4 = PyLong_FromString("0xffffffff", NULL, 0);
    assert(globals->maxintipv4 != NULL);
    ASSERT_RESULT(PyModule_AddObject(globals->silkmod, "_maxintipv4",
                                     globals->maxintipv4),
                  int, 0);

#if SK_ENABLE_IPV6
    globals->maxintipv6 =
        PyLong_FromString("0xffffffffffffffffffffffffffffffff", NULL, 0);
    assert(globals->maxintipv6 != NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_maxintipv6",
                                     globals->maxintipv6),
                  int, 0);
#endif  /* SK_ENABLE_IPV6 */

    globals->newrawrec = PyObject_CallFunctionObjArgs(
        (PyObject *)&silkPyRawRWRecType, NULL);
    ASSERT_RESULT(PyModule_AddObject(silkmod, "_newrawrec",
                                     globals->newrawrec),
                  int, 0);

    return silkmod;

  err:
    if (silkmod) {
        Py_DECREF(silkmod);
    }
#if PY_MAJOR_VERSION >= 3
    return NULL;
#else /* PY_MAJOR_VERSION < 3 */
    if (PyErr_Occurred()) {
        PyErr_Print();
    }
    exit(EXIT_FAILURE);
#endif  /* PY_MAJOR_VERSION */
}

PyMODINIT_FUNC PYSILK_INIT(void)
{
    PyObject *nameobj;

    nameobj = BYTES_FROM_XCHAR(Py_GetProgramName());

    if (nameobj == NULL) {
        skAppRegister("PySiLK_program");
    } else {
        skAppRegister(PyBytes_AS_STRING(nameobj));
        Py_DECREF(nameobj);
    }
    INIT_RETURN(initpysilkbase(STR(PYSILK_NAME)));
}

PyMODINIT_FUNC PYSILK_NL_INIT(void)
{
    INIT_RETURN(initpysilkbase(STR(PYSILK_NL_NAME)));
}


/*
** Local Variables:
** mode:c
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hello again, Michael.

On Thu, Oct 25, 2012 at 09:41:11AM -0400, Michael Welsh Duggan wrote:
> A new recipe.  This is a strange one.  I hope it works for you as well.

> emacs -Q
> M-x tool-bar-mode
> resize the frame to be height 56 (according to the window manager)
> Load the included file
> M-x c-toggle-parse-state-debug
> C-s FIXME C-s
> Move cursor to the "o" on the next line (1175).
> Type: 
> PyTYPE( 
> C-f 
> )

> I get errors at the first open paren, and the close paren.  Resizing the
> fame is important!  I think it's due to caching that happens during font
> locking. 

Thanks for this one.

The cause was the cache invalidation function being called from
after-change-functions.  The newly inserted parens were fouling up the
invalidation algorithm.  :-(  The solution is to call that function from
before-change-functions instead.

The changes are currently as follows.  Would you try out the patch as
usual, please, and let me know how it goes.



diff -r ac6584d14c06 cc-engine.el
--- a/cc-engine.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-engine.el	Sun Oct 28 11:31:48 2012 +0000
@@ -2560,8 +2560,11 @@
 	      start-point cache-pos)))
 
     ;; Might we be better off starting from the top level, two defuns back,
-    ;; instead?
-    (when (> how-far c-state-cache-too-far)
+    ;; instead?  This heuristic no longer works well in C++, where
+    ;; declarations inside namespace brace blocks are frequently placed at
+    ;; column zero.
+    (when (and (not (c-major-mode-is 'c++-mode))
+	       (> how-far c-state-cache-too-far))
       (setq BOD-pos (c-get-fallback-scan-pos here)) ; somewhat EXPENSIVE!!!
       (if (< (- here BOD-pos) how-far)
 	  (setq strategy 'BOD
@@ -2648,17 +2651,19 @@
 	;; If we're essentially repeating a fruitless search, just give up.
 	(unless (and c-state-brace-pair-desert
 		     (eq cache-pos (car c-state-brace-pair-desert))
+		     (> from (car c-state-brace-pair-desert))
 		     (<= from (cdr c-state-brace-pair-desert)))
-	  ;; DESERT-LIM.  Only search what we absolutely need to,
+	  ;; DESERT-LIM.  Avoid repeated searching through the cached desert.
 	  (let ((desert-lim
 		 (and c-state-brace-pair-desert
 		      (eq cache-pos (car c-state-brace-pair-desert))
+		      (>= from (cdr c-state-brace-pair-desert))
 		      (cdr c-state-brace-pair-desert)))
 		;; CACHE-LIM.  This limit will be necessary when an opening
 		;; paren at `cache-pos' has just had its matching close paren
-		;; inserted.  `cache-pos' continues to be a search bound, even
-		;; though the algorithm below would skip over the new paren
-		;; pair.
+		;; inserted into the buffer.  `cache-pos' continues to be a
+		;; search bound, even though the algorithm below would skip
+		;; over the new paren pair.
 		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
 	    (narrow-to-region
 		(cond
@@ -3354,13 +3359,19 @@
   (fset 'c-real-parse-state (symbol-function 'c-parse-state)))
 (cc-bytecomp-defun c-real-parse-state)
 
+(defvar c-parse-state-point nil)
 (defvar c-parse-state-state nil)
 (make-variable-buffer-local 'c-parse-state-state)
 (defun c-record-parse-state-state ()
+  (setq c-parse-state-point (point))
   (setq c-parse-state-state
 	(mapcar
 	 (lambda (arg)
-	   (cons arg (symbol-value arg)))
+	   (let ((val (symbol-value arg)))
+	     (cons arg
+		   (if (consp val)
+		       (copy-tree val)
+		     val))))
 	 '(c-state-cache
 	   c-state-cache-good-pos
 	   c-state-nonlit-pos-cache
@@ -3373,7 +3384,8 @@
 	   c-state-point-min-lit-start
 	   c-state-min-scan-pos
 	   c-state-old-cpp-beg
-	   c-state-old-cpp-end))))
+	   c-state-old-cpp-end
+	   c-parse-state-point))))
 (defun c-replay-parse-state-state ()
   (message
    (concat "(setq "
@@ -3416,7 +3428,8 @@
       (message "Old state:")
       (c-replay-parse-state-state))
     (c-record-parse-state-state)
-    res1))
+    res2 ; res1 correct a cascading series of errors ASAP
+    ))
 
 (defun c-toggle-parse-state-debug (&optional arg)
   (interactive "P")
diff -r ac6584d14c06 cc-mode.el
--- a/cc-mode.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-mode.el	Sun Oct 28 11:31:48 2012 +0000
@@ -1058,7 +1058,10 @@
 	      (mapc (lambda (fn)
 		      (funcall fn beg end))
 		    c-get-state-before-change-functions))
-	  )))))
+	  )))
+    ;; The following must be done here rather than in `c-after-change' because
+    ;; newly inserted parens would foul up the invalidation algorithm.
+    (c-invalidate-state-cache beg)))
 
 (defvar c-in-after-change-fontification nil)
 (make-variable-buffer-local 'c-in-after-change-fontification)
@@ -1108,7 +1111,7 @@
 
 	(c-trim-found-types beg end old-len) ; maybe we don't need all of these.
 	(c-invalidate-sws-region-after beg end)
-	(c-invalidate-state-cache beg)
+	;; (c-invalidate-state-cache beg) ; moved to `c-before-change'.
 	(c-invalidate-find-decl-cache beg)
 
 	(when c-recognize-<>-arglists


> 
> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Chong Yidong]

Alan Mackenzie <acm@muc.de> writes:

> The cause was the cache invalidation function being called from
> after-change-functions.  The newly inserted parens were fouling up the
> invalidation algorithm.  :-(  The solution is to call that function from
> before-change-functions instead.
>
> The changes are currently as follows.  Would you try out the patch as
> usual, please, and let me know how it goes.

Hi Alan,

When you are ready to commit this, please commit to the emacs-24 branch
(as opposed to the trunk).  I think this should be fixed for Emacs 24.3.
Thanks.

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Yidong,

On Sun, Nov 04, 2012 at 11:43:26AM +0800, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> Hi Alan,

> When you are ready to commit this, please commit to the emacs-24 branch
> (as opposed to the trunk).  I think this should be fixed for Emacs 24.3.
> Thanks.

Will do!  I agree it should be fixed for 24.3, as it has become a rather
"popular" bug out in the wild.  :-(  I'm hoping I'm close to a fix, now.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hello, Yidong.

On Sun, Nov 04, 2012 at 11:43:26AM +0800, Chong Yidong wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > The cause was the cache invalidation function being called from
> > after-change-functions.  The newly inserted parens were fouling up the
> > invalidation algorithm.  :-(  The solution is to call that function from
> > before-change-functions instead.

> > The changes are currently as follows.  Would you try out the patch as
> > usual, please, and let me know how it goes.

> Hi Alan,

> When you are ready to commit this, please commit to the emacs-24 branch
> (as opposed to the trunk).  I think this should be fixed for Emacs 24.3.
> Thanks.

I've just committed the current state of the bug fixes to Emacs-24 in
anticipation of the coming pretest.  (I also bumped CC Mode's version
number.)  Can I assume that you or Stefan will copy the changes to the
trunk?

There is, however, at least one outstanding bug, as reported by Michael
on 2012-11-14.  I'm trying to track this one down, but at the moment I
can't reproduce it.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Stefan Monnier]

> I've just committed the current state of the bug fixes to Emacs-24 in
> anticipation of the coming pretest.  (I also bumped CC Mode's version
> number.)

Thanks.

> Can I assume that you or Stefan will copy the changes to the trunk?

Could also be someone else, but yes you can assume it will happen within
a few days.  Except we call it "merge" rather than "copy" ;-)

> There is, however, at least one outstanding bug, as reported by Michael
> on 2012-11-14.  I'm trying to track this one down, but at the moment I
> can't reproduce it.

I wish you luck,


        Stefan

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Kim,

could you please try out this patch on your C++ files, and let me know
if the main problem has been resolved.

Michael,

there was a rather crass bug in last week's patch.  Sorry.  It's fixed
now, below.  M-x c-toggle-parse-state-debug has been enhanced to display
messages when the state-cache is intrinsically invalid.  It also now
gives a message when you {en,dis}able it.  Would you try out this patch
now, please.

Thanks to you both!

On Sun, Oct 28, 2012 at 11:36:10AM +0000, Alan Mackenzie wrote:

> On Thu, Oct 25, 2012 at 09:41:11AM -0400, Michael Welsh Duggan wrote:
> > A new recipe.  This is a strange one.  I hope it works for you as well.

> > emacs -Q
> > M-x tool-bar-mode
> > resize the frame to be height 56 (according to the window manager)
> > Load the included file
> > M-x c-toggle-parse-state-debug
> > C-s FIXME C-s
> > Move cursor to the "o" on the next line (1175).
> > Type: 
> > PyTYPE( 
> > C-f 
> > )

> > I get errors at the first open paren, and the close paren.  Resizing the
> > fame is important!  I think it's due to caching that happens during font
> > locking. 

> Thanks for this one.

> The cause was the cache invalidation function being called from
> after-change-functions.  The newly inserted parens were fouling up the
> invalidation algorithm.  :-(  The solution is to call that function from
> before-change-functions instead.



diff -r ac6584d14c06 cc-engine.el
--- a/cc-engine.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-engine.el	Sun Nov 04 20:24:06 2012 +0000
@@ -2560,8 +2560,11 @@
 	      start-point cache-pos)))
 
     ;; Might we be better off starting from the top level, two defuns back,
-    ;; instead?
-    (when (> how-far c-state-cache-too-far)
+    ;; instead?  This heuristic no longer works well in C++, where
+    ;; declarations inside namespace brace blocks are frequently placed at
+    ;; column zero.
+    (when (and (not (c-major-mode-is 'c++-mode))
+	       (> how-far c-state-cache-too-far))
       (setq BOD-pos (c-get-fallback-scan-pos here)) ; somewhat EXPENSIVE!!!
       (if (< (- here BOD-pos) how-far)
 	  (setq strategy 'BOD
@@ -2648,17 +2651,20 @@
 	;; If we're essentially repeating a fruitless search, just give up.
 	(unless (and c-state-brace-pair-desert
 		     (eq cache-pos (car c-state-brace-pair-desert))
+		     (or (null (car c-state-brace-pair-desert))
+			 (> from (car c-state-brace-pair-desert)))
 		     (<= from (cdr c-state-brace-pair-desert)))
-	  ;; DESERT-LIM.  Only search what we absolutely need to,
+	  ;; DESERT-LIM.  Avoid repeated searching through the cached desert.
 	  (let ((desert-lim
 		 (and c-state-brace-pair-desert
 		      (eq cache-pos (car c-state-brace-pair-desert))
+		      (>= from (cdr c-state-brace-pair-desert))
 		      (cdr c-state-brace-pair-desert)))
 		;; CACHE-LIM.  This limit will be necessary when an opening
 		;; paren at `cache-pos' has just had its matching close paren
-		;; inserted.  `cache-pos' continues to be a search bound, even
-		;; though the algorithm below would skip over the new paren
-		;; pair.
+		;; inserted into the buffer.  `cache-pos' continues to be a
+		;; search bound, even though the algorithm below would skip
+		;; over the new paren pair.
 		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
 	    (narrow-to-region
 		(cond
@@ -3354,13 +3360,19 @@
   (fset 'c-real-parse-state (symbol-function 'c-parse-state)))
 (cc-bytecomp-defun c-real-parse-state)
 
+(defvar c-parse-state-point nil)
 (defvar c-parse-state-state nil)
 (make-variable-buffer-local 'c-parse-state-state)
 (defun c-record-parse-state-state ()
+  (setq c-parse-state-point (point))
   (setq c-parse-state-state
 	(mapcar
 	 (lambda (arg)
-	   (cons arg (symbol-value arg)))
+	   (let ((val (symbol-value arg)))
+	     (cons arg
+		   (if (consp val)
+		       (copy-tree val)
+		     val))))
 	 '(c-state-cache
 	   c-state-cache-good-pos
 	   c-state-nonlit-pos-cache
@@ -3373,7 +3385,8 @@
 	   c-state-point-min-lit-start
 	   c-state-min-scan-pos
 	   c-state-old-cpp-beg
-	   c-state-old-cpp-end))))
+	   c-state-old-cpp-end
+	   c-parse-state-point))))
 (defun c-replay-parse-state-state ()
   (message
    (concat "(setq "
@@ -3383,6 +3396,16 @@
      c-parse-state-state "  ")
     ")")))
 
+(defun c-debug-parse-state-double-cons (state)
+  (let (state-car conses-not-ok)
+    (while state
+      (setq state-car (car state)
+	    state (cdr state))
+      (if (and (consp state-car)
+	       (consp (car state)))
+	  (setq conses-not-ok t)))
+    conses-not-ok))
+
 (defun c-debug-parse-state ()
   (let ((here (point)) (res1 (c-real-parse-state)) res2)
     (let ((c-state-cache nil)
@@ -3415,8 +3438,16 @@
 	       here res1 res2)
       (message "Old state:")
       (c-replay-parse-state-state))
+
+    (when (c-debug-parse-state-double-cons res1)
+      (message "c-parse-state INVALIDITY at %s: %s"
+	       here res1)
+      (message "Old state:")
+      (c-replay-parse-state-state))
+
     (c-record-parse-state-state)
-    res1))
+    res2 ; res1 correct a cascading series of errors ASAP
+    ))
 
 (defun c-toggle-parse-state-debug (&optional arg)
   (interactive "P")
@@ -3424,7 +3455,9 @@
   (fset 'c-parse-state (symbol-function (if c-debug-parse-state
 					    'c-debug-parse-state
 					  'c-real-parse-state)))
-  (c-keep-region-active))
+  (c-keep-region-active)
+  (message "c-debug-parse-state %sabled"
+	   (if c-debug-parse-state "en" "dis")))
 (when c-debug-parse-state
   (c-toggle-parse-state-debug 1))
 
diff -r ac6584d14c06 cc-mode.el
--- a/cc-mode.el	Sun Sep 09 11:15:13 2012 +0000
+++ b/cc-mode.el	Sun Nov 04 20:24:06 2012 +0000
@@ -1058,7 +1058,10 @@
 	      (mapc (lambda (fn)
 		      (funcall fn beg end))
 		    c-get-state-before-change-functions))
-	  )))))
+	  )))
+    ;; The following must be done here rather than in `c-after-change' because
+    ;; newly inserted parens would foul up the invalidation algorithm.
+    (c-invalidate-state-cache beg)))
 
 (defvar c-in-after-change-fontification nil)
 (make-variable-buffer-local 'c-in-after-change-fontification)
@@ -1108,7 +1111,7 @@
 
 	(c-trim-found-types beg end old-len) ; maybe we don't need all of these.
 	(c-invalidate-sws-region-after beg end)
-	(c-invalidate-state-cache beg)
+	;; (c-invalidate-state-cache beg) ; moved to `c-before-change'.
 	(c-invalidate-find-decl-cache beg)
 
 	(when c-recognize-<>-arglists


> > -- 
> > Michael Welsh Duggan
> > (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2012-11-04 21:39, Alan Mackenzie wrote:
> Kim,
>
> could you please try out this patch on your C++ files, and let me know
> if the main problem has been resolved.
>
> Michael,
Hi Michael and Alan

Sorry for the silence on this bug - I've worked on some other projects 
where the
bug has only shown up once or twice, so I couldn't really do any useful 
testing.

Also, my files are all C and not C++, so I didn't think the fixes you 
discussed would
be relevant for me.

In any case, I will install the patch and see what happens.

Kim

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> there was a rather crass bug in last week's patch.  Sorry.  It's fixed
> now, below.  M-x c-toggle-parse-state-debug has been enhanced to display
> messages when the state-cache is intrinsically invalid.  It also now
> gives a message when you {en,dis}able it.  Would you try out this patch
> now, please.

Sorry it has taken me so long to respond.  I was working in non-C-based
languages for a while.  today I started working in C again, and ran into
issues like this:

Error during redisplay: (jit-lock-function 3411) signaled (wrong-type-argument number-or-marker-p nil)
c-parse-state inconsistency at 3599: using cache: nil, from scratch: ((3409 . 3599))
Old state:
(setq c-state-cache '((3409 . 3599))  c-state-cache-good-pos 3599  c-state-nonlit-pos-cache '(6307 3307)  c-state-nonlit-pos-cache-limit 3592  c-state-semi-nonlit-pos-cache '(3307)  c-state-semi-nonlit-pos-cache-limit 3592  c-state-brace-pair-desert '(nil . 3396)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil  c-parse-state-point 3600)

I have finally managed to come up with a recipe for this one.  I am
using "110803 rgm@gnu.org-20121105111732-ilq2sbfo09xg9i9z" with your
patches.

emacs -Q rwfileinfo.c
M-x c-toggle-parse-state-debug
C-x 4 b *Messages* RET
C-s enum RET
C-M-f C-p C-e ,

Error occurs after the comma (should be on line 197).


[-- Attachment #2: rwfileinfo.c --]
[-- Type: text/plain, Size: 24828 bytes --]

/*
** Copyright (C) 2003-2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  rwfileinfo
**
**  Prints information about an rw-file:
**    file type
**    file version
**    byte order
**    compression level
**    header size
**    record size
**    record count
**    file size
**    command line args used to generate file
**
*/


#include <silk/silk.h>

RCSIDENT("$Id$");

#include <silk/sksite.h>
#include <silk/utils.h>
#include <silk/skstream.h>
#include <silk/skstringmap.h>


/* LOCAL DEFINES AND TYPEDEFS */

/* where to write --help output */
#define USAGE_FH stdout

/* when determining number of records in file, number of bytes to
 * request at one time */
#define RWINFO_BLOCK_SIZE 0x40000

/* format for a label */
#define LABEL_FMT "  %-20s"

/* format for a label that is a number, such as when printing command lines */
#define LABEL_NUM_FMT "%20d  "

/* A list of the properties that can be printed; keep in sync with the
 * info_props[] array below */
enum info_fileinfo_id {
    RWINFO_FORMAT,
    RWINFO_VERSION,
    RWINFO_BYTE_ORDER,
    RWINFO_COMPRESSION,
    RWINFO_HEADER_LENGTH,
    RWINFO_RECORD_LENGTH,
    RWINFO_COUNT_RECORDS,
    RWINFO_FILE_SIZE,
    RWINFO_COMMAND_LINES,
    RWINFO_RECORD_VERSION,
    RWINFO_SILK_VERSION,
    RWINFO_PACKED_FILE_INFO,
    RWINFO_PROBE_NAME,
    RWINFO_ANNOTIONS,
    RWINFO_PREFIX_MAP,
    RWINFO_IPSET,
    RWINFO_BAG,
    /* Last item is used to get a count of the above; it must be last */
    RWINFO_PROPERTY_COUNT
};

/* A property */
typedef struct info_property_st {
    /* property's label */
    const char   *label;
    /* nonzero if printed */
    unsigned      will_print :1;
} info_property_t;


/* LOCAL VARIABLES */

/* keep in sync with the info_fileinfo_id enum above */
static info_property_t info_props[RWINFO_PROPERTY_COUNT] = {
    {"format(id)",          1},
    {"version",             1},
    {"byte-order",          1},
    {"compression(id)",     1},
    {"header-length",       1},
    {"record-length",       1},
    {"count-records",       1},
    {"file-size",           1},
    {"command-lines",       1},
    {"record-version",      1},
    {"silk-version",        1},
    {"packed-file-info",    1},
    {"probe-name",          1},
    {"annotations",         1},
    {"prefix-map",          1},
    {"ipset",               1},
    {"bag",                 1}
};

/* whether to print the summary */
static int print_summary = 0;

/* whether to not print titles (0==print titles, 1==no titles) */
static int no_titles = 0;

/* for looping over files on the command line */
static sk_options_ctx_t *optctx = NULL;


/* OPTIONS SETUP */

/* Create constants for the option processor */
typedef enum rwinfoOptionIds {
    OPT_FIELDS,
    OPT_SUMMARY,
    OPT_NO_TITLES
} appOptionsEnum;

static struct option appOptions[] = {
    {"fields",          REQUIRED_ARG, 0, OPT_FIELDS},
    {"summary",         NO_ARG,       0, OPT_SUMMARY},
    {"no-titles",       NO_ARG,       0, OPT_NO_TITLES},
    {0,0,0,0}           /* sentinel entry */
};

static const char *appHelp[] = {
    NULL, /* built dynamically */
    "Print a summary of total files, file sizes, and records",
    ("Do not print file names or field names; only print the\n"
     "\tvalues, one per line"),
    (char *)NULL /* sentinel entry */
};


/* LOCAL FUNCTION PROTOTYPES */

static int  appOptionsHandler(clientData cData, int opt_index, char *opt_arg);
static int  parseFields(const char *field_str);
static int  printFileInfo(
    const char *path,
    int64_t    *recs,
    int64_t    *bytes);


/* FUNCTION DEFINITIONS */

/*
 *  appUsageLong();
 *
 *    Print complete usage information to USAGE_FH.  Pass this
 *    function to skOptionsSetUsageCallback(); skOptionsParse() will
 *    call this funciton and then exit the program when the --help
 *    option is given.
 */
static void appUsageLong(void)
{
#define USAGE_MSG                                                       \
    ("[SWITCHES] <FILES>\n"                                             \
     "\tPrint information (type, version, etc.) about a SiLK Flow,\n"   \
     "\tIPset, or Bag file.  Use the fields switch to control what\n"   \
     "\tinformation is printed.\n")

    FILE *fh = USAGE_FH;
    int i, j;

    fprintf(fh, "%s %s", skAppName(), USAGE_MSG);
    fprintf(fh, "\nSWITCHES:\n");
    skOptionsDefaultUsage(fh);
    for (i = 0; appOptions[i].name; ++i) {
        fprintf(fh, "--%s %s. ", appOptions[i].name,
                SK_OPTION_HAS_ARG(appOptions[i]));
        switch (appOptions[i].val) {
          case OPT_FIELDS:
            fprintf(fh,
                    ("List of fields to print; specify by name or by value.\n"
                     "\tDef. All fields.  Available fields:"));
            for (j = 0; j < RWINFO_PROPERTY_COUNT; ++j) {
                fprintf(fh, "\n\t  %3d %s", 1+j, info_props[j].label);
            }
            break;

          default:
            fprintf(fh, "%s", appHelp[i]);
            break;
        }
        fprintf(fh, "\n");
    }
    skOptionsCtxOptionsUsage(optctx, fh);
    sksiteOptionsUsage(fh);
}


/*
 *  appTeardown()
 *
 *    Teardown all modules, close all files, and tidy up all
 *    application state.
 *
 *    This function is idempotent.
 */
static void appTeardown(void)
{
    static int teardownFlag = 0;

    if (teardownFlag) {
        return;
    }
    teardownFlag = 1;

    skOptionsCtxDestroy(&optctx);
    skAppUnregister();
}


/*
 *  appSetup(argc, argv);
 *
 *    Perform all the setup for this application include setting up
 *    required modules, parsing options, etc.  This function should be
 *    passed the same arguments that were passed into main().
 *
 *    Returns to the caller if all setup succeeds.  If anything fails,
 *    this function will cause the application to exit with a FAILURE
 *    exit status.
 */
static void appSetup(int argc, char **argv)
{
    SILK_FEATURES_DEFINE_STRUCT(features);
    int optctx_flags;
    int rv;

    /* verify same number of options and help strings */
    assert((sizeof(appHelp)/sizeof(char *)) ==
           (sizeof(appOptions)/sizeof(struct option)));

    /* register the application */
    skAppRegister(argv[0]);
    skAppVerifyFeatures(&features, NULL);
    skOptionsSetUsageCallback(&appUsageLong);

    optctx_flags = (SK_OPTIONS_CTX_INPUT_BINARY);

    /* register the options */
    if (skOptionsCtxCreate(&optctx, optctx_flags)
        || skOptionsCtxOptionsRegister(optctx)
        || skOptionsRegister(appOptions, &appOptionsHandler, NULL)
        || sksiteOptionsRegister(SK_SITE_FLAG_CONFIG_FILE))
    {
        skAppPrintErr("Unable to register options");
        exit(EXIT_FAILURE);
    }

    /* register the teardown handler */
    if (atexit(appTeardown) < 0) {
        skAppPrintErr("Unable to register appTeardown() with atexit()");
        appTeardown();
        exit(EXIT_FAILURE);
    }

    /* parse options */
    rv = skOptionsCtxOptionsParse(optctx, argc, argv);
    if (rv < 0) {
        skAppUsage();  /* never returns */
    }

    /* try to load the site file to resolve sensor information */
    sksiteConfigure(0);

    return;  /* OK */
}


/*
 *  status = appOptionsHandler(cData, opt_index, opt_arg);
 *
 *    Called by skOptionsParse(), this handles a user-specified switch
 *    that the application has registered, typically by setting global
 *    variables.  Returns 1 if the switch processing failed or 0 if it
 *    succeeded.  Returning a non-zero from from the handler causes
 *    skOptionsParse() to return a negative value.
 *
 *    The clientData in 'cData' is typically ignored; 'opt_index' is
 *    the index number that was specified as the last value for each
 *    struct option in appOptions[]; 'opt_arg' is the user's argument
 *    to the switch for options that have a REQUIRED_ARG or an
 *    OPTIONAL_ARG.
 */
static int appOptionsHandler(
    clientData  UNUSED(cData),
    int         opt_index,
    char       *opt_arg)
{
    switch ((appOptionsEnum)opt_index) {
      case OPT_FIELDS:
        if (parseFields(opt_arg)) {
            return 1;
        }
        break;

      case OPT_SUMMARY:
        print_summary = 1;
        break;

      case OPT_NO_TITLES:
        no_titles = 1;
        break;
    }

    return 0;
}


/*
 *  ok = parseFields(field_list);
 *
 *    Parse the user's field list, setting the appropriate flags in
 *    the info_props[] vector.
 */
static int parseFields(const char *field_str)
{
    sk_stringmap_t *str_map = NULL;
    sk_stringmap_status_t rv_map;
    sk_stringmap_entry_t *map_entry;
    sk_stringmap_entry_t insert_entry[2];
    sk_stringmap_iter_t *iter = NULL;
    char *fieldnum_string = NULL;
    char *cp;
    size_t len;
    int sz;
    uint32_t i;
    int rv = -1;

    /* create a char array to hold the field values as strings */
    len = 11 * RWINFO_PROPERTY_COUNT;
    fieldnum_string = (char*)calloc(len, sizeof(char));
    if (fieldnum_string == NULL) {
        skAppPrintErr("Unable to create string");
        goto END;
    }

    /* create a stringmap of the available entries */
    if (SKSTRINGMAP_OK != skStringMapCreate(&str_map)) {
        skAppPrintErr("Unable to create stringmap");
        goto END;
    }

    cp = fieldnum_string;
    for (i = 0; i < RWINFO_PROPERTY_COUNT; ++i) {
        sz = snprintf(cp, len, ("%" PRIu32), (uint32_t)(i+1));
        if ((size_t)sz > len) {
            skAppPrintErr("Internal buffer too small");
            skAbort();
        }
        memset(insert_entry, 0, sizeof(insert_entry));
        insert_entry[0].name = info_props[i].label;
        insert_entry[0].id = i;
        insert_entry[1].name = cp;
        insert_entry[1].id = i;
        if (skStringMapAddEntries(str_map, 2, insert_entry)
            != SKSTRINGMAP_OK)
        {
            goto END;
        }
        cp += sz + 1;
        len -= sz + 1;
    }

    /* attempt to match */
    rv_map = skStringMapParse(str_map, field_str, SKSTRINGMAP_DUPES_KEEP,
                              &iter, &cp);
    if (rv_map) {
        skAppPrintErr("Invalid %s: %s", appOptions[OPT_FIELDS].name, cp);
        goto END;
    }

    /* turn off printing for all fields */
    for (i = 0; i < RWINFO_PROPERTY_COUNT; ++i) {
        info_props[i].will_print = 0;
    }

    /* enable fields user listed */
    while (skStringMapIterNext(iter, &map_entry, NULL) == SK_ITERATOR_OK) {
        info_props[map_entry->id].will_print = 1;
    }

    rv = 0;

  END:
    if (str_map) {
        skStringMapDestroy(str_map);
    }
    if (iter) {
        skStringMapIterDestroy(iter);
    }
    if (fieldnum_string) {
        free(fieldnum_string);
    }
    return rv;
}


/*
 *  status = getNumberRecs(stream, record_size, &count);
 *
 *    Given 'stream' to the opened file, read the file to determine
 *    the number of 'record_size' records in the file, and set 'count'
 *    to that value.  If the file was successfully read, return 0.  If
 *    an error occurs while reading the file, return -1.
 */
static int getNumberRecs(
    skstream_t *stream,
    size_t      rec_size,
    int64_t    *count)
{
    size_t block_size = RWINFO_BLOCK_SIZE;
    int64_t bytes = 0;
    ssize_t saw;
    imaxdiv_t rec;
    int rv = 0;

    if (0 == rec_size) {
        rec_size = 1;
    }

    /* modify block_size to read an integer number of records */
    if (rec_size > block_size) {
        block_size = rec_size;
    } else {
        block_size -= block_size % rec_size;
    }

    /* get number of bytes in file */
    while ((saw = skStreamRead(stream, NULL, block_size)) > 0) {
        bytes += saw;
    }
    if (saw != 0) {
        skStreamPrintLastErr(stream, saw, &skAppPrintErr);
        rv = -1;
    }

    /* compute number of records */
    rec = imaxdiv(bytes, rec_size);
    if (rec.rem != 0) {
        skAppPrintErr("Short read (%" PRIdMAX "/%lu)",
                      rec.rem, (unsigned long)rec_size);
        rv = -1;
    }
    *count = (int64_t)rec.quot;
    return rv;
}


/*
 *  status = printFileInfo(info, &total_recs, &total_bytes);
 *
 *    Given the file information in the 'info' structure, print the
 *    fields requested by the user---given in the 'info_props'
 *    global---to the standard output.  Update the values pointed at
 *    by 'total_recs' and 'total_bytes' with the number of records and
 *    bytes in this file.  Return -1 if there is a problem opening or
 *    reading the file.  Return 0 otherwise.
 */
static int printFileInfo(
    const char *path,
    int64_t    *recs,
    int64_t    *bytes)
{
    char buf[1024];
    int count;
    int64_t rec_count;
    skstream_t *stream = NULL;
    sk_file_header_t *hdr;
    sk_header_entry_t *he;
    sk_hentry_iterator_t iter;
    int rv = SKSTREAM_OK;
    int retval = 0;

    if (SKSTREAM_OK == rv) {
        rv = skStreamCreate(&stream, SK_IO_READ, SK_CONTENT_SILK);
    }
    if (SKSTREAM_OK == rv) {
        rv = skStreamBind(stream, path);
    }
    if (SKSTREAM_OK == rv) {
        rv = skStreamOpen(stream);
    }
    if (SKSTREAM_OK == rv) {
        rv = skStreamReadSilkHeaderStart(stream);
    }

    /* Give up if we can't read the beginning of the silk header */
    if (rv != SKSTREAM_OK) {
        skStreamPrintLastErr(stream, rv, &skAppPrintErr);
        skStreamDestroy(&stream);
        return -1;
    }

    /* print file name */
    if (!no_titles) {
        printf("%s:\n", path);
    }

    /* read the header */
    rv = skStreamReadSilkHeader(stream, &hdr);
    switch (rv) {
      case SKSTREAM_OK:
        break;
      case SKHEADER_ERR_LEGACY:
        /* unrecognized file format.  disable printing of record
         * version and record size */
        info_props[RWINFO_HEADER_LENGTH].will_print = 0;
        info_props[RWINFO_RECORD_LENGTH].will_print = 0;
        info_props[RWINFO_RECORD_VERSION].will_print = 0;
        info_props[RWINFO_SILK_VERSION].will_print = 0;
        info_props[RWINFO_COUNT_RECORDS].will_print = 0;
        break;
      default:
        /* print an error but continue */
        skStreamPrintLastErr(stream, rv, &skAppPrintErr);
        retval = -1;
        break;
    }

    if (info_props[RWINFO_FORMAT].will_print) {
        sksiteFileformatGetName(buf, sizeof(buf),
                                skHeaderGetFileFormat(hdr));
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_FORMAT].label);
        }
        printf("%s(0x%02x)\n", buf, skHeaderGetFileFormat(hdr));
    }

    if (info_props[RWINFO_VERSION].will_print) {
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_VERSION].label);
        }
        printf("%u\n", skHeaderGetFileVersion(hdr));
    }

    if (info_props[RWINFO_BYTE_ORDER].will_print) {
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_BYTE_ORDER].label);
        }
        printf("%s\n", ((skHeaderGetByteOrder(hdr) == SILK_ENDIAN_BIG)
                        ? "BigEndian"
                        : "littleEndian"));
    }

    if (info_props[RWINFO_COMPRESSION].will_print) {
        sksiteCompmethodGetName(buf, sizeof(buf),
                                skHeaderGetCompressionMethod(hdr));
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_COMPRESSION].label);
        }
        printf("%s(%u)\n", buf, skHeaderGetCompressionMethod(hdr));
    }

    if (info_props[RWINFO_HEADER_LENGTH].will_print) {
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_HEADER_LENGTH].label);
        }
        printf("%u\n", (unsigned int)skHeaderGetLength(hdr));
    }

    if (info_props[RWINFO_RECORD_LENGTH].will_print) {
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_RECORD_LENGTH].label);
        }
        printf("%u\n", (unsigned int)skHeaderGetRecordLength(hdr));
    }

    if (info_props[RWINFO_RECORD_VERSION].will_print) {
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_RECORD_VERSION].label);
        }
        printf("%d\n", skHeaderGetRecordVersion(hdr));
    }

    if (info_props[RWINFO_SILK_VERSION].will_print) {
        uint32_t vers = skHeaderGetSilkVersion(hdr);
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_SILK_VERSION].label);
        }
        if (vers == 0) {
            printf("0\n");
        } else {
            printf(("%" PRId32 ".%" PRId32 ".%" PRId32 "\n"),
                   (vers / 1000000), (vers / 1000 % 1000), (vers % 1000));
        }
    }

    if (info_props[RWINFO_COUNT_RECORDS].will_print) {
        rv = getNumberRecs(stream, skHeaderGetRecordLength(hdr), &rec_count);
        if (rv) {
            retval = -1;
        }
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_COUNT_RECORDS].label);
        }
        printf(("%" PRId64 "\n"), rec_count);
        *recs += rec_count;
    }

    if (info_props[RWINFO_FILE_SIZE].will_print) {
        int64_t sz = (int64_t)skFileSize(path);
        if (!no_titles) {
            printf(LABEL_FMT, info_props[RWINFO_FILE_SIZE].label);
        }
        printf(("%" PRId64 "\n"), sz);
        *bytes += sz;
    }

    if (info_props[RWINFO_PACKED_FILE_INFO].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_PACKEDFILE_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (!no_titles) {
                if (count == 0) {
                    printf(LABEL_FMT,
                           info_props[RWINFO_PACKED_FILE_INFO].label);
                } else {
                    printf(LABEL_FMT, "");
                }
            }
            ++count;
            skHentryPackedfilePrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_PROBE_NAME].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_PROBENAME_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (!no_titles) {
                if (count == 0) {
                    printf(LABEL_FMT, info_props[RWINFO_PROBE_NAME].label);
                } else {
                    printf(LABEL_FMT, "");
                }
            }
            ++count;
            skHentryProbenamePrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_PREFIX_MAP].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_PREFIXMAP_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (!no_titles) {
                if (count == 0) {
                    printf(LABEL_FMT, info_props[RWINFO_PREFIX_MAP].label);
                } else {
                    printf(LABEL_FMT, "");
                }
            }
            ++count;
            skHentryPrefixmapPrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_IPSET].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_IPSET_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (!no_titles) {
                if (count == 0) {
                    printf(LABEL_FMT, info_props[RWINFO_IPSET].label);
                } else {
                    printf(LABEL_FMT, "");
                }
            }
            ++count;
            skHentryIPSetPrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_BAG].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_BAG_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (!no_titles) {
                if (count == 0) {
                    printf(LABEL_FMT, info_props[RWINFO_BAG].label);
                } else {
                    printf(LABEL_FMT, "");
                }
            }
            ++count;
            skHentryBagPrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_COMMAND_LINES].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_INVOCATION_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (count == 0 && !no_titles) {
                printf((LABEL_FMT "\n"),
                       info_props[RWINFO_COMMAND_LINES].label);
            }
            ++count;
            if (!no_titles) {
                printf(LABEL_NUM_FMT, count);
            }
            skHentryInvocationPrint(he, stdout);
            printf("\n");
        }
    }

    if (info_props[RWINFO_ANNOTIONS].will_print) {
        count = 0;
        skHeaderIteratorBindType(&iter, hdr, SK_HENTRY_ANNOTATION_ID);
        while ((he = skHeaderIteratorNext(&iter)) != NULL) {
            if (count == 0 && !no_titles) {
                printf((LABEL_FMT "\n"), info_props[RWINFO_ANNOTIONS].label);
            }
            ++count;
            if (!no_titles) {
                printf(LABEL_NUM_FMT, count);
            }
            skHentryAnnotationPrint(he, stdout);
            printf("\n");
        }
    }

    skStreamDestroy(&stream);
    return retval;
}


/*
 *  For each file, get the file's info then print it
 */
int main(int argc, char **argv)
{
    int64_t total_files = 0;
    int64_t total_bytes = 0;
    int64_t total_recs = 0;
    int rv = EXIT_SUCCESS;
    char *path;

    appSetup(argc, argv);       /* never returns on error */

    while (skOptionsCtxNextArgument(optctx, &path) == 0) {
        if (printFileInfo(path, &total_recs, &total_bytes)) {
            rv = EXIT_FAILURE;
        }
        ++total_files;
    }

    if (print_summary) {
        if (!no_titles) {
            printf("**SUMMARY**:\n");
            printf(LABEL_FMT, "number-files");
        }
            printf(("%" PRId64 "\n"), total_files);
        if (info_props[RWINFO_COUNT_RECORDS].will_print) {
            if (!no_titles) {
                printf(LABEL_FMT, "total-records");
            }
            printf(("%" PRId64 "\n"), total_recs);
        }
        if (info_props[RWINFO_FILE_SIZE].will_print) {
            if (!no_titles) {
                printf(LABEL_FMT, "all-file-sizes");
            }
            printf(("%" PRId64 "\n"), total_bytes);
        }
    }

    /* done */
    return rv;
}


/*
** Local Variables:
** mode:c
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael.

On Wed, Nov 14, 2012 at 11:52:41AM -0500, Michael Welsh Duggan wrote:

> ... Today I started working in C again, and ran into issues like this:

> Error during redisplay: (jit-lock-function 3411) signaled (wrong-type-argument number-or-marker-p nil)
> c-parse-state inconsistency at 3599: using cache: nil, from scratch: ((3409 . 3599))
> Old state:
> (setq c-state-cache '((3409 . 3599))  c-state-cache-good-pos 3599  c-state-nonlit-pos-cache '(6307 3307)  c-state-nonlit-pos-cache-limit 3592  c-state-semi-nonlit-pos-cache '(3307)  c-state-semi-nonlit-pos-cache-limit 3592  c-state-brace-pair-desert '(nil . 3396)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil  c-parse-state-point 3600)

> I have finally managed to come up with a recipe for this one.  I am
> using "110803 rgm@gnu.org-20121105111732-ilq2sbfo09xg9i9z" with your
> patches.

> emacs -Q rwfileinfo.c
> M-x c-toggle-parse-state-debug
> C-x 4 b *Messages* RET

I assumed a C-x o here.  ;-)

> C-s enum RET
> C-M-f C-p C-e ,

> Error occurs after the comma (should be on line 197).

I find myself on L96.  I can't reproduce this error, neither on a tty or
in X windows.  I tried repeating the C-M-f, so that the C-p C-e brought
point to just after "RWINFO_PROPERTY_COUNT" and typing a comma there.
That still didn't signal an error.

I've just committed the current state of the fixes to the Emacs-24 branch
at savannah, for the pretest next weekend.

Thanks again for all the input.

> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Michael.
>
> On Wed, Nov 14, 2012 at 11:52:41AM -0500, Michael Welsh Duggan wrote:
>
>> ... Today I started working in C again, and ran into issues like this:
>
>> Error during redisplay: (jit-lock-function 3411) signaled
>> (wrong-type-argument number-or-marker-p nil)
>> c-parse-state inconsistency at 3599: using cache: nil, from scratch:
>> ((3409 . 3599))
>> Old state:
>> (setq c-state-cache '((3409 . 3599)) c-state-cache-good-pos 3599
>> c-state-nonlit-pos-cache '(6307 3307) c-state-nonlit-pos-cache-limit
>> 3592 c-state-semi-nonlit-pos-cache '(3307)
>> c-state-semi-nonlit-pos-cache-limit 3592 c-state-brace-pair-desert
>> (nil . 3396) c-state-point-min 1 c-state-point-min-lit-type nil
>> c-state-point-min-lit-start nil c-state-min-scan-pos 1
>> c-state-old-cpp-beg nil c-state-old-cpp-end nil c-parse-state-point
>> 3600)
>
>> I have finally managed to come up with a recipe for this one.  I am
>> using "110803 rgm@gnu.org-20121105111732-ilq2sbfo09xg9i9z" with your
>> patches.
>
>> emacs -Q rwfileinfo.c
>> M-x c-toggle-parse-state-debug
>> C-x 4 b *Messages* RET
>
> I assumed a C-x o here.  ;-)
>
>> C-s enum RET
>> C-M-f C-p C-e ,
>
>> Error occurs after the comma (should be on line 197).
>
> I find myself on L96.  I can't reproduce this error, neither on a tty or
> in X windows.  I tried repeating the C-M-f, so that the C-p C-e brought
> point to just after "RWINFO_PROPERTY_COUNT" and typing a comma there.
> That still didn't signal an error.

I'm sorry.  I realized that I attached the wrong file.  And the line
number should have been 97, not 197.  Unfortunately, I cannot reliably
trigger the problem using the above recipe using a recent Emacs trunk
check-out.  Even more unfortunately, I have still encountered the
problem (but have been unable to recreate it).  All in all, though, I
think your changes have been beneficial.  I'll try to isolate an example
over the next few days.


[-- Attachment #2: rwsiteinfo.c --]
[-- Type: text/plain, Size: 61958 bytes --]

/*
** Copyright (C) 2011-2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  Prints information about SiLK site configurations
**
**  Michael Duggan
**  September 2011
*/


#include <silk/silk.h>

RCSIDENT("$Id$");

#include <silk/skstream.h>
#include <silk/skstringmap.h>
#include <silk/sksite.h>
#include <silk/utils.h>


/* LOCAL DEFINES AND TYPEDEFS */

/* where to write --help output */
#define USAGE_FH stdout

/*
 *  Printer function typedef.
 *
 *    rwsiteinfo "prints" the output twice.  In the first pass, it
 *    uses a print function that does not produce output but instead
 *    determines the sizes of what it would print.  In the second
 *    pass, it uses an actual print function (fprintf) to produce the
 *    output.
 */
typedef int (*rws_fprintf_t)(FILE *f, const char *fmt, ...);


/* Sub-iterator types */
typedef enum {
    RWS_NULL,
    RWS_FLOWTYPE,
    RWS_CLASS,
    RWS_SENSOR,
    RWS_CLASS_FROM_SENSOR,
    RWS_SENSOR_FROM_CLASS,
    RWS_FLOWTYPE_FROM_CLASS,
    RWS_DEFAULT_FLOWTYPE_FROM_CLASS
} rws_iter_type_t;

/* Site iterator */
typedef struct rws_iter_st {
    /* The iterators */
    flowtype_iter_t flowtype_iter;
    class_iter_t    class_iter;
    sensor_iter_t   sensor_iter;

    /* The values */
    flowtypeID_t    flowtype_id;
    classID_t       class_id;
    sensorID_t      sensor_id;

    /* Order and type of iterators */
    rws_iter_type_t order[3];
    /* Number of iterators  */
    int             level;
    /* Highest bound iterator */
    int             bound;
    /* Highest started iterator */
    int             started;

    /* Whether RWS_DEFAULT_FLOWTYPE_FROM_CLASS is one of the
     * iterators */
    unsigned        default_type : 1;
} rws_iter_t;


/* LOCAL VARIABLE DEFINITIONS */

/* Masks for flowtypes, classes, and sensors as set by the --classes,
 * --types, --flowtypes, and --sensors switches.  When the bitmap is
 * NULL, all values are printed; when the bitmap is not NULL, only
 * values where the bit is high are printed. */
static sk_bitmap_t *flowtype_mask = NULL;
static sk_bitmap_t *class_mask   = NULL;
static sk_bitmap_t *sensor_mask   = NULL;

/* paging program */
static const char *pager;

/* the output stream, set by --output-path or --pager */
static sk_fileptr_t output;

/* raw filter arguments */
static char *classes_arg   = NULL;
static char *types_arg     = NULL;
static char *flowtypes_arg = NULL;
static char *sensors_arg   = NULL;
static char *fields_arg    = NULL;

/* delimiters */
static char column_separator = '|';
static char list_separator   = ',';

/* how to format output */
static int no_columns         = 0;
static int no_final_delimiter = 0;
static int no_titles          = 0;

/* final delimiter */
static char final_delim[] = {'\0', '\0'};

/* Field types */
typedef enum {
    RWST_CLASS,
    RWST_TYPE,
    RWST_FLOWTYPE,
    RWST_FLOWTYPE_ID,
    RWST_SENSOR,
    RWST_SENSOR_ID,
    RWST_SENSOR_DESC,
    RWST_DEFAULT_CLASS,
    RWST_DEFAULT_TYPE,
    RWST_MARK_DEFAULTS,
    RWST_CLASS_LIST,
    RWST_TYPE_LIST,
    RWST_FLOWTYPE_LIST,
    RWST_FLOWTYPE_ID_LIST,
    RWST_SENSOR_LIST,
    RWST_SENSOR_ID_LIST,
    RWST_DEFAULT_CLASS_LIST,
    RWST_DEFAULT_TYPE_LIST,
    /* Number of field types */
    RWST_MAX_FIELD_COUNT
} rws_field_t;

/* Field names, types, descriptions, and titles.  MUST be in the same
 * order as the rws_field_t enumeration. */
static const sk_stringmap_entry_t field_map_entries[] = {
    {"class",              RWST_CLASS,
     "class name",                             "Class"},
    {"type",               RWST_TYPE,
     "type name",                              "Type"},
    {"flowtype",           RWST_FLOWTYPE,
     "flowtype name",                          "Flowtype"},
    {"id-flowtype",        RWST_FLOWTYPE_ID,
     "flowtype integer identifier",            "Flowtype-ID"},
    {"sensor",             RWST_SENSOR,
     "sensor name",                            "Sensor"},
    {"id-sensor",          RWST_SENSOR_ID,
     "sensor integer identifier",              "Sensor-ID"},
    {"describe-sensor",    RWST_SENSOR_DESC,
     "sensor description",                     "Sensor-Description"},
    {"default-class",      RWST_DEFAULT_CLASS,
     "default class name",                     "Default-Class"},
    {"default-type",       RWST_DEFAULT_TYPE,
     "default type name",                      "Default-Type"},
    {"mark-defaults",      RWST_MARK_DEFAULTS,
     "'+' for default classes, '*' for types", "Defaults"},
    {"class:list",         RWST_CLASS_LIST,
     "list of class names",                    "Class:list"},
    {"type:list",          RWST_TYPE_LIST,
     "list of type names",                     "Type:list"},
    {"flowtype:list",      RWST_FLOWTYPE_LIST,
     "list of flowtype names",                 "Flowtype:list"},
    {"id-flowtype:list",   RWST_FLOWTYPE_ID_LIST,
     "list of flowtype integer identifier",    "Flowtype-ID:list"},
    {"sensor:list",        RWST_SENSOR_LIST,
     "list of sensor names",                   "Sensor:list"},
    {"id-sensor:list",     RWST_SENSOR_ID_LIST,
     "list of sensor integer identifiers",     "Sensor-ID:list"},
    {"default-class:list", RWST_DEFAULT_CLASS_LIST,
     "list of default class names",            "Default-Class:list"},
    {"default-type:list",  RWST_DEFAULT_TYPE_LIST,
     "list of default type names",             "Default-Type:list"},
    SK_STRINGMAP_SENTINEL
};

/* Fields to print, in the order in which to print them */
static rws_field_t fields[RWST_MAX_FIELD_COUNT];

/* Number of fields in list */
static size_t num_fields = 0;

/* Width of the columns, where index is an rws_field_t. */
static int col_width[RWST_MAX_FIELD_COUNT];


/* OPTIONS SETUP */

typedef enum {
    OPT_FIELDS, OPT_CLASSES, OPT_TYPES, OPT_FLOWTYPES,
    OPT_SENSORS, OPT_NO_TITLES, OPT_NO_COLUMNS, OPT_COLUMN_SEPARATOR,
    OPT_NO_FINAL_DELIMITER, OPT_DELIMITED, OPT_LIST_DELIMETER,
    OPT_PAGER, OPT_DATA_ROOTDIR
} appOptionsEnum;

static struct option appOptions[] = {
    {"fields",              REQUIRED_ARG, 0, OPT_FIELDS},
    {"classes",             REQUIRED_ARG, 0, OPT_CLASSES},
    {"types",               REQUIRED_ARG, 0, OPT_TYPES},
    {"flowtypes",           REQUIRED_ARG, 0, OPT_FLOWTYPES},
    {"sensors",             REQUIRED_ARG, 0, OPT_SENSORS},
    {"no-titles",           NO_ARG,       0, OPT_NO_TITLES},
    {"no-columns",          NO_ARG,       0, OPT_NO_COLUMNS},
    {"column-separator",    REQUIRED_ARG, 0, OPT_COLUMN_SEPARATOR},
    {"no-final-delimiter",  NO_ARG,       0, OPT_NO_FINAL_DELIMITER},
    {"delimited",           OPTIONAL_ARG, 0, OPT_DELIMITED},
    {"list-delimiter",      REQUIRED_ARG, 0, OPT_LIST_DELIMETER},
    {"pager",               REQUIRED_ARG, 0, OPT_PAGER},
    {"data-rootdir",        REQUIRED_ARG, 0, OPT_DATA_ROOTDIR},
    {0,0,0,0}               /* sentinel entry */
};

static const char *appHelp[] = {
    ("Print the fields named in this comma-separated list. Choices:"),
    ("Restrict the output using classes named in this comma-\n"
     "\tseparated list. Use '@' to designate the default class.\n"
     "\tDef. Print data for all classes"),
    ("Restrict the output using the types named in this comma-\n"
     "\tseparated list. Use '@' to designate the default type(s) for a class.\n"
     "\tDef. Print data for all types"),
    ("Restrict the output using the class/type pairs named in\n"
     "\tthis comma-separated list. May use 'all' for class and/or type. This\n"
     "\tis an alternate way to specify class/type; switch may not be used\n"
     "\twith --class or --type. Def. Print data for all class/type pairs"),
    ("Restrict the output using the sensors named in this comma-\n"
     "\tseparated list. Sensors may be designated by name, ID (integer),\n"
     "\tand/or ranges of IDs. Def. Print data for all sensors"),
    ("Do not print column headers. Def. Print titles"),
    ("Disable fixed-width columnar output. Def. Columnar"),
    ("Use specified character between columns. Def. '|'"),
    ("Suppress column delimiter at end of line. Def. No"),
    ("Shortcut for --no-columns --no-final-del --column-sep=CHAR"),
    ("Use specified character between items in FIELD:list\n"
     "\tfields. Def. ','"),
    ("Program to invoke to page output. Def. $SILK_PAGER or $PAGER"),
    ("Root of directory tree containing packed data."),
    (char *)NULL
};



/* LOCAL FUNCTION PROTOTYPES */

static int  appOptionsHandler(clientData cData, int opt_index, char *opt_arg);
static sk_stringmap_t *createStringmap(void);
static int rws_parse_fields(void);
static int rws_parse_restrictions(void);
static int rws_print_list_field(
    rws_fprintf_t  printer,
    FILE          *fd,
    rws_iter_t    *iter,
    rws_field_t    field,
    int            width);
static int fprintf_size(FILE *stream, const char *format, ...);


/* FUNCTION DEFINITIONS */

/*
 *  appUsageLong();
 *
 *    Print complete usage information to USAGE_FH.  Pass this
 *    function to skOptionsSetUsageCallback(); skOptionsParse() will
 *    call this funciton and then exit the program when the --help
 *    option is given.
 */
static void appUsageLong(void)
{
#define MIN_TEXT_ON_LINE  15
#define MAX_TEXT_ON_LINE  72
#define USAGE_MSG                                                            \
    ("--fields=<FIELDS> [SWITCHES]\n"                                        \
     "\tPrint selected information about the classes, types, flowtypes\n"    \
     "\tand sensors defined in the SiLK site configuration file.  By\n"      \
     "\tdefault, the selected information is printed for every class,\n"     \
     "\ttype, and sensor defined in the file; to restrict the output,\n"     \
     "\tspecify one or more of --classes, --types, --flowtypes, or\n"        \
     "\t--sensors.\n")

    FILE *fh = USAGE_FH;
    char *cp, *ep, *sp;
    char buf[2 * PATH_MAX];
    char path[PATH_MAX];
    int i;

    fprintf(fh, "%s %s", skAppName(), USAGE_MSG);
    fprintf(fh, "\nSWITCHES:\n");
    skOptionsDefaultUsage(fh);

    for (i = 0; appOptions[i].name; i++ ) {
        fprintf(fh, "--%s %s. ",
                appOptions[i].name, SK_OPTION_HAS_ARG(appOptions[i]));
        /* Print the static help text from the appHelp array */
        fprintf(fh, "%s\n", appHelp[i]);
        switch (appOptions[i].val) {
          case OPT_FIELDS:
            {
                sk_stringmap_t *map = createStringmap();
                if (map == NULL) {
                    skAppPrintErr("Error creating string map");
                    exit(EXIT_FAILURE);
                }
                skStringMapPrintUsage(map, fh, 8);
                skStringMapDestroy(map);
            }
            break;
          case OPT_DATA_ROOTDIR:
            /* put the text into a buffer, and then wrap the text in
             * the buffer at space characters. */
            snprintf(buf, sizeof(buf),
                     ("Currently '%s'. Def. $" SILK_DATA_ROOTDIR_ENVAR
                      " or '%s'"),
                     sksiteGetRootDir(path, sizeof(path)),
                     sksiteGetDefaultRootDir());
            sp = buf;
            while (strlen(sp) > MAX_TEXT_ON_LINE) {
                cp = &sp[MIN_TEXT_ON_LINE];
                while ((ep = strchr(cp+1, ' ')) != NULL) {
                    /* text is now too long */
                    if (ep - sp > MAX_TEXT_ON_LINE) {
                        if (cp == &sp[MIN_TEXT_ON_LINE]) {
                            /* this is the first space character we have
                             * on this line; so use it */
                            cp = ep;
                        }
                        break;
                    }
                    cp = ep;
                }
                if (cp == &sp[MIN_TEXT_ON_LINE]) {
                    /* no space characters anywhere on the line */
                    break;
                }
                assert(' ' == *cp);
                *cp = '\0';
                fprintf(fh, "\t%s\n", sp);
                sp = cp + 1;
            }
            if (*sp) {
                fprintf(fh, "\t%s\n", sp);
            }
            break;
          default:
            break;
        }
    }

    sksiteOptionsUsage(fh);
}


/*
 *  appTeardown()
 *
 *    Teardown all modules, close all files, and tidy up all
 *    application state.
 *
 *    This function is idempotent.
 */
static void appTeardown(void)
{
    static int teardownFlag = 0;

    if (teardownFlag) {
        return;
    }
    teardownFlag = 1;

    /* close the output file or process */
    if (output.of_name) {
        skFileptrClose(&output, &skAppPrintErr);
    }

    skBitmapDestroy(&flowtype_mask);
    skBitmapDestroy(&class_mask);
    skBitmapDestroy(&sensor_mask);

    skAppUnregister();
}


/*
 *  appSetup(argc, argv);
 *
 *    Perform all the setup for this application include setting up
 *    required modules, parsing options, etc.  This function should be
 *    passed the same arguments that were passed into main().
 *
 *    Returns to the caller if all setup succeeds.  If anything fails,
 *    this function will cause the application to exit with a FAILURE
 *    exit status.
 */
static void appSetup(int argc, char **argv)
{
    SILK_FEATURES_DEFINE_STRUCT(features);
    int arg_index;
    int rv;

    /* verify same number of options and help strings */
    assert((sizeof(appHelp)/sizeof(char *)) ==
           (sizeof(appOptions)/sizeof(struct option)));

    /* register the application */
    skAppRegister(argv[0]);
    skAppVerifyFeatures(&features, NULL);
    skOptionsSetUsageCallback(&appUsageLong);

    /* initialize globals */
    memset(&output, 0, sizeof(output));
    output.of_fp = stdout;

    /* register the options */
    if (skOptionsRegister(appOptions, &appOptionsHandler, NULL)
        || sksiteOptionsRegister(SK_SITE_FLAG_CONFIG_FILE))
    {
        skAppPrintErr("Unable to register options");
        exit(EXIT_FAILURE);
    }

    /* register the teardown handler */
    if (atexit(appTeardown) < 0) {
        skAppPrintErr("Unable to register appTeardown() with atexit()");
        appTeardown();
        exit(EXIT_FAILURE);
    }

    /* parse the options */
    arg_index = skOptionsParse(argc, argv);
    if (arg_index < 0) {
        skAppUsage();           /* never returns */
    }

    /* try to load site config file; if it fails, we will not be able
     * to resolve flowtype and sensor from input file names */
    if (sksiteConfigure(1)) {
        exit(EXIT_FAILURE);
    }

    /* parse fields */
    if (rws_parse_fields()) {
        exit(EXIT_FAILURE);
    }

    /* parse restrictions (--classes, --types, etc) */
    if (rws_parse_restrictions()) {
        exit(EXIT_FAILURE);
    }

    /* check for extraneous arguments */
    if (arg_index != argc) {
        skAppPrintErr("Too many arguments or unrecognized switch '%s'",
                      argv[arg_index]);
        skAppUsage();           /* never returns */
    }

    /* Initialize column widths */
    if (no_titles || no_columns) {
        memset(col_width, 0, sizeof(col_width));
    } else {
        size_t i;
        for (i = 0; i < RWST_MAX_FIELD_COUNT; i++) {
            col_width[i] = strlen((char *)field_map_entries[i].userdata);

            /* While looping through, verify that the
             * field_map_entries is in the same order as the
             * enumeration. */
            assert(i == field_map_entries[i].id);
        }
    }

    /* Set the final delimiter, if used */
    if (!no_final_delimiter) {
        final_delim[0] = column_separator;
    }

    rv = skFileptrOpenPager(&output, pager);
    if (rv && rv != SK_FILEPTR_PAGER_IGNORED) {
        skAppPrintErr("Unable to invoke pager");
    }

    return;  /* OK */
}


/*
 *  status = appOptionsHandler(cData, opt_index, opt_arg);
 *
 *    This function is passed to skOptionsRegister(); it will be called
 *    by skOptionsParse() for each user-specified switch that the
 *    application has registered; it should handle the switch as
 *    required---typically by setting global variables---and return 1
 *    if the switch processing failed or 0 if it succeeded.  Returning
 *    a non-zero from from the handler causes skOptionsParse() to return
 *    a negative value.
 *
 *    The clientData in 'cData' is typically ignored; 'opt_index' is
 *    the index number that was specified as the last value for each
 *    struct option in appOptions[]; 'opt_arg' is the user's argument
 *    to the switch for options that have a REQUIRED_ARG or an
 *    OPTIONAL_ARG.
 */
static int appOptionsHandler(
    clientData  UNUSED(cData),
    int         opt_index,
    char       *opt_arg)
{
    switch ((appOptionsEnum)opt_index) {
      case OPT_DATA_ROOTDIR:
        if (!skDirExists(opt_arg)) {
            skAppPrintErr("Root data directory '%s' does not exist", opt_arg);
            return -1;
        }
        if (sksiteSetRootDir(opt_arg)) {
            skAppPrintErr("Unable to set root data directory to %s", opt_arg);
            return -1;
        }
        break;

      case OPT_CLASSES:
        if (classes_arg != NULL) {
            skAppPrintErr("Invalid %s: Switch specified multiple times",
                          appOptions[opt_index].name);
            return -1;
        }
        classes_arg = opt_arg;
        break;

      case OPT_TYPES:
        if (types_arg != NULL) {
            skAppPrintErr("Invalid %s: Switch specified multiple times",
                          appOptions[opt_index].name);
            return -1;
        }
        types_arg = opt_arg;
        break;

      case OPT_FLOWTYPES:
        if (flowtypes_arg != NULL) {
            skAppPrintErr("Invalid %s: Switch specified multiple times",
                          appOptions[opt_index].name);
            return -1;
        }
        flowtypes_arg = opt_arg;
        break;

      case OPT_SENSORS:
        if (sensors_arg != NULL) {
            skAppPrintErr("Invalid %s: Switch specified multiple times",
                          appOptions[opt_index].name);
            return -1;
        }
        sensors_arg = opt_arg;
        break;

      case OPT_FIELDS:
        if (fields_arg != NULL) {
            skAppPrintErr("Invalid %s: Switch specified multiple times",
                          appOptions[opt_index].name);
            return -1;
        }
        fields_arg = opt_arg;
        break;

      case OPT_NO_TITLES:
        no_titles = 1;
        break;

      case OPT_NO_COLUMNS:
        no_columns = 1;
        break;

      case OPT_COLUMN_SEPARATOR:
        column_separator = opt_arg[0];
        break;

      case OPT_NO_FINAL_DELIMITER:
        no_final_delimiter = 1;
        break;

      case OPT_DELIMITED:
        no_columns = 1;
        no_final_delimiter = 1;
        if (opt_arg) {
            column_separator = opt_arg[0];
        }
        break;

      case OPT_LIST_DELIMETER:
        list_separator = opt_arg[0];
        break;

      case OPT_PAGER:
        pager = opt_arg;
        break;
    }

    return 0;  /* OK */
}


/*
 *  stringmap = createStringmap();
 *
 *    Create the string map that is used to parse the --fields
 *    paramater.
 */
static sk_stringmap_t *createStringmap(void)
{
    sk_stringmap_t *field_map;

    /* Create the map */
    if (SKSTRINGMAP_OK != skStringMapCreate(&field_map)) {
        return NULL;
    }

    /* add entries */
    if (skStringMapAddEntries(field_map, -1, field_map_entries)
        != SKSTRINGMAP_OK)
    {
        skStringMapDestroy(field_map);
        return NULL;
    }

    return field_map;
}


/*
 *  status = rws_parse_fields();
 *
 *     Parse the --fields argument.  Return 0 on success, -1 on
 *     failure.
 */
static int rws_parse_fields(void)
{
    sk_stringmap_t *field_map = NULL;
    sk_stringmap_iter_t *iter = NULL;
    sk_stringmap_entry_t *entry;
    char *errmsg;
    int rv = -1;

    if (fields_arg == NULL) {
        skAppPrintErr("The --%s switch is required",
                      appOptions[OPT_FIELDS].name);
        return rv;
    }
    field_map = createStringmap();
    if (NULL == field_map) {
        skAppPrintOutOfMemory(NULL);
        goto END;
    }

    /* parse the field-list */
    if (skStringMapParse(field_map, fields_arg, SKSTRINGMAP_DUPES_ERROR,
                         &iter, &errmsg))
    {
        skAppPrintErr("Invalid %s: %s",
                      appOptions[OPT_FIELDS].name, errmsg);
        goto END;
    }

    /* add the selected fields to the 'fields[]' array */
    while (skStringMapIterNext(iter, &entry, NULL) == SK_ITERATOR_OK) {
        assert(num_fields < RWST_MAX_FIELD_COUNT);
        fields[num_fields] = (rws_field_t)entry->id;
         ++num_fields;
    }

    rv = 0;

  END:
    if (iter != NULL) {
        skStringMapIterDestroy(iter);
    }
    if (field_map != NULL) {
        skStringMapDestroy(field_map);
    }
    return rv;
}


/*
 *  status = rws_parse_sensors(sn_bitmap);
 *
 *      Parse the --sensors argument from the global 'sensors_arg'.
 *      Set a bit in 'sn_bitmap' for each sensor we see.  Return 0 on
 *      success, or -1 if any invalid sensors are found.
 */
static int rws_parse_sensors(
    sk_bitmap_t    *sn_bitmap)
{
    char *user_arg = NULL;
    char *user_arg_freeable = NULL;
    char *sensor_token = NULL;
    uint32_t min_sensor_id;
    uint32_t max_sensor_id;
    uint32_t val_min, val_max;
    sensorID_t sid = SK_INVALID_SENSOR;
    int p_err;
    int rv = 0;

    /* nothing to do when no --sensors switch is given */
    if (NULL == sensors_arg) {
        return 0;
    }

    assert(sn_bitmap);
    assert(skBitmapGetSize(sn_bitamp) > sksiteSensorGetMaxID());

    min_sensor_id = sksiteSensorGetMinID();
    max_sensor_id = sksiteSensorGetMaxID();

    /* create a copy of the input string and maintain a reference to
     * it so we can free it */
    user_arg = strdup(sensors_arg);
    user_arg_freeable = user_arg;
    if (user_arg == NULL) {
        skAppPrintOutOfMemory(NULL);
        rv = -1;
        goto END;
    }

    /* parse the sensors as a comma separated list of tokens */
    while ((sensor_token = strsep(&user_arg, ",")) != NULL) {
        /* check for empty token (e.g., double comma) */
        if ('\0' == *sensor_token) {
            continue;
        }

        /* look up sensor_token as a sensor name */
        sid = sksiteSensorLookup(sensor_token);
        if (SK_INVALID_SENSOR != sid) {
            /* found it. add it to our lists and continue */
            skBitmapSetBit(sn_bitmap, sid);
            continue;
        }

        /* parsing failed. does the token look like a number? */
        if (!isdigit((int)(*sensor_token))) {
            /* not a number: error */
            skAppPrintErr("Invalid %s '%s': Unknown sensor name",
                          appOptions[OPT_SENSORS].name, sensor_token);
            rv = -1;
            continue;
        }

        /* it is a digit: parse the token as a single number or a
         * range */
        p_err = skStringParseRange32(&val_min, &val_max, sensor_token,
                                     min_sensor_id, max_sensor_id,
                                     SKUTILS_RANGE_NO_OPEN);
        if (p_err != 0) {
            /* an error */
            skAppPrintErr("Invalid %s '%s': %s",
                          appOptions[OPT_SENSORS].name, sensor_token,
                          skStringParseStrerror(p_err));
            rv = -1;
            continue;
        }

        /* verify that start of range is valid */
        sid = (sensorID_t)val_min;
        if (!sksiteSensorExists(sid)) {
            skAppPrintErr(("Invalid %s: Value %" PRIu32
                           " is not a valid sensor id"),
                          appOptions[OPT_SENSORS].name, val_min);
            rv = -1;
            continue;
        }

        /* verify that end of range is valid */
        if (val_min != val_max && !sksiteSensorExists((sensorID_t)val_max)) {
            skAppPrintErr(("Invalid %s: Value %" PRIu32
                           " is not a valid sensor id"),
                          appOptions[OPT_SENSORS].name, val_max);
            rv = -1;
            continue;
        }

        /* add all sensor IDs in range */
        skBitmapRangeSet(sn_bitmap, sid, val_max);
    }

    /* did we find any sensors? */
    if ((0 == rv) && (0 == skBitmapGetHighCount(sn_bitmap))) {
        skAppPrintErr("Invalid %s: No valid sensors",
                      appOptions[OPT_SENSORS].name);
        rv = -1;
    }

  END:
    if (user_arg_freeable) {
        free(user_arg_freeable);
    }
    return rv;
}


/*
 *  status = rws_parse_flowtypes(cl_bitmap, ft_bitmap);
 *
 *    Parse the --flowtypes argument from the global 'flowtypes_arg'.
 *    Set a bit on 'cl_bitmap' for each valid class and and a bit on
 *    'ft_bitmap' for each valid flowtype.  Return 0 on success, or -1
 *    on if any class/type value is not a valid pair.
 */
static int rws_parse_flowtypes(
    sk_bitmap_t    *cl_bitmap,
    sk_bitmap_t    *ft_bitmap)
{
    char *user_arg = NULL;
    char *user_arg_freeable = NULL;
    char *class_name;
    char *type_name;
    flowtype_iter_t ft_iter;
    flowtypeID_t ft;
    classID_t class_id;
    int rv = 0;

    /* nothing to do when no --flowtypes switch is given */
    if (NULL == flowtypes_arg) {
        return 0;
    }

    /* create a modifiable version of the user's input. store a
     * pointer to it we can free it */
    user_arg = strdup(flowtypes_arg);
    user_arg_freeable = user_arg;
    if (NULL == user_arg) {
        skAppPrintOutOfMemory(NULL);
        rv = -1;
        goto END;
    }

    /* parse user's string as a comma separated list */
    while ((class_name = strsep(&user_arg, ",")) != NULL) {
        if (class_name[0] == '\0') {
            /* empty token (e.g., double comma) */
            continue;
        }

        /* break token into class and type separated by '/' */
        type_name = strchr(class_name, '/');
        if (type_name == NULL) {
            skAppPrintErr("Invalid %s: Missing '/' in token '%s'",
                          appOptions[OPT_FLOWTYPES].name, class_name);
            rv = -1;
            continue;
        }
        *type_name = '\0';
        ++type_name;

        /* find class and type.  if lookup fails, test for special
         * "all" keyword */
        ft = sksiteFlowtypeLookupByClassType(class_name, type_name);
        if (SK_INVALID_FLOWTYPE != ft) {
            /* Yay!  Class and type are specific */
            skBitmapSetBit(ft_bitmap, ft);
            skBitmapSetBit(cl_bitmap, sksiteFlowtypeGetClassID(ft));

        } else if (0 == strcmp(class_name, "all")) {
            if (0 == strcmp(type_name, "all")) {
                /* Use all classes and all types. */
                sksiteFlowtypeIterator(&ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
                    skBitmapSetBit(ft_bitmap, ft);
                    skBitmapSetBit(cl_bitmap, sksiteFlowtypeGetClassID(ft));
                }
            } else {
                /* Loop over all classes and add flowtype if type_name
                 * is valid for that class.  Don't complain unless the
                 * type is not valid for any class. */
                class_iter_t ci;
                int found_type = 0;

                sksiteClassIterator(&ci);
                while (sksiteClassIteratorNext(&ci, &class_id)) {
                    ft = sksiteFlowtypeLookupByClassIDType(class_id,type_name);
                    if (SK_INVALID_FLOWTYPE != ft) {
                        ++found_type;
                        skBitmapSetBit(ft_bitmap, ft);
                        skBitmapSetBit(cl_bitmap, class_id);
                    }
                }
                if (!found_type) {
                    skAppPrintErr(("Invalid %s:"
                                   " Type '%s' not valid for any class"),
                                  appOptions[OPT_FLOWTYPES].name, type_name);
                    rv = -1;
                }
            }

        } else if (0 == strcmp(type_name, "all")) {
            /* Use all types in the specified class */
            class_id = sksiteClassLookup(class_name);
            if (SK_INVALID_CLASS == class_id) {
                skAppPrintErr("Invalid %s: Invalid class '%s'",
                              appOptions[OPT_FLOWTYPES].name, class_name);
                rv = -1;
            } else {
                skBitmapSetBit(cl_bitmap, class_id);
                sksiteClassFlowtypeIterator(class_id, &ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
                    skBitmapSetBit(ft_bitmap, ft);
                }
            }

        } else {
            /* Invalid class/type */
            skAppPrintErr("Invalid %s: Unknown class/type pair '%s/%s'",
                          appOptions[OPT_FLOWTYPES].name,
                          class_name, type_name);
            rv = -1;
        }
    }

    if ((rv == 0) && (skBitmapGetHighCount(ft_bitmap) == 0)) {
        skAppPrintErr("Invalid %s: No valid class/type pairs",
                      appOptions[OPT_FLOWTYPES].name);
        rv = -1;
    }

  END:
    if (user_arg_freeable) {
        free(user_arg_freeable);
    }
    return rv;
}


/*
 *  status = rws_parse_classes_and_types();
 *
 *    Parse the --classes and/or --types arguments from the globals
 *    'classes_arg' and 'types_arg'.  Set a bit on 'cl_bitmap' for
 *    each valid class and and a bit on 'ft_bitmap' for each valid
 *    flowtype.  Return 0 on success, or -1 on if any class/type value
 *    is not a valid pair.
 */
static int rws_parse_classes_and_types(
    sk_bitmap_t    *cl_bitmap,
    sk_bitmap_t    *ft_bitmap)
{
    sk_bitmap_iter_t bmap_iter;
    uint32_t bmap_val;
    class_iter_t ci;
    flowtype_iter_t ft_iter;
    char *user_arg = NULL;
    char *user_arg_freeable = NULL;
    char *class_token = NULL;
    char *type_token = NULL;
    classID_t class_id;
    flowtypeID_t ft;
    int found_type;
    int rv = 0;

    if (NULL == classes_arg) {
        /* temporarily enable all classes */
        sksiteClassIterator(&ci);
        while (sksiteClassIteratorNext(&ci, &class_id)) {
            skBitmapSetBit(cl_bitmap, class_id);
        }

    } else {
        /* create a copy of the input string and maintain a reference to
         * it so we can free it */
        user_arg = strdup(classes_arg);
        user_arg_freeable = user_arg;
        if (user_arg == NULL) {
            skAppPrintOutOfMemory(NULL);
            rv = -1;
            goto END;
        }

        /* parse the classes as a comma separated list of tokens */
        while ((class_token = strsep(&user_arg, ",")) != NULL) {
            /* check for empty token (e.g., double comma) */
            if ('\0' == *class_token) {
                continue;
            }

            /* Handle default class token */
            if (strcmp(class_token, "@") == 0) {
                skBitmapSetBit(cl_bitmap, sksiteClassGetDefault());
                continue;
            }

            /* look up class_token as a class name */
            class_id = sksiteClassLookup(class_token);
            if (SK_INVALID_CLASS != class_id) {
                /* found it. add it to our lists and continue */
                skBitmapSetBit(cl_bitmap, class_id);
                continue;
            }

            skAppPrintErr("Invalid %s '%s': Unknown class name",
                          appOptions[OPT_CLASSES].name, class_token);
            rv = -1;
            continue;
        }

        free(user_arg_freeable);
        user_arg_freeable = NULL;

        /* did we find any classes? */
        if ((0 == rv) && (0 == skBitmapGetHighCount(cl_bitmap))) {
            skAppPrintErr("Invalid %s: No valid classes",
                          appOptions[OPT_CLASSES].name);
            rv = -1;
        }

        if (NULL == types_arg) {
            if (rv != 0) {
                goto END;
            }
            /* there is no --types; enable all flowtypes that exist in
             * the specified class(es) */
            skBitmapIteratorBind(cl_bitmap, &bmap_iter);
            while (skBitmapIteratorNext(&bmap_iter, &bmap_val)==SK_ITERATOR_OK)
            {
                class_id = (classID_t)bmap_val;
                sksiteClassFlowtypeIterator(class_id, &ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
                    skBitmapSetBit(ft_bitmap, ft);
                }
            }
            goto END;
        }
    }

    /* create a modifiable version of the user's input. store a
     * pointer to it we can free */
    user_arg = strdup(types_arg);
    user_arg_freeable = user_arg;
    if (NULL == user_arg) {
        skAppPrintOutOfMemory(NULL);
        rv = -1;
        goto END;
    }

    /* parse user's string as a comma separated list */
    while ((type_token = strsep(&user_arg, ",")) != NULL) {
        found_type = 0;
        if (type_token[0] == '\0') {
            /* empty token (e.g., double comma) */
            continue;
        }

        /* check for the type in all the classes we found above */
        skBitmapIteratorBind(cl_bitmap, &bmap_iter);
        while (skBitmapIteratorNext(&bmap_iter, &bmap_val)==SK_ITERATOR_OK) {
            class_id = (classID_t)bmap_val;

            /* find class and type */
            ft = sksiteFlowtypeLookupByClassIDType(class_id, type_token);
            if (SK_INVALID_FLOWTYPE != ft) {
                /* Yay!  Class and type are specific */
                skBitmapSetBit(ft_bitmap, ft);
                ++found_type;

            } else if (0 == strcmp(type_token, "all")) {
                /* Use all types in the specified class */
                sksiteClassFlowtypeIterator(class_id, &ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
                    ++found_type;
                    skBitmapSetBit(ft_bitmap, ft);
                }

            } else if (0 == strcmp(type_token, "@")) {
                sksiteClassDefaultFlowtypeIterator(class_id, &ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &ft)) {
                    ++found_type;
                    skBitmapSetBit(ft_bitmap, ft);
                }
            }
        }

        if (!found_type) {
            skAppPrintErr("Invalid %s: Type '%s' not valid for %s",
                          appOptions[OPT_TYPES].name, type_token,
                          (classes_arg ? "specified classes" : "any class"));
            rv = -1;
        }
    }

    if ((rv == 0) && (skBitmapGetHighCount(ft_bitmap) == 0)) {
        skAppPrintErr("Invalid --%s: No valid types",
                      appOptions[OPT_TYPES].name);
        rv = -1;
    }

    if ((rv == 0) && (NULL == classes_arg)) {
        /* no --classes were specified.  reset the cl_bitmap based on
         * the flowtypes we found */
        skBitmapClearAllBits(cl_bitmap);

        skBitmapIteratorBind(ft_bitmap, &bmap_iter);
        while (skBitmapIteratorNext(&bmap_iter, &bmap_val)==SK_ITERATOR_OK) {
            ft = (flowtypeID_t)bmap_val;
            skBitmapSetBit(cl_bitmap, sksiteFlowtypeGetClassID(ft));
        }
    }

  END:
    if (user_arg_freeable) {
        free(user_arg_freeable);
    }
    return rv;
}


/*
 *  status = rws_parse_restrictions();
 *
 *    Parse the --classes, --types, --flowtypes, and --sensors options
 *    and create/fill bitmaps that restrict the output.
 *
 *    Returns 0 on success, -1 on error.
 */
static int rws_parse_restrictions(void)
{
    sk_bitmap_iter_t bmap_iter;
    uint32_t bmap_val;
    sk_bitmap_t *cl_mask = NULL;
    sk_bitmap_t *ft_mask = NULL;
    sk_bitmap_t *sn_mask = NULL;
    class_iter_t class_iter;
    flowtype_iter_t ft_iter;
    sensor_iter_t sensor_iter;
    classID_t class_id;
    sensorID_t sensor_id;
    flowtypeID_t flowtype_id;
    int sensors_only = 0;
    int rv = 0;

    if (!classes_arg && !types_arg && !flowtypes_arg) {
        if (!sensors_arg) {
            /* nothing to do */
            return 0;
        }
        /* else, only need to process --sensors */
        sensors_only = 1;
    }

    /* create the global bitmaps for all classes, all flowtypes, and
     * all sensors */
    if (skBitmapCreate(&class_mask, 1 + sksiteClassGetMaxID())) {
        skAppPrintOutOfMemory("class bitmap");
        return -1;
    }
    if (skBitmapCreate(&flowtype_mask, 1 + sksiteFlowtypeGetMaxID())) {
        skAppPrintOutOfMemory("flowtype bitmap");
        return -1;
    }
    if (skBitmapCreate(&sensor_mask, 1 + sksiteSensorGetMaxID())) {
        skAppPrintOutOfMemory("sensor bitmap");
        return -1;
    }

    if (!sensors_only && sensors_arg) {
        /* need to create temporary bitmaps */
        if (skBitmapCreate(&cl_mask, 1 + sksiteClassGetMaxID())) {
            skAppPrintOutOfMemory("class bitmap");
            rv = -1;
            goto END;
        }
        if (skBitmapCreate(&ft_mask, 1 + sksiteFlowtypeGetMaxID())) {
            skAppPrintOutOfMemory("flowtype bitmap");
            rv = -1;
            goto END;
        }
        if (skBitmapCreate(&sn_mask, 1 + sksiteSensorGetMaxID())) {
            skAppPrintOutOfMemory("sensor bitmap");
            rv = -1;
            goto END;
        }
    } else {
        /* else we can point the temporaries at the real thing */
        cl_mask = class_mask;
        ft_mask = flowtype_mask;
        sn_mask = sensor_mask;
    }

    if (sensors_arg) {
        rv = rws_parse_sensors(sensor_mask);

        if (sensors_only && rv != 0) {
            goto END;
        }

        /* set class_mask and flowtype_mask based on the sensors we saw */
        skBitmapIteratorBind(sensor_mask, &bmap_iter);
        while (skBitmapIteratorNext(&bmap_iter, &bmap_val)==SK_ITERATOR_OK) {
            sensor_id = (sensorID_t)bmap_val;
            sksiteSensorClassIterator(sensor_id, &class_iter);
            while (sksiteClassIteratorNext(&class_iter, &class_id)) {
                skBitmapSetBit(cl_mask, class_id);
                sksiteClassFlowtypeIterator(class_id, &ft_iter);
                while (sksiteFlowtypeIteratorNext(&ft_iter, &flowtype_id)) {
                    skBitmapSetBit(ft_mask, flowtype_id);
                }
            }
        }

        if (sensors_only) {
            return 0;
        }
    }

    /* handle case when --flowtypes is given */
    if (flowtypes_arg) {
        if (classes_arg || types_arg) {
            skAppPrintErr(("Cannot use --%s when either --%s or --%s is"
                           " specified"),
                          appOptions[OPT_FLOWTYPES].name,
                          appOptions[OPT_CLASSES].name,
                          appOptions[OPT_TYPES].name);
            goto END;
        }
        rv |= rws_parse_flowtypes(class_mask, flowtype_mask);
        if (rv != 0) {
            goto END;
        }
    } else {
        assert(classes_arg || types_arg);
        rv |= rws_parse_classes_and_types(class_mask, flowtype_mask);
        if (rv != 0) {
            goto END;
        }
    }

    /* set sensor_mask based on the classes we saw */
    skBitmapIteratorBind(class_mask, &bmap_iter);
    while (skBitmapIteratorNext(&bmap_iter, &bmap_val)==SK_ITERATOR_OK) {
        class_id = (classID_t)bmap_val;
        sksiteClassSensorIterator(class_id, &sensor_iter);
        while (sksiteSensorIteratorNext(&sensor_iter, &sensor_id)) {
            skBitmapSetBit(sn_mask, sensor_id);
        }
    }

    /* perform the intersection of the masks with the temporaries */
    if (sn_mask && sn_mask != sensor_mask) {
        skBitmapIntersection(sensor_mask, sn_mask);
        skBitmapDestroy(&sn_mask);
    }
    if (cl_mask && cl_mask != class_mask) {
        skBitmapIntersection(class_mask, cl_mask);
        skBitmapDestroy(&cl_mask);
    }
    if (ft_mask && ft_mask != flowtype_mask) {
        skBitmapIntersection(flowtype_mask, ft_mask);
        skBitmapDestroy(&ft_mask);
    }

  END:
    if (sn_mask && sn_mask != sensor_mask) {
        skBitmapDestroy(&sn_mask);
    }
    if (cl_mask && cl_mask != class_mask) {
        skBitmapDestroy(&cl_mask);
    }
    if (ft_mask && ft_mask != flowtype_mask) {
        skBitmapDestroy(&ft_mask);
    }
    return rv;
}


/*
 *  rws_iter_bind(&iter, level);
 *
 *    Bind the sub-iterator of a site iterator at the given level.
 */
static void rws_iter_bind(
    rws_iter_t *iter,
    int         level)
{
    assert(iter);
    assert(iter->level >= level);

    /* Negative level is for a non-iterable iterator.  This is used
     * for options that require no iteration, like
     * --fields=class-default. */
    if (level >= 0) {
        switch (iter->order[level]) {
          case RWS_FLOWTYPE:
            sksiteFlowtypeIterator(&iter->flowtype_iter);
            break;
          case RWS_CLASS:
            sksiteClassIterator(&iter->class_iter);
            break;
          case RWS_SENSOR:
            sksiteSensorIterator(&iter->sensor_iter);
            break;
          case RWS_FLOWTYPE_FROM_CLASS:
            sksiteClassFlowtypeIterator(iter->class_id, &iter->flowtype_iter);
            break;
          case RWS_CLASS_FROM_SENSOR:
            sksiteSensorClassIterator(iter->sensor_id, &iter->class_iter);
            break;
          case RWS_SENSOR_FROM_CLASS:
            sksiteClassSensorIterator(iter->class_id, &iter->sensor_iter);
            break;
          case RWS_DEFAULT_FLOWTYPE_FROM_CLASS:
            sksiteClassDefaultFlowtypeIterator(
                iter->class_id, &iter->flowtype_iter);
            iter->default_type = 1;
            break;
          case RWS_NULL:
            skAbortBadCase(iter->order[level]);
            break;
        }
    }

    /* We are now bound at this level */
    iter->bound = level;

    /* But mark as not having stared iteration yet */
    iter->started = level - 1;
}


/*
 *  status = rws_iter_next(iter, level);
 *
 *    Site iterator iteration, at a particular sub-iterator level.
 *    Returns 1 on success, 0 if the iteration is complete at the
 *    given level.
 */
static int rws_iter_next(
    rws_iter_t *iter,
    int         level)
{
    int rv;

    assert(iter);
    assert(iter->level >= level);

    /* Prevent re-iteration after already having completed
     * iteration */
    if (iter->bound < level) {
        assert(level == 0);
        return 0;
    }

    do {
        /* Iterate at the next level if we've already started
         * iteration on this level */
        if (iter->started >= level && level < iter->level) {
            if (iter->bound == level) {
                rws_iter_bind(iter, level + 1);
            }
            rv = rws_iter_next(iter, level + 1);
            if (rv != 0) {
                /* Leaf iter succeeded */
                return 1;
            }
        }

        /* Iterate at the current level */
        switch (iter->order[level]) {
          case RWS_FLOWTYPE:
          case RWS_FLOWTYPE_FROM_CLASS:
          case RWS_DEFAULT_FLOWTYPE_FROM_CLASS:
            /* Flowtype iteration */
            while ((rv = sksiteFlowtypeIteratorNext(&iter->flowtype_iter,
                                                    &iter->flowtype_id))
                   && flowtype_mask != NULL
                   && !skBitmapGetBit(flowtype_mask, iter->flowtype_id))
                ; /* empty */
            /* Set class from flowtype */
            if (rv) {
                iter->class_id = sksiteFlowtypeGetClassID(iter->flowtype_id);
            }
            break;

          case RWS_CLASS:
          case RWS_CLASS_FROM_SENSOR:
            /* Class iteration */
            while ((rv = sksiteClassIteratorNext(&iter->class_iter,
                                                 &iter->class_id))
                   && class_mask != NULL
                   && !skBitmapGetBit(class_mask, iter->class_id))
                ; /* empty */
            break;

          case RWS_SENSOR:
          case RWS_SENSOR_FROM_CLASS:
            /* Sensor iteration */
            while ((rv = sksiteSensorIteratorNext(&iter->sensor_iter,
                                                  &iter->sensor_id))
                   && sensor_mask != NULL
                   && !skBitmapGetBit(sensor_mask, iter->sensor_id))
                ; /* empty */
            break;

          default:
            skAbortBadCase(iter->order[level]);
        }

        /* Mark that we've started iterating at the current level */
        if (iter->started < level) {
            iter->started = level;
        }

        if (rv && level == iter->level) {
            /* return success at leaf */
            return 1;
        }

        /* Iterate until a leaf iteration succeeds, or we fail to
         * iterate at this level. */
    } while (rv);

    /* Iteration is over.  We are no longer bound at this level.  */
    iter->bound = level - 1;

    return 0;
}


/*
 *  len = rws_print_field(printer, fd, iter, field, width);
 *
 *    Print a 'width'-wide column containing the value for 'field'
 *    using the fprintf-style function 'printer' to the FILE pointer
 *    'fd', where 'iter' is the current context.
 *
 *    Return the number of characters printed.
 */
static int rws_print_field(
    rws_fprintf_t  printer,
    FILE          *fd,
    rws_iter_t    *iter,
    rws_field_t    field,
    int            width)
{
    /* buffer large enough to hold a single sensor name or flowtype
     * name */
    char buf[SK_MAX_STRLEN_SENSOR + SK_MAX_STRLEN_FLOWTYPE];
    const char *s;
    int rv = 0;

    switch (field) {
      case RWST_CLASS:
        if (iter->class_id != SK_INVALID_CLASS) {
            sksiteClassGetName(buf, sizeof(buf), iter->class_id);
            rv = printer(fd, "%*s", width, buf);
        }
        break;

      case RWST_DEFAULT_TYPE:
        if (!iter->default_type) {
            /* When there's no default_type iterator, we can't do a
             * default_type */
            break;
        }
        /* Fall through */
      case RWST_TYPE:
        if (iter->flowtype_id != SK_INVALID_FLOWTYPE) {
            sksiteFlowtypeGetType(buf, sizeof(buf), iter->flowtype_id);
            rv = printer(fd, "%*s", width, buf);
        }
        break;

      case RWST_FLOWTYPE:
        if (iter->flowtype_id != SK_INVALID_FLOWTYPE) {
            sksiteFlowtypeGetName(buf, sizeof(buf), iter->flowtype_id);
            rv = printer(fd, "%*s", width, buf);
        }
        break;

      case RWST_FLOWTYPE_ID:
        if (iter->flowtype_id != SK_INVALID_FLOWTYPE) {
            rv = printer(fd, "%*" PRIu8, width, iter->flowtype_id);
        }
        break;

      case RWST_SENSOR:
        if (iter->sensor_id != SK_INVALID_SENSOR) {
            sksiteSensorGetName(buf, sizeof(buf), iter->sensor_id);
            rv = printer(fd, "%*s", width, buf);
        }
        break;

      case RWST_SENSOR_ID:
        if (iter->sensor_id != SK_INVALID_SENSOR) {
            rv = printer(fd, "%*" PRIu16, width, iter->sensor_id);
        }
        break;

      case RWST_SENSOR_DESC:
        if (iter->sensor_id != SK_INVALID_SENSOR) {
            s = sksiteSensorGetDescription(iter->sensor_id);
            if (s != NULL) {
                rv = printer(fd, "%*s", width, s);
            }
        }
        break;

      case RWST_DEFAULT_CLASS:
      case RWST_DEFAULT_CLASS_LIST:
        {
            classID_t cid = sksiteClassGetDefault();
            if (cid != SK_INVALID_CLASS) {
                sksiteClassGetName(buf, sizeof(buf), cid);
                rv = printer(fd, "%*s", width, buf);
            }
        }
        break;

      case RWST_MARK_DEFAULTS:
        {
            char mark[3] = {' ', ' ', '\0'};
            int i = 0;
            if (iter->class_id != SK_INVALID_CLASS) {
                if (no_columns) {
                    memset(mark, 0, sizeof(mark));
                }
                if (iter->class_id == sksiteClassGetDefault())
                {
                    mark[0] = '+';
                    ++i;
                }
                if (iter->flowtype_id != SK_INVALID_FLOWTYPE) {
                    flowtype_iter_t fi;
                    flowtypeID_t ft;

                    /* See if this type is a default for this class */
                    sksiteClassDefaultFlowtypeIterator(iter->class_id, &fi);
                    while (sksiteFlowtypeIteratorNext(&fi, &ft)) {
                        if (iter->flowtype_id == ft) {
                            mark[no_columns ? i : 1] = '*';
                            ++i;
                            break;
                        }
                    }
                }
            }
            rv = printer(fd, "%*s", width, mark);
        }
        break;

      case RWST_CLASS_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_CLASS, width);
        break;
      case RWST_TYPE_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_TYPE, width);
        break;
      case RWST_FLOWTYPE_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_FLOWTYPE, width);
        break;
      case RWST_FLOWTYPE_ID_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_FLOWTYPE_ID, width);
        break;
      case RWST_SENSOR_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_SENSOR, width);
        break;
      case RWST_SENSOR_ID_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_SENSOR_ID, width);
        break;
      case RWST_DEFAULT_TYPE_LIST:
        rv = rws_print_list_field(printer, fd, iter, RWST_DEFAULT_TYPE, width);
        break;

      case RWST_MAX_FIELD_COUNT:
        skAbortBadCase(field);
    }

    /* Fill in any missing spaces. */
    if (rv < width) {
        printer(fd, "%*s", width - rv, "");
    }

    return rv;
}


/*
 *  len = rws_print_list_field(printer, fd, iter, field, width);
 *
 *    Print a 'width'-wide column containing the value for 'field'
 *    (where 'field' represents a "FIELD:list" field) using the
 *    fprintf-style function 'printer' to the FILE pointer 'fd', where
 *    'iter' is the current context.
 *
 *    Return the number of characters printed.
 */
static int rws_print_list_field(
    rws_fprintf_t  printer,
    FILE          *fd,
    rws_iter_t    *iter,
    rws_field_t    field,
    int            width)
{
    int total = 0;
    int len;
    rws_iter_t subiter;
    int first;

    /* Create a site iterator containing a single sub-iterator of the
     * correct type.  */
    memset(&subiter, 0, sizeof(subiter));
    switch (field) {
      case RWST_CLASS:
        if (iter->sensor_id == SK_INVALID_SENSOR) {
            subiter.order[0] = RWS_CLASS;
        } else {
            subiter.order[0] = RWS_CLASS_FROM_SENSOR;
        }
        subiter.sensor_id = iter->sensor_id;
        break;
      case RWST_TYPE:
      case RWST_FLOWTYPE:
      case RWST_FLOWTYPE_ID:
        if (iter->class_id == SK_INVALID_CLASS) {
            subiter.order[0] = RWS_FLOWTYPE;
        } else {
            subiter.order[0] = RWS_FLOWTYPE_FROM_CLASS;
        }
        subiter.class_id = iter->class_id;
        subiter.flowtype_id = iter->flowtype_id;
        break;
      case RWST_SENSOR:
      case RWST_SENSOR_ID:
        if (iter->class_id == SK_INVALID_CLASS) {
            subiter.order[0] = RWS_SENSOR;
        } else {
            subiter.order[0] = RWS_SENSOR_FROM_CLASS;
        }
        subiter.class_id = iter->class_id;
        break;
      case RWST_DEFAULT_TYPE:
        if (iter->class_id == SK_INVALID_CLASS) {
            return 0;
        }
        subiter.order[0] = RWS_DEFAULT_FLOWTYPE_FROM_CLASS;
        subiter.class_id = iter->class_id;
        break;
      default:
        skAbortBadCase(field);
    }
    rws_iter_bind(&subiter, 0);

    /* Call ourself with the fake-fprintf to determine the number of
     * printed characters in this field, and then print the padding */
    if (width != 0) {
        len = rws_print_list_field(fprintf_size, NULL, iter, field, 0);
        if (len < width) {
            total += printer(fd, "%*s", width - len, "");
        }
    }

    /* Iterate over the fields, printing each */
    first = 1;
    while (rws_iter_next(&subiter, 0)) {
        if (!first) {
            len = printer(fd, "%c", list_separator);
            total += len;
        }
        len = rws_print_field(printer, fd, &subiter, field, 0);
        total += len;
        first = 0;
    }

    return total;
}


/*
 *  rws_print_row(iter);
 *
 *    Print a single row from an iterator.
 */
static void rws_print_row(
    rws_iter_t *iter)
{
    size_t i;

    for (i = 0; i < num_fields; ++i) {
        if (i > 0) {
            fprintf(output.of_fp, "%c", column_separator);
        }
        rws_print_field(fprintf, output.of_fp, iter,
                        fields[i], col_width[fields[i]]);
    }
    fprintf(output.of_fp, "%s\n", final_delim);
}


/*
 *  len = fprintf_size(fs, format, ...);
 *
 *    Used to calculate print sizes.  This fprintf() variation does
 *    not produce outout, it simply returns how many charcters it
 *    WOULD have printed.
 */
static int fprintf_size(
    FILE    UNUSED(*stream),
    const char     *format,
    ...)
{
    char buf;
    int len;
    va_list ap;
    va_start(ap, format);
    len = vsnprintf(&buf, 0, format, ap);
    va_end(ap);
    return len;
}


/*
 *  rws_calcsize_row(iter);
 *
 *    Updates column sizes based on a single row that would be printed
 *    from the given iterator.
 */
static void rws_calcsize_row(
    rws_iter_t *iter)
{
    size_t i;
    int len;

    for (i = 0; i < num_fields; i++) {
        len = rws_print_field(fprintf_size, NULL, iter, fields[i], 0);
        if (len > col_width[fields[i]]) {
            col_width[fields[i]] = len;
        }
    }
}


/*
 *  rws_print_titles();
 *
 *    Print the column headings unless --no-titles was specified.
 */
static void rws_print_titles(void)
{
    size_t i;

    if (no_titles) {
        return;
    }

    for (i = 0; i < num_fields; ++i) {
        if (i > 0) {
            fprintf(output.of_fp, "%c", column_separator);
        }
        fprintf(output.of_fp, "%*s", col_width[fields[i]],
                (char *)field_map_entries[fields[i]].userdata);
    }
    fprintf(output.of_fp, "%s\n", final_delim);
}


/*
 *  status = rws_setup_iter_from_fields(iter);
 *
 *    Set up the iterator for iterating based on the field list.
 *
 *    Return 0 for normal iteration.  Return -1 for no output (other
 *    than titles).  Return 1 for outputting a single non-iterated
 *    entry.
 */
static int rws_setup_iter_from_fields(
    rws_iter_t *iter)
{
    size_t i;
    /* Next iterator level to initialize */
    int level = 0;
    /* Boolean: class iterator set? */
    int class_set = 0;
    /* Boolean: flowtype iterator set? */
    int flowtype_set = 0;
    /* Boolean: sensor iterator set? */
    int sensor_set = 0;
    /* Default-type iterator set at this level (0 == none, since
     * default-type cannot exist at level 0. */
    int default_type = 0;
    /* Boolean: could be a non-iterable singleton */
    int singleton = 0;

    memset(iter, 0, sizeof(*iter));
    iter->level = -1;

    for (i = 0; i < num_fields; i++) {
        switch (fields[i]) {
          case RWST_CLASS:
          case RWST_DEFAULT_TYPE_LIST:
            /* Need classes to generate default types */
            if (class_set || flowtype_set) {
                break;
            }
            iter->order[level] = sensor_set ? RWS_CLASS_FROM_SENSOR : RWS_CLASS;
            iter->level = level;
            ++level;
            class_set = 1;
            break;
          case RWST_TYPE:
          case RWST_FLOWTYPE:
          case RWST_FLOWTYPE_ID:
            if (flowtype_set) {
                break;
            }
            if (default_type) {
                /* Replace default type, as flowtype and default type
                   is nonsensical */
                iter->order[default_type] =
                    class_set ? RWS_FLOWTYPE_FROM_CLASS : RWS_FLOWTYPE;
                default_type = 0;
            } else {
                iter->order[level] =
                    class_set ? RWS_FLOWTYPE_FROM_CLASS : RWS_FLOWTYPE;
                iter->level = level;
                ++level;
            }
            flowtype_set = 1;
            break;
          case RWST_SENSOR:
          case RWST_SENSOR_ID:
          case RWST_SENSOR_DESC:
            if (sensor_set) {
                break;
            }
            iter->order[level] = class_set ? RWS_SENSOR_FROM_CLASS : RWS_SENSOR;
            iter->level = level;
            ++level;
            sensor_set = 1;
            break;
          case RWST_DEFAULT_TYPE:
            assert(!default_type);
            if (flowtype_set) {
                break;
            }
            if (!class_set) {
                /* Default-type needs a class iterator, so add one */
                iter->order[level] =
                    sensor_set ? RWS_CLASS_FROM_SENSOR : RWS_CLASS;
                ++level;
                class_set = 1;
            }
            iter->order[level] = RWS_DEFAULT_FLOWTYPE_FROM_CLASS;
            iter->level = level;
            ++level;
            default_type = level;
            break;
          case RWST_DEFAULT_CLASS:
          case RWST_CLASS_LIST:
          case RWST_TYPE_LIST:
          case RWST_FLOWTYPE_LIST:
          case RWST_FLOWTYPE_ID_LIST:
          case RWST_SENSOR_LIST:
          case RWST_SENSOR_ID_LIST:
          case RWST_DEFAULT_CLASS_LIST:
            /* These fields can generate null-iterators with a single
             * row of output. */
            singleton = 1;
            break;
          case RWST_MARK_DEFAULTS:
            break;
          case RWST_MAX_FIELD_COUNT:
          default:
            skAbortBadCase(fields[i]);
        }
    }

    /* Bind the iterator (at level -1 if this is a null-iterator) */
    iter->flowtype_id = SK_INVALID_FLOWTYPE;
    iter->class_id    = SK_INVALID_CLASS;
    iter->sensor_id   = SK_INVALID_SENSOR;
    rws_iter_bind(iter, level ? 0 : -1);

    if (level != 0) {
        /* Return 0 if this is not a null iterator */
        return 0;
    }

    /* Return 1 if this is a singleton null-iterator.  -1 otherwise */
    return singleton ? 1 : -1;
}


int main(int argc, char **argv)
{
    rws_iter_t iter;
    rws_iter_t calciter;
    int rv;

    appSetup(argc, argv);       /* never returns on error */

    /* Set up site iterator */
    rv = rws_setup_iter_from_fields(&iter);
    if (rv == -1) {
        /* Nothing to do */
        rws_print_titles();
        return EXIT_SUCCESS;
    }

    /* Calculate column sizes */
    if (!no_columns) {
        calciter = iter;
        if (rv == 1) {
            /* Null iterator case */
            rws_calcsize_row(&calciter);
        } else {
            while (rws_iter_next(&calciter, 0)) {
                rws_calcsize_row(&calciter);
            }
        }
    }

    /* Print titles */
    rws_print_titles();

    /* Print rows */
    if (rv == 1) {
        /* Null iterator case */
        rws_print_row(&iter);
    } else {
        while (rws_iter_next(&iter, 0)) {
            rws_print_row(&iter);
        }
    }

    return EXIT_SUCCESS;
}


/*
** Local Variables:
** mode:c
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

Here's another case.  

emacs -Q rwrec.h
M-x c-toggle-parse-state-debug
C-v {about 24-27 times}

This is with current bzr.  
111170 rgm@gnu.org-20121210020042-arkhaf5eej6ujgtn

rwrec.h is attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rwrec.h --]
[-- Type: text/x-chdr, Size: 81424 bytes --]

/*
** Copyright (C) 2001-2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  rwrec.h
**
**    The SiLK Flow record (rwRec) definition and functions/macros for
**    manipulating it.
**
*/
#ifndef _RWREC_H
#define _RWREC_H
#ifdef __cplusplus
extern "C" {
#endif

#include <silk/silk.h>

RCSIDENTVAR(rcsID_RWREC_H, "$Id$");

#include <silk/silk_types.h>

#ifndef RWREC_OPAQUE
#  define RWREC_OPAQUE 0
#  include <silk/skipaddr.h>
#endif


/*
 * The following documents macros defined in this file that work to
 * access and manipulate the data in an rwRec.  Although these are
 * macros, they have been documented as functions in order to convey
 * the expected types of the arguments.
 *
 * ****  Convenience macros  ****
 *
 *   void RWREC_CLEAR(rwRec *r);
 *    Zero out the record, and set Sensor ID and Flowtype to invalid
 *    values.
 *
 *   void RWREC_COPY(rwRec *dst, const rwRec *src);
 *    Copy the rwRec from 'src' to 'dst'.
 *
 *   int rwRecIsICMP(const rwRec *r);
 *    Returns non-zero if the record is an ICMP record, zero otherwise.
 *
 *   int rwRecIsWeb(const rwRec *r);
 *    Returns non-zero if the record can be represented using the SiLK
 *    web-specific file formats, zero otherwise.
 *
 *
 * **** More macros ****
 *
 * Function versions of all of the following macros exist.  Simply
 * change the "rwRec" prefix of the macro to "rwrec_" to use the
 * function version.
 *
 * ***  Whether a record is IPv6  ***
 *
 * int      rwRecIsIPv6(const rwRec *r);
 *
 * ***  Set record as IPv4 or IPv6  ***
 *
 * void     rwRecSetIPv6(rwRec *r);
 * void     rwRecSetIPv4(rwRec *r);
 *
 * It is important to note that the above two macros do not do any
 * conversions on the contained IP addresses.  They are primarily to
 * be used when creating a new rwRec from scratch.  See the following
 * for conversion.
 *
 * ***  Convert record to IPv4 or IPv6  ***
 *
 * int      rwRecConvertToIPv4(rwRec *r);
 * void     rwRecConvertToIPv6(rwRec *r);
 *
 * These macros convert an rwRec to IPv4 or IPv6.  The latter always
 * succeeds.  The former will return -1 if unable to convert due to
 * the existence of IPv6 addresses that cannot be represented as IPv4
 * (and return zero on success).
 *
 * ****  rwRec accessor macros ****
 *
 * Most of the following accessor macros come in five standard
 * variations:
 *
 *   <fieldtype> rwRecGet<field>(const reRec *r)
 *    Gets the value of the field directly
 *
 *   void rwRecSet<field>(rwRec *r, <fieldtype> in_v)
 *    Sets the value of the field to 'in_v'
 *
 *   void rwRecMemGet<field>(const rwRec *r, <fieldtype> *out_vp)
 *    Copies the value of the field to location out_vp
 *
 *   void rwRecMemSet<field>(rwRec *r, const <fieldtype> *in_vp)
 *    Copies the value in location in_vp into the field
 *
 *   int rwRecMemCmp<field>(const rwRec *r, const <fieldtype> *vp)
 *    Compares the field to the value in location vp, returning a
 *    negative integer if the field is less than vp, 0 if equal, and a
 *    positive integer if greater than vp.
 *
 * For the rwRecMem{Get,Set,Cmp}<field>() macros, we use of
 * <fieldtype> in the comment to explain the size of the data
 * involved.  With the exception of those dealing the skipaddr_t
 * objects, the actual functions use void pointers; as a result, those
 * macros do not require the value pointer to be aligned.  The macros
 * that handle values larger than uint8_t use memcpy() or memcmp() to
 * copy or compare the values.
 *
 * ** IP Address Macros **
 *
 * For handling IP addresses, there are three sets of macros: one for
 * IPv4 addresses: one for IPv6 addresses, and one for skipaddr_t
 * objects, which can represent either an IPv4 address or and IPv6
 * object.
 *
 * In addition to the Get, Set, and Comparison macros, the follow
 * macro exists in IPv4, IPv6, and skipaddr_t varieties:
 *
 *   void rwRecApplyMask<field>(rwRec *r, const <fieldtype> mask);
 *    Modify the rwRec's <field> IP addresses by applying the
 *    specified mask (using a bit-wise AND) to that IP address.
 *
 * All of the macros that deal with IPv4 addresses assume that you
 * know the rwRec holds IPv4 addresses.  No conversion of the rwRec or
 * of the address occurs.
 *
 * Using the Get macros for IPv6 on an rwRec that contains V4 data
 * will return a properly encoded IPv4-in-IPv6 address.  All other
 * macros for IPv6 assume you know that the rwRec holds IPv6
 * addresses.  For IPv6, the rwRecGet<field>() and rwRecSet<field>()
 * macros do not exist; you must use the rwRecMemGet<field>() and
 * rwRecMemSet<field>() versions.
 *
 * Since an skipaddr_t can hold an IPv4 or an IPv6 address, the macros
 * that use skipaddr_t objects will correctly handle with rwRecs that
 * hold IPv4 or IPv6 data.  Setting or masking an IPv4 rwRec with an
 * IPv6 skipaddr_t may convert the record to IPv6.  The comparison
 * macros will operate in IPv6 space if either argument involves IPv6.
 * Unlike all other rwRecMem{Get,Set,Cmp}<field>() macros, the
 * functions that work with skipaddr_t pointers require the skipaddr_t
 * to be properly aligned.
 *
 * The IPv4 macros include an extra DEPRECATED macro:
 *
 *   uint32_t rwRecGetMask<field>v4(rwRec *r, uint32_t mask)
 *    Gets the value of the field with the given bit-mask applied
 *
 * ***  Source IPv4 Address (sIP)  ***
 *
 * uint32_t rwRecGetSIPv4(const rwRec *r);
 * void     rwRecSetSIPv4(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetSIPv4(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetSIPv4(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpSIPv4(const rwRec *r, const uint32_t *vp);
 * void     rwRecApplyMaskSIPv4(rwRec *r, uint32_t mask);
 * uint32_t rwRecGetMaskSIPv4(const rwRec *r, uint32_t mask);
 *
 * ***  Source IPv6 Address (sIP)  ***
 *
 * void     rwRecMemGetSIPv6(const rwRec *r, uint8_t[16] out_vp);
 * void     rwRecMemSetSIPv6(rwRec *r, const uint8_t[16] in_vp);
 * int      rwRecMemCmpSIPv6(const rwRec *r, const uint8_t[16] vp);
 * void     rwRecApplyMaskSIPv6(rwRec *r, const uint8_t[16] mask);
 *
 * ***  Source IP Address (sIP) as skipaddr_t  ***
 *
 * void     rwRecMemGetSIP(const rwRec *r, skipaddr_t *out_addr);
 * void     rwRecMemSetSIP(rwRec *r, const skipaddr_t *in_addr);
 * int      rwRecMemCmpSIP(const rwRec *r, const skipaddr_t *addr);
 * void     rwRecApplyMaskSIP(rwRec *r, const skipaddr_t *mask);
 *
 * ***  Destination IP Address (dIP)  ***
 *
 * uint32_t rwRecGetDIPv4(const rwRec *r);
 * void     rwRecSetDIPv4(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetDIPv4(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetDIPv4(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpDIPv4(const rwRec *r, const uint32_t *vp);
 * void     rwRecApplyMaskDIPv4(rwRec *r, uint32_t mask);
 * uint32_t rwRecGetMaskDIPv4(const rwRec *r, uint32_t mask);
 *
 * ***  Destination IPv6 Address (dIP)  ***
 *
 * void     rwRecMemGetDIPv6(const rwRec *r, uint8_t[16] out_vp);
 * void     rwRecMemSetDIPv6(rwRec *r, const uint8_t[16] in_vp);
 * int      rwRecMemCmpDIPv6(const rwRec *r, const uint8_t[16] vp);
 * void     rwRecApplyMaskDIPv6(rwRec *r, const uint8_t[16] mask);
 *
 * ***  Destination IP Address (dIP) as skipaddr_t  ***
 *
 * void     rwRecMemGetDIP(const rwRec *r, skipaddr_t *out_addr);
 * void     rwRecMemSetDIP(rwRec *r, const skipaddr_t *in_addr);
 * int      rwRecMemCmpDIP(const rwRec *r, const skipaddr_t *addr);
 * void     rwRecApplyMaskDIP(rwRec *r, const skipaddr_t *mask);
 *
 * ***  Next Hop IP Address  ***
 *
 * uint32_t rwRecGetNhIPv4(const rwRec *r);
 * void     rwRecSetNhIPv4(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetNhIPv4(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetNhIPv4(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpNhIPv4(const rwRec *r, const uint32_t *vp);
 * void     rwRecApplyMaskNhIPv4(rwRec *r, uint32_t mask);
 * uint32_t rwRecGetMaskNhIPv4(const rwRec *r, uint32_t mask);
 *
 * ***  Next Hop IPv6 Address (nhIP)  ***
 *
 * void     rwRecMemGetNhIPv6(const rwRec *r, uint8_t[16] out_vp);
 * void     rwRecMemSetNhIPv6(rwRec *r, const uint8_t[16] in_vp);
 * int      rwRecMemCmpNhIPv6(const rwRec *r, const uint8_t[16] vp);
 * void     rwRecApplyMaskNhIPv6(rwRec *r, const uint8_t[16] mask);
 *
 * ***  Next Hop IP Address (nhIP) as skipaddr_t  ***
 *
 * void     rwRecMemGetNhIP(const rwRec *r, skipaddr_t *out_addr);
 * void     rwRecMemSetNhIP(rwRec *r, const skipaddr_t *in_addr);
 * int      rwRecMemCmpNhIP(const rwRec *r, const skipaddr_t *addr);
 * void     rwRecApplyMaskNhIP(rwRec *r, const skipaddr_t *mask);
 *
 * ***  Source Port (sPort)  ***
 *
 * uint16_t rwRecGetSPort(const rwRec *r);
 * void     rwRecSetSPort(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetSPort(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetSPort(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpSPort(const rwRec *r, const uint16_t *vp);
 *
 * ***  Destination Port (dPort)  ***
 *
 * uint16_t rwRecGetDPort(const rwRec *r);
 * void     rwRecSetDPort(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetDPort(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetDPort(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpDPort(const rwRec *r, const uint16_t *vp);
 *
 * ***  Protocol  ***
 *
 * uint8_t  rwRecGetProto(const rwRec *r);
 * void     rwRecSetProto(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetProto(const rwRec *r, uint8_t *out_vp);
 * void     rwRecMemSetProto(rwRec *r, const uint8_t *in_vp);
 * int      rwRecMemCmpProto(const rwRec *r, const uint8_t *vp);
 *
 * ***  Packet Count (pkts)  ***
 *
 * uint32_t rwRecGetPkts(const rwRec *r);
 * void     rwRecSetPkts(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetPkts(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetPkts(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpPkts(const rwRec *r, const uint32_t *vp);
 *
 * ***  Byte count  ***
 *
 * uint32_t rwRecGetBytes(const rwRec *r);
 * void     rwRecSetBytes(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetBytes(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetBytes(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpBytes(const rwRec *r, const uint32_t *vp);
 *
 * ***  Bitwise OR of TCP Flags on ALL packets in flow  ***
 *
 * uint8_t  rwRecGetFlags(const rwRec *r);
 * void     rwRecSetFlags(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetFlags(const rwRec *r, uint8_t *out_vp);
 * void     rwRecMemSetFlags(rwRec *r, const uint8_t *in_vp);
 * int      rwRecMemCmpFlags(const rwRec *r, const uint8_t *vp);
 *
 * ***  TCP Flags seen on initial packet of flow  ***
 *
 * uint8_t  rwRecGetInitFlags(const rwRec *r);
 * void     rwRecSetInitFlags(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetInitFlags(const rwRec *r, uint8_t *out_vp);
 * void     rwRecMemSetInitFlags(rwRec *r, const uint8_t *in_vp);
 * int      rwRecMemCmpInitFlags(const rwRec *r, const uint8_t *vp);
 *
 * ***  Bitwise OR of TCP Flags on all packets in session except first  ***
 *
 * uint8_t  rwRecGetRestFlags(const rwRec *r);
 * void     rwRecSetRestFlags(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetRestFlags(const rwRec *r, uint8_t *out_vp);
 * void     rwRecMemSetRestFlags(rwRec *r, const uint8_t *in_vp);
 * int      rwRecMemCmpRestFlags(const rwRec *r, const uint8_t *vp);
 *
 * ***  Start Time as milliseconds since UNIX epoch (sTime)  ***
 *
 * sktime_t rwRecGetStartTime(const rwRec *r);
 * void     rwRecSetStartTime(rwRec *r, sktime_t in_v);
 * void     rwRecMemGetStartTime(const rwRec *r, sktime_t *out_vp);
 * void     rwRecMemSetStartTime(rwRec *r, const sktime_t *in_vp);
 * int      rwRecMemCmpStartTime(const rwRec *r, const sktime_t *vp);
 *
 * uint32_t rwRecGetStartSeconds(const rwRec *r);
 * void     rwRecMemGetStartSeconds(const rwRec *r, uint32_t *out_vp);
 * uint16_t rwRecGetStartMSec(const rwRec *r);
 * void     rwRecMemGetStartMSec(const rwRec *r, uint16_t *out_vp);
 *
 * The second set of macros return the integer number of seconds and
 * the millisecond remainder.  In particular:
 *
 *   rwRecGetStartTime(r) ==
 *     rwRecGetStartSeconds(r) * (uint64_t)1000 + rwRecGetStartMSec(r)
 *
 * ***  End Time is derived from the sTime and duration (eTime)  ***
 *
 * sktime_t rwRecGetEndTime(const rwRec *r);
 * void     rwRecMemGetEndTime(const rwRec *r, sktime_t *out_vp);
 *
 * uint32_t rwRecGetEndSeconds(const rwRec *r);
 * void     rwRecMemGetEndSeconds(const rwRec *r, uint32_t *out_vp);
 * uint16_t rwRecGetEndMSec(const rwRec *r);
 * void     rwRecMemGetEndMSec(const rwRec *r, uint16_t *out_vp);
 *
 * There are no setter macros for end time, because end time is
 * derived from start time and duration (elapsed time).
 *
 * The second set of macros return the integer number of seconds and
 * the millisecond remainder.  In particular:
 *
 *   rwRecGetEndTime(r) ==
 *     rwRecGetEndSeconds(r) * (uint64_t)1000 + rwRecGetEndMSec(r)
 *
 * ***  Elapsed (duration) of the flow, in milliseconds  ***
 *
 * uint32_t rwRecGetElapsed(const rwRec *r);
 * void     rwRecSetElapsed(rwRec *r, uint32_t in_v);
 * void     rwRecMemGetElapsed(const rwRec *r, uint32_t *out_vp);
 * void     rwRecMemSetElapsed(rwRec *r, const uint32_t *in_vp);
 * int      rwRecMemCmpElapsed(const rwRec *r, const uint32_t *vp);
 *
 * uint32_t rwRecGetElapsedSeconds(const rwRec *r);
 * void     rwRecMemGetElapsedSeconds(const rwRec *r, uint32_t *out_vp);
 * uint16_t rwRecGetElapsedMSec(const rwRec *r);
 * void     rwRecMemGetElapsedMSec(const rwRec *r, uint16_t *out_vp);
 *
 * The second set of macros return the integer number of seconds and
 * the millisecond remainder.  In particular:
 *
 *   rwRecGetElapsedTime(r) ==
 *     rwRecGetElapsedSeconds(r) * 1000 + rwRecGetElapsedMSec(r)
 *
 * ***  Sensor ID (sID)  ***
 *
 * sensorID_t rwRecGetSensor(const rwRec *r);
 * void     rwRecSetSensor(rwRec *r, sensorID_t in_v);
 * void     rwRecMemGetSensor(const rwRec *r, sensorID_t *out_vp);
 * void     rwRecMemSetSensor(rwRec *r, const sensorID_t *in_vp);
 * int      rwRecMemCmpSensor(const rwRec *r, const sensorID_t *vp);
 *
 * ***  FlowType holds Class and Type  ***
 *
 * flowtypeID_t rwRecGetFlowType(const rwRec *r);
 * void     rwRecSetFlowType(rwRec *r, flowtypeID_t in_v);
 * void     rwRecMemGetFlowType(const rwRec *r, flowtypeID_t *out_vp);
 * void     rwRecMemSetFlowType(rwRec *r, const flowtypeID_t *in_vp);
 * int      rwRecMemCmpFlowType(const rwRec *r, const flowtypeID_t *vp);
 *
 * ***  SNMP Input Value (Router incoming/ingress interface/vlanId) ***
 *
 * uint16_t rwRecGetInput(const rwRec *r);
 * void     rwRecSetInput(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetInput(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetInput(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpInput(const rwRec *r, const uint16_t *vp);
 *
 * ***  SNMP Output Value (Router outgoing/egress interface/postVlanId) ***
 *
 * uint16_t rwRecGetOutput(const rwRec *r);
 * void     rwRecSetOutput(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetOutput(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetOutput(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpOutput(const rwRec *r, const uint16_t *vp);
 *
 * ***  TCP State (the Attributes field)  ***
 *
 * uint8_t  rwRecGetTcpState(const rwRec *r);
 * void     rwRecSetTcpState(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetTcpState(const rwRec *r, uint8_t *out_vp);
 * void     rwRecMemSetTcpState(rwRec *r, const uint8_t *in_vp);
 * int      rwRecMemCmpTcpState(const rwRec *r, const uint8_t *vp);
 *
 * The TCP state field is a bit field which states certain miscellaneous
 * information about the flow record.  The following constants are
 * defined which represent this information:
 *
 * #define SK_TCPSTATE_EXPANDED              0x01
 *  Additional TCP-state machine information is available.  This must
 *  be set for every TCP flow where the init_flags and rest_flags
 *  fields are set.
 *
 * #define SK_TCPSTATE_FIN_FOLLOWED_NOT_ACK  0x08
 *  Flow received packets following the FIN packet that were not ACK or
 *  RST packets.
 *
 * #define SK_TCPSTATE_UNIFORM_PACKET_SIZE   0x10
 *  Flow has packets all of the same size
 *
 * #define SK_TCPSTATE_TIMEOUT_KILLED        0x20
 *  Flow ends prematurely due to a timeout by the collector.
 *
 * #define SK_TCPSTATE_TIMEOUT_STARTED       0x40
 *  Flow is a continuation of a previous flow that was killed
 *  prematurely due to a timeout by the collector.
 *
 * Note: the most significant bit of tcp_state (0x80) is used as a flag
 * to mark a record as having IPv6 addresses.  The rwRecSetIPv4() and
 * rwRecSetIPv6() macros should be used to modify this bit.
 *
 * Be careful when setting the TCP state.  You usually want get the
 * current TCP state, add or remove specific bits by masking, then set it
 * with the resulting value.
 *
 * ***  Application  ***
 *
 * uint16_t rwRecGetApplication(const rwRec *r);
 * void     rwRecSetApplication(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetApplication(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetApplication(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpApplication(const rwRec *r, const uint16_t *vp);
 *
 * The application field can be set by the yaf flow generation
 * software when it is configured with the "applabel" feature.  This
 * feature causes yaf to inspect the packets in the flow and guess as
 * to the type of application (HTTP, SMTP, SSH, etc) the packets
 * represent.  The value for the field is the standard service port
 * for that service (80, 25, 22, etc).
 *
 * ***  Memo  ***
 *
 * uint16_t rwRecGetMemo(const rwRec *r);
 * void     rwRecSetMemo(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetMemo(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetMemo(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpMemo(const rwRec *r, const uint16_t *vp);
 *
 * Currently unused.
 *
 * ***  ICMP Type and Code is derived from the DPort  ***
 *
 * uint8_t  rwRecGetIcmpType(const rwRec *r);
 * void     rwRecSetIcmpType(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetIcmpType(const rwRec *r, uint8_t *out_vp);
 *
 * uint8_t  rwRecGetIcmpCode(const rwRec *r);
 * void     rwRecSetIcmpCode(rwRec *r, uint8_t in_v);
 * void     rwRecMemGetIcmpCode(const rwRec *r, uint8_t *out_vp);
 *
 * uint16_t rwRecGetIcmpTypeAndCode(const rwRec *r);
 * void     rwRecSetIcmpTypeAndCode(rwRec *r, uint16_t in_v);
 * void     rwRecMemGetIcmpTypeAndCode(const rwRec *r, uint16_t *out_vp);
 * void     rwRecMemSetIcmpTypeAndCode(rwRec *r, const uint16_t *in_vp);
 * int      rwRecMemCmpIcmpTypeAndCode(const rwRec *r, const uint16_t *vp);
 *
 * Since the ICMP Type and Code are stored in the dport field, modifying
 * these will modify the return value of rwRecGetDPort() and friends.
 */


#define SK_WEBPORT_CHECK(p) ((p) == 80 || (p) == 443 || (p) == 8080)
/*
 *    Return a true value if port 'p' is a "web" port; false otherwise
 */

#define RWREC_CLEAR(rec)                                     \
    do {                                                     \
        memset((rec), 0, sizeof(rwRec));                     \
        rwRecSetSensor((rec), SK_INVALID_SENSOR);            \
        rwRecSetFlowType((rec), SK_INVALID_FLOWTYPE);        \
    } while(0)
/*
 *    Zero out the record, and set Sensor ID and Flowtype to invalid
 *    values.
 */


#define RWREC_COPY(dst, src)                    \
    memcpy((dst), (src), sizeof(rwRec))
/*
 *    Copy the rwRec from 'src' to 'dst'.
 */


/*
 *  This is the generic SiLK Flow record returned from ANY file format
 *  containing packed SiLK Flow records.
 *
 *  typedef struct rwGenericRec_V5_st rwGenericRec_V5_t; // silk_types.h
 *  typedef rwGenericRec_V5_t rwRec;                     // silk_types.h
 */
struct rwGenericRec_V5_st {
#if RWREC_OPAQUE && !defined(RWREC_DEFINE_BODY)
#if SK_ENABLE_IPV6
    uint8_t         ar[88];
#else
    uint8_t         ar[52];
#endif
#else
    int64_t         sTime;       /*  0- 7  Flow start time in milliseconds
                                  *        since UNIX epoch */

    uint32_t        elapsed;     /*  8-11  Duration of flow in millisecs */

    uint16_t        sPort;       /* 12-13  Source port */
    uint16_t        dPort;       /* 14-15  Destination port */

    uint8_t         proto;       /* 16     IP protocol */
    flowtypeID_t    flow_type;   /* 17     Class & Type info */
    sensorID_t      sID;         /* 18-19  Sensor ID */

    uint8_t         flags;       /* 20     OR of all flags (Netflow flags) */
    uint8_t         init_flags;  /* 21     TCP flags in first packet
                                  *        or blank for "legacy" data */
    uint8_t         rest_flags;  /* 22     TCP flags on non-initial packet
                                  *        or blank for "legacy" data */
    uint8_t         tcp_state;   /* 23     TCP state machine info (below) */

    uint16_t        application; /* 24-25  "Service" port set by collector */
    uint16_t        memo;        /* 26-27  Application specific field */

    uint16_t        input;       /* 28-29  Router incoming SNMP interface */
    uint16_t        output;      /* 30-31  Router outgoing SNMP interface */

    uint32_t        pkts;        /* 32-35  Count of packets */
    uint32_t        bytes;       /* 36-39  Count of bytes */

    ipUnion         sIP;         /* 40-43  (or 40-55 if IPv6) Source IP */
    ipUnion         dIP;         /* 44-47  (or 56-71 if IPv6) Destination IP */
    ipUnion         nhIP;        /* 48-51  (or 72-87 if IPv6) Router Next Hop IP */
#endif  /* RWREC_OPAQUE && !defined(RWREC_DEFINE_BODY) */
};


/*
**  Values for tcp_state value in rwGeneric and packed formats
*/

/* No additional TCP-state machine information is available */
#define SK_TCPSTATE_NO_INFO               0x00

/* Additional TCP-state machine information is available.  This must
 * be set for every TCP flow where the init_flags and rest_flags
 * fields are set.  */
#define SK_TCPSTATE_EXPANDED              0x01

/* Flow received packets following the FIN packet that were not ACK or
 * RST packets. */
#define SK_TCPSTATE_FIN_FOLLOWED_NOT_ACK  0x08

/* Flow has packets all of the same size */
#define SK_TCPSTATE_UNIFORM_PACKET_SIZE   0x10

/* Flow ends prematurely due to a timeout by the collector. */
#define SK_TCPSTATE_TIMEOUT_KILLED        0x20

/* Flow is a continuation of a previous flow that was killed
 * prematurely due to a timeout by the collector. */
#define SK_TCPSTATE_TIMEOUT_STARTED       0x40

/* Note: the most significant bit of tcp_state (0x80) is used as a
 * flag to mark a record as having IPv6 addresses. */


/* the sizeof the fields in an rwRec */
#define RWREC_SIZEOF_SIPv4        4
#define RWREC_SIZEOF_DIPv4        4
#define RWREC_SIZEOF_NHIPv4       4
#define RWREC_SIZEOF_SPORT        2
#define RWREC_SIZEOF_DPORT        2
#define RWREC_SIZEOF_INPUT        2
#define RWREC_SIZEOF_OUTPUT       2
#define RWREC_SIZEOF_STIME        8
#define RWREC_SIZEOF_ELAPSED      4
#define RWREC_SIZEOF_PKTS         4
#define RWREC_SIZEOF_BYTES        4
#define RWREC_SIZEOF_PROTO        1
#define RWREC_SIZEOF_FLOW_TYPE    1
#define RWREC_SIZEOF_SID          2
#define RWREC_SIZEOF_FLAGS        1
#define RWREC_SIZEOF_INIT_FLAGS   1
#define RWREC_SIZEOF_REST_FLAGS   1
#define RWREC_SIZEOF_TCP_STATE    1
#define RWREC_SIZEOF_APPLICATION  2
#define RWREC_SIZEOF_MEMO         2
#define RWREC_SIZEOF_SIPv6       16
#define RWREC_SIZEOF_DIPv6       16
#define RWREC_SIZEOF_NHIPv6      16

#if !SK_ENABLE_IPV6
#  define RWREC_SIZEOF_SIP  RWREC_SIZEOF_SIPv4
#  define RWREC_SIZEOF_DIP  RWREC_SIZEOF_DIPv4
#  define RWREC_SIZEOF_NHIP RWREC_SIZEOF_NHIPv4
#else
#  define RWREC_SIZEOF_SIP  RWREC_SIZEOF_SIPv6
#  define RWREC_SIZEOF_DIP  RWREC_SIZEOF_DIPv6
#  define RWREC_SIZEOF_NHIP RWREC_SIZEOF_NHIPv6
#endif /* SK_ENABLE_IPV6 */


/* Helper macros */
#if 0
#  define MEMCPY8(dst, src)   { *((uint8_t*)(dst))  = *((uint8_t*)(src)); }
#  define MEMCPY16(dst, src)  { *((uint16_t*)(dst)) = *((uint16_t*)(src)); }
#  define MEMCPY32(dst, src)  { *((uint32_t*)(dst)) = *((uint32_t*)(src)); }
#else
#  define MEMCPY8(dst, src)   memcpy((dst), (src), sizeof(uint8_t))
#  define MEMCPY16(dst, src)  memcpy((dst), (src), sizeof(uint16_t))
#  define MEMCPY32(dst, src)  memcpy((dst), (src), sizeof(uint32_t))
#endif


/***  Whether record is ICMP  ***/

#define rwRecIsICMP(r)                                                  \
    ((IPPROTO_ICMP == rwRecGetProto(r))                                 \
     || (rwRecIsIPv6(r) && (IPPROTO_ICMPV6 == rwRecGetProto(r))))


/***  Whether record is WEB/HTTP  ***/

#define rwRecIsWeb(r)                                   \
    ((IPPROTO_TCP == rwRecGetProto(r))                  \
     && (SK_WEBPORT_CHECK(rwRecGetSPort(r))             \
         || SK_WEBPORT_CHECK(rwRecGetDPort(r))))


/***  Whether record is IPv6  ***/

#if !SK_ENABLE_IPV6
#  define rwRecIsIPv6(r)  0
#else

int  rwrec_IsIPv6(const rwRec *r);
void rwrec_SetIPv6(rwRec *r);
void rwrec_SetIPv4(rwRec *r);

#define _recIsIPv6(r)                           \
    (((r)->tcp_state & 0x80) ? 1 : 0)
#define _recSetIPv6(r)                          \
    { (r)->tcp_state |= 0x80; }
#define _recSetIPv4(r)                          \
    { (r)->tcp_state &= 0x7F; }

#if RWREC_OPAQUE
#  define rwRecIsIPv6(r)    rwrec_IsIPv6(r)
#  define rwRecSetIPv6(r)   rwrec_SetIPv6(r)
#  define rwRecSetIPv4(r)   rwrec_SetIPv4(r)
#else
#  define rwRecIsIPv6(r)    _recIsIPv6(r)
#  define rwRecSetIPv6(r)   _recSetIPv6(r)
#  define rwRecSetIPv4(r)   _recSetIPv4(r)
#endif /* RWREC_OPAQUE */

#endif  /* SK_ENABLE_IPV6 */


/***  Convert a Record to IPv6 or IPv4  ***/

#if SK_ENABLE_IPV6

void rwrec_ConvertToIPv6(rwRec *r);
int  rwrec_ConvertToIPv4(rwRec *r);

#define _recConvertToIPv6(r)                    \
    {                                           \
        ipUnion4to6(&(r)->sIP, &(r)->sIP);      \
        ipUnion4to6(&(r)->dIP, &(r)->dIP);      \
        ipUnion4to6(&(r)->nhIP, &(r)->nhIP);    \
        _recSetIPv6(r);                         \
    }

#if RWREC_OPAQUE
#  define rwRecConvertToIPv6(r)   rwrec_ConvertToIPv6(r)
#  define rwRecConvertToIPv4(r)   rwrec_ConvertToIPv4(r)
#else
#  define rwRecConvertToIPv6(r)   _recConvertToIPv6(r)
#  define rwRecConvertToIPv4(r)   rwrec_ConvertToIPv4(r)
#endif /* RWREC_OPAQUE */

#endif  /* SK_ENABLE_IPV6 */


/***  Source IPv4 Address (sIP)  ***/

uint32_t rwrec_GetSIPv4(const rwRec *r);
void     rwrec_SetSIPv4(rwRec *r, uint32_t in_v);
void     rwrec_MemGetSIPv4(const rwRec *r, void *out_vp);
void     rwrec_MemSetSIPv4(rwRec *r, const void *in_vp);
int      rwrec_MemCmpSIPv4(const rwRec *r, const void *vp);
uint32_t rwrec_GetMaskSIPv4(const rwRec *r, uint32_t mask);
void     rwrec_ApplyMaskSIPv4(rwRec *r, uint32_t mask);

#define _recGetSIPv4(r)                         \
    ((r)->sIP.ipu_ipv4)
#define _recSetSIPv4(r, in_v)                   \
    { ((r)->sIP.ipu_ipv4) = (in_v); }
#define _recMemGetSIPv4(r, out_vp)                              \
    memcpy((out_vp), &((r)->sIP.ipu_ipv4), RWREC_SIZEOF_SIPv4)
#define _recMemSetSIPv4(r, in_vp)                               \
    memcpy(&((r)->sIP.ipu_ipv4), (in_vp), RWREC_SIZEOF_SIPv4)
#define _recMemCmpSIPv4(r, vp)                          \
    memcmp(&((r)->sIP.ipu_ipv4), (vp), RWREC_SIZEOF_SIPv4)
#define _recGetMaskSIPv4(r, mask)               \
    (((r)->sIP.ipu_ipv4) & mask)
#define _recApplyMaskSIPv4(r, mask)             \
    ipUnionApplyMaskV4(&((r)->sIP), (mask))

#if RWREC_OPAQUE
#  define rwRecGetSIPv4(r)              rwrec_GetSIPv4(r)
#  define rwRecSetSIPv4(r, in_v)        rwrec_SetSIPv4(r, in_v)
#  define rwRecMemGetSIPv4(r, out_vp)   rwrec_MemGetSIPv4(r, out_vp)
#  define rwRecMemSetSIPv4(r, in_vp)    rwrec_MemSetSIPv4(r, in_vp)
#  define rwRecMemCmpSIPv4(r, vp)       rwrec_MemCmpSIPv4(r, vp)
#  define rwRecGetMaskSIPv4(r, mask)    rwrec_GetMaskSIPv4(r, mask)
#  define rwRecApplyMaskSIPv4(r, mask)  rwrec_ApplyMaskSIPv4(r, mask)
#else
#  define rwRecGetSIPv4(r)              _recGetSIPv4(r)
#  define rwRecSetSIPv4(r, in_v)        _recSetSIPv4(r, in_v)
#  define rwRecMemGetSIPv4(r, out_vp)   _recMemGetSIPv4(r, out_vp)
#  define rwRecMemSetSIPv4(r, in_vp)    _recMemSetSIPv4(r, in_vp)
#  define rwRecMemCmpSIPv4(r, vp)       _recMemCmpSIPv4(r, vp)
#  define rwRecGetMaskSIPv4(r, mask)    _recGetMaskSIPv4(r, mask)
#  define rwRecApplyMaskSIPv4(r, mask)  _recApplyMaskSIPv4(r, mask)
#endif /* RWREC_OPAQUE */


/***  Source IPv6 Address (sIP)  ***/

#if SK_ENABLE_IPV6

void     rwrec_MemGetSIPv6(const rwRec *r, void *out_vp);
void     rwrec_MemSetSIPv6(rwRec *r, const void *in_vp);
int      rwrec_MemCmpSIPv6(const rwRec *r, const void *vp);
void     rwrec_ApplyMaskSIPv6(rwRec *r, const void *mask_vp);

#define _recMemGetSIPv6(r, out_vp)                                      \
    if (_recIsIPv6(r)) {                                                \
        ipUnionGetV6(&((r)->sIP), (out_vp));                            \
    } else {                                                            \
        ipUnionGetV4AsV6(&((r)->sIP), (out_vp));                        \
    }
#define _recMemSetSIPv6(r, in_vp)                               \
    memcpy(&((r)->sIP.ipu_ipv6), (in_vp), RWREC_SIZEOF_SIPv6)
#define _recMemCmpSIPv6(r, vp)                                  \
    memcmp(&((r)->sIP.ipu_ipv6), (vp), RWREC_SIZEOF_SIPv6)
#define _recApplyMaskSIPv6(r, mask)             \
    ipUnionApplyMaskV6(&((r)->sIP), (mask))

#if RWREC_OPAQUE
#  define rwRecMemGetSIPv6(r, out_vp)   rwrec_MemGetSIPv6(r, out_vp)
#  define rwRecMemSetSIPv6(r, in_vp)    rwrec_MemSetSIPv6(r, in_vp)
#  define rwRecMemCmpSIPv6(r, vp)       rwrec_MemCmpSIPv6(r, vp)
#  define rwRecApplyMaskSIPv6(r, mask)  rwrec_ApplyMaskSIPv6(r, mask)
#else
#  define rwRecMemGetSIPv6(r, out_vp)   _recMemGetSIPv6(r, out_vp)
#  define rwRecMemSetSIPv6(r, in_vp)    _recMemSetSIPv6(r, in_vp)
#  define rwRecMemCmpSIPv6(r, vp)       _recMemCmpSIPv6(r, vp)
#  define rwRecApplyMaskSIPv6(r, mask)  _recApplyMaskSIPv6(r, mask)
#endif /* RWREC_OPAQUE */

#endif  /* SK_ENABLE_IPV6 */


/***  Source IP Address (sIP) as skipaddr_t  ***/

void     rwrec_MemGetSIP(const rwRec *r, skipaddr_t *out_addr);
void     rwrec_MemSetSIP(rwRec *r, const skipaddr_t *in_addr);
int      rwrec_MemCmpSIP(const rwRec *r, const skipaddr_t *addr);
void     rwrec_ApplyMaskSIP(rwRec *r, const skipaddr_t *mask_addr);

#if !SK_ENABLE_IPV6
#define _recMemGetSIP _recMemGetSIPv4
#define _recMemSetSIP _recMemSetSIPv4
#define _recMemCmpSIP(r, addr)                          \
    _recMemCmpSIPv4((r), &((addr)->ip_ip.ipu_ipv4))
#define _recApplyMaskSIP(r, addr)                       \
    _recApplyMaskSIPv4((r), (addr)->ip_ip.ipu_ipv4)
#else
#define _recMemGetSIP(r, out_addr)                      \
    do {                                                \
        memcpy(out_addr, &((r)->sIP), sizeof(ipUnion)); \
        skipaddrSetVersion((out_addr), _recIsIPv6(r));  \
    } while(0)
#define _recMemSetSIP(r, in_addr)                               \
    do {                                                        \
        if (skipaddrIsV6(in_addr) == _recIsIPv6(r)) {           \
            /* both are either V4 or V6 */                      \
            memcpy(&((r)->sIP), (in_addr), sizeof(ipUnion));    \
        } else if (_recIsIPv6(r)) {                             \
            /* convert V4 IP to V6 */                           \
            ipUnion4to6(&((in_addr)->ip_ip), &((r)->sIP));      \
        } else {                                                \
            /* must convert record to V6 */                     \
            _recConvertToIPv6(r);                               \
            memcpy(&((r)->sIP), (in_addr), sizeof(ipUnion));    \
        }                                                       \
    } while(0)
#define _recMemCmpSIP(r, addr)      rwrec_MemCmpSIP(r, addr)
#define _recApplyMaskSIP(r, addr)   rwrec_ApplyMaskSIP(r, addr)
#endif /* SK_ENABLE_IPV6 */

#if RWREC_OPAQUE
#  define rwRecMemGetSIP(r, out_addr)   rwrec_MemGetSIP(r, out_addr)
#  define rwRecMemSetSIP(r, in_addr)    rwrec_MemSetSIP(r, in_addr)
#  define rwRecMemCmpSIP(r, addr)       rwrec_MemCmpSIP(r, addr)
#  define rwRecApplyMaskSIP(r, addr)    rwrec_ApplyMaskSIP(r, addr)
#else
#  define rwRecMemGetSIP(r, out_addr)   _recMemGetSIP(r, out_addr)
#  define rwRecMemSetSIP(r, in_addr)    _recMemSetSIP(r, in_addr)
#  define rwRecMemCmpSIP(r, addr)       _recMemCmpSIP(r, addr)
#  define rwRecApplyMaskSIP(r, addr)    _recApplyMaskSIP(r, addr)
#endif /* RWREC_OPAQUE */


/***  Destination IP Address (dIP)  ***/

uint32_t rwrec_GetDIPv4(const rwRec *r);
void     rwrec_SetDIPv4(rwRec *r, uint32_t in_v);
void     rwrec_MemGetDIPv4(const rwRec *r, void *out_vp);
void     rwrec_MemSetDIPv4(rwRec *r, const void *in_vp);
int      rwrec_MemCmpDIPv4(const rwRec *r, const void *vp);
uint32_t rwrec_GetMaskDIPv4(const rwRec *r, uint32_t mask);
void     rwrec_ApplyMaskDIPv4(rwRec *r, uint32_t mask);

#define _recGetDIPv4(r)                         \
    ((r)->dIP.ipu_ipv4)
#define _recSetDIPv4(r, in_v)                   \
    { ((r)->dIP.ipu_ipv4) = (in_v); }
#define _recMemGetDIPv4(r, out_vp)                              \
    memcpy((out_vp), &((r)->dIP.ipu_ipv4), RWREC_SIZEOF_DIPv4)
#define _recMemSetDIPv4(r, in_vp)                               \
    memcpy(&((r)->dIP.ipu_ipv4), (in_vp), RWREC_SIZEOF_DIPv4)
#define _recMemCmpDIPv4(r, vp)                          \
    memcmp(&((r)->dIP.ipu_ipv4), (vp), RWREC_SIZEOF_DIPv4)
#define _recGetMaskDIPv4(r, mask)               \
    (((r)->dIP.ipu_ipv4) & mask)
#define _recApplyMaskDIPv4(r, mask)             \
    ipUnionApplyMaskV4(&((r)->dIP), (mask))


#if RWREC_OPAQUE
#  define rwRecGetDIPv4(r)              rwrec_GetDIPv4(r)
#  define rwRecSetDIPv4(r, in_v)        rwrec_SetDIPv4(r, in_v)
#  define rwRecMemGetDIPv4(r, out_vp)   rwrec_MemGetDIPv4(r, out_vp)
#  define rwRecMemSetDIPv4(r, in_vp)    rwrec_MemSetDIPv4(r, in_vp)
#  define rwRecMemCmpDIPv4(r, vp)       rwrec_MemCmpDIPv4(r, vp)
#  define rwRecGetMaskDIPv4(r, mask)    rwrec_GetMaskDIPv4(r, mask)
#  define rwRecApplyMaskDIPv4(r, mask)  rwrec_ApplyMaskDIPv4(r, mask)
#else
#  define rwRecGetDIPv4(r)              _recGetDIPv4(r)
#  define rwRecSetDIPv4(r, in_v)        _recSetDIPv4(r, in_v)
#  define rwRecMemGetDIPv4(r, out_vp)   _recMemGetDIPv4(r, out_vp)
#  define rwRecMemSetDIPv4(r, in_vp)    _recMemSetDIPv4(r, in_vp)
#  define rwRecMemCmpDIPv4(r, vp)       _recMemCmpDIPv4(r, vp)
#  define rwRecGetMaskDIPv4(r, mask)    _recGetMaskDIPv4(r, mask)
#  define rwRecApplyMaskDIPv4(r, mask)  _recApplyMaskDIPv4(r, mask)
#endif /* RWREC_OPAQUE */


/***  Destination IPv6 Address (dIP)  ***/

#if SK_ENABLE_IPV6

void     rwrec_MemGetDIPv6(const rwRec *r, void *out_vp);
void     rwrec_MemSetDIPv6(rwRec *r, const void *in_vp);
int      rwrec_MemCmpDIPv6(const rwRec *r, const void *vp);
void     rwrec_ApplyMaskDIPv6(rwRec *r, const void *mask_vp);

#define _recMemGetDIPv6(r, out_vp)                                      \
    if (_recIsIPv6(r)) {                                                \
        ipUnionGetV6(&((r)->dIP), (out_vp));                            \
    } else {                                                            \
        ipUnionGetV4AsV6(&((r)->dIP), (out_vp));                        \
    }
#define _recMemSetDIPv6(r, in_vp)                               \
    memcpy(&((r)->dIP.ipu_ipv6), (in_vp), RWREC_SIZEOF_DIPv6)
#define _recMemCmpDIPv6(r, vp)                                  \
    memcmp(&((r)->dIP.ipu_ipv6), (vp), RWREC_SIZEOF_DIPv6)
#define _recApplyMaskDIPv6(r, mask)             \
    ipUnionApplyMaskV6(&((r)->dIP), (mask))

#if RWREC_OPAQUE
#  define rwRecMemGetDIPv6(r, out_vp)   rwrec_MemGetDIPv6(r, out_vp)
#  define rwRecMemSetDIPv6(r, in_vp)    rwrec_MemSetDIPv6(r, in_vp)
#  define rwRecMemCmpDIPv6(r, vp)       rwrec_MemCmpDIPv6(r, vp)
#  define rwRecApplyMaskDIPv6(r, mask)  rwrec_ApplyMaskDIPv6(r, mask)
#else
#  define rwRecMemGetDIPv6(r, out_vp)   _recMemGetDIPv6(r, out_vp)
#  define rwRecMemSetDIPv6(r, in_vp)    _recMemSetDIPv6(r, in_vp)
#  define rwRecMemCmpDIPv6(r, vp)       _recMemCmpDIPv6(r, vp)
#  define rwRecApplyMaskDIPv6(r, mask)  _recApplyMaskDIPv6(r, mask)
#endif /* RWREC_OPAQUE */

#endif  /* SK_ENABLE_IPV6 */


/***  Destination IP Address (dIP) as skipaddr_t  ***/

void     rwrec_MemGetDIP(const rwRec *r, skipaddr_t *out_addr);
void     rwrec_MemSetDIP(rwRec *r, const skipaddr_t *in_addr);
int      rwrec_MemCmpDIP(const rwRec *r, const skipaddr_t *addr);
void     rwrec_ApplyMaskDIP(rwRec *r, const skipaddr_t *mask_addr);

#if !SK_ENABLE_IPV6
#define _recMemGetDIP _recMemGetDIPv4
#define _recMemSetDIP _recMemSetDIPv4
#define _recMemCmpDIP(r, addr)                          \
    _recMemCmpDIPv4((r), &((addr)->ip_ip.ipu_ipv4))
#define _recApplyMaskDIP(r, addr)                       \
    _recApplyMaskDIPv4((r), (addr)->ip_ip.ipu_ipv4)
#else
#define _recMemGetDIP(r, out_addr)                      \
    do {                                                \
        memcpy(out_addr, &((r)->dIP), sizeof(ipUnion)); \
        skipaddrSetVersion((out_addr), _recIsIPv6(r));  \
    } while(0)
#define _recMemSetDIP(r, in_addr)                               \
    do {                                                        \
        if (skipaddrIsV6(in_addr) == _recIsIPv6(r)) {           \
            /* both are either V4 or V6 */                      \
            memcpy(&((r)->dIP), (in_addr), sizeof(ipUnion));    \
        } else if (_recIsIPv6(r)) {                             \
            /* convert V4 IP to V6 */                           \
            ipUnion4to6(&((in_addr)->ip_ip), &((r)->dIP));      \
        } else {                                                \
            /* must convert record to V6 */                     \
            _recConvertToIPv6(r);                               \
            memcpy(&((r)->dIP), (in_addr), sizeof(ipUnion));    \
        }                                                       \
    } while(0)
#define _recMemCmpDIP(r, addr)      rwrec_MemCmpDIP(r, addr)
#define _recApplyMaskDIP(r, addr)   rwrec_ApplyMaskDIP(r, addr)
#endif /* SK_ENABLE_IPV6 */

#if RWREC_OPAQUE
#  define rwRecMemGetDIP(r, out_addr)   rwrec_MemGetDIP(r, out_addr)
#  define rwRecMemSetDIP(r, in_addr)    rwrec_MemSetDIP(r, in_addr)
#  define rwRecMemCmpDIP(r, addr)       rwrec_MemCmpDIP(r, addr)
#  define rwRecApplyMaskDIP(r, addr)    rwrec_ApplyMaskDIP(r, addr)
#else
#  define rwRecMemGetDIP(r, out_addr)   _recMemGetDIP(r, out_addr)
#  define rwRecMemSetDIP(r, in_addr)    _recMemSetDIP(r, in_addr)
#  define rwRecMemCmpDIP(r, addr)       _recMemCmpDIP(r, addr)
#  define rwRecApplyMaskDIP(r, addr)    _recApplyMaskDIP(r, addr)
#endif /* RWREC_OPAQUE */


/***  Next Hop IP (nhIP) Address  ***/

uint32_t rwrec_GetNhIPv4(const rwRec *r);
void     rwrec_SetNhIPv4(rwRec *r, uint32_t in_v);
void     rwrec_MemGetNhIPv4(const rwRec *r, void *out_vp);
void     rwrec_MemSetNhIPv4(rwRec *r, const void *in_vp);
int      rwrec_MemCmpNhIPv4(const rwRec *r, const void *vp);
uint32_t rwrec_GetMaskNhIPv4(const rwRec *r, uint32_t mask);
void     rwrec_ApplyMaskNhIPv4(rwRec *r, uint32_t mask);

#define _recGetNhIPv4(r)                        \
    ((r)->nhIP.ipu_ipv4)
#define _recSetNhIPv4(r, in_v)                  \
    { ((r)->nhIP.ipu_ipv4) = (in_v); }
#define _recMemGetNhIPv4(r, out_vp)                             \
    memcpy((out_vp), &((r)->nhIP.ipu_ipv4), RWREC_SIZEOF_NHIPv4)
#define _recMemSetNhIPv4(r, in_vp)                              \
    memcpy(&((r)->nhIP.ipu_ipv4), (in_vp), RWREC_SIZEOF_NHIPv4)
#define _recMemCmpNhIPv4(r, vp)                         \
    memcmp(&((r)->nhIP.ipu_ipv4), (vp), RWREC_SIZEOF_NHIPv4)
#define _recGetMaskNhIPv4(r, mask)              \
    (((r)->nhIP.ipu_ipv4) & mask)
#define _recApplyMaskNhIPv4(r, mask)            \
    ipUnionApplyMaskV4(&((r)->nhIP), (mask))


#if RWREC_OPAQUE
#  define rwRecGetNhIPv4(r)             rwrec_GetNhIPv4(r)
#  define rwRecSetNhIPv4(r, in_v)       rwrec_SetNhIPv4(r, in_v)
#  define rwRecMemGetNhIPv4(r, out_vp)  rwrec_MemGetNhIPv4(r, out_vp)
#  define rwRecMemSetNhIPv4(r, in_vp)   rwrec_MemSetNhIPv4(r, in_vp)
#  define rwRecMemCmpNhIPv4(r, vp)      rwrec_MemCmpNhIPv4(r, vp)
#  define rwRecGetMaskNhIPv4(r, mask)   rwrec_GetMaskNhIPv4(r, mask)
#  define rwRecApplyMaskNhIPv4(r, mask) rwrec_ApplyMaskNhIPv4(r, mask)
#else
#  define rwRecGetNhIPv4(r)             _recGetNhIPv4(r)
#  define rwRecSetNhIPv4(r, in_v)       _recSetNhIPv4(r, in_v)
#  define rwRecMemGetNhIPv4(r, out_vp)  _recMemGetNhIPv4(r, out_vp)
#  define rwRecMemSetNhIPv4(r, in_vp)   _recMemSetNhIPv4(r, in_vp)
#  define rwRecMemCmpNhIPv4(r, vp)      _recMemCmpNhIPv4(r, vp)
#  define rwRecGetMaskNhIPv4(r, mask)   _recGetMaskNhIPv4(r, mask)
#  define rwRecApplyMaskNhIPv4(r, mask) _recApplyMaskNhIPv4(r, mask)
#endif /* RWREC_OPAQUE */


/***  Next Hop IPv6 Address (nhIP)  ***/

#if SK_ENABLE_IPV6

void     rwrec_MemGetNhIPv6(const rwRec *r, void *out_vp);
void     rwrec_MemSetNhIPv6(rwRec *r, const void *in_vp);
int      rwrec_MemCmpNhIPv6(const rwRec *r, const void *vp);
void     rwrec_ApplyMaskNhIPv6(rwRec *r, const void *mask_vp);

#define _recMemGetNhIPv6(r, out_vp)                                     \
    if (_recIsIPv6(r)) {                                                \
        ipUnionGetV6(&((r)->nhIP), (out_vp));                           \
    } else {                                                            \
        ipUnionGetV4AsV6(&((r)->nhIP), (out_vp));                       \
    }
#define _recMemSetNhIPv6(r, in_vp)                              \
    memcpy(&((r)->nhIP.ipu_ipv6), (in_vp), RWREC_SIZEOF_NHIPv6)
#define _recMemCmpNhIPv6(r, vp)                                 \
    memcmp(&((r)->nhIP.ipu_ipv6), (vp), RWREC_SIZEOF_NHIPv6)
#define _recApplyMaskNhIPv6(r, mask)             \
    ipUnionApplyMaskV6(&((r)->nhIP), (mask))

#if RWREC_OPAQUE
#  define rwRecMemGetNhIPv6(r, out_vp)   rwrec_MemGetNhIPv6(r, out_vp)
#  define rwRecMemSetNhIPv6(r, in_vp)    rwrec_MemSetNhIPv6(r, in_vp)
#  define rwRecMemCmpNhIPv6(r, vp)       rwrec_MemCmpNhIPv6(r, vp)
#  define rwRecApplyMaskNhIPv6(r, mask)  rwrec_ApplyMaskNhIPv6(r, mask)
#else
#  define rwRecMemGetNhIPv6(r, out_vp)   _recMemGetNhIPv6(r, out_vp)
#  define rwRecMemSetNhIPv6(r, in_vp)    _recMemSetNhIPv6(r, in_vp)
#  define rwRecMemCmpNhIPv6(r, vp)       _recMemCmpNhIPv6(r, vp)
#  define rwRecApplyMaskNhIPv6(r, mask)  _recApplyMaskNhIPv6(r, mask)
#endif /* RWREC_OPAQUE */

#endif /* SK_ENABLE_IPV6 */


/***  Next Hop IP Address (nhIP) as skipaddr_t  ***/

void     rwrec_MemGetNhIP(const rwRec *r, skipaddr_t *out_addr);
void     rwrec_MemSetNhIP(rwRec *r, const skipaddr_t *in_addr);
int      rwrec_MemCmpNhIP(const rwRec *r, const skipaddr_t *addr);
void     rwrec_ApplyMaskNhIP(rwRec *r, const skipaddr_t *mask_addr);

#if !SK_ENABLE_IPV6
#define _recMemGetNhIP    _recMemGetNhIPv4
#define _recMemSetNhIP    _recMemSetNhIPv4
#define _recMemCmpNhIP(r, addr)                         \
    _recMemCmpNhIPv4((r), &((addr)->ip_ip.ipu_ipv4))
#define _recApplyMaskNhIP(r, addr)                      \
    _recApplyMaskNhIPv4((r), (addr)->ip_ip.ipu_ipv4)
#else
#define _recMemGetNhIP(r, out_addr)                             \
    do {                                                        \
        memcpy(out_addr, &((r)->nhIP), sizeof(ipUnion));        \
        skipaddrSetVersion((out_addr), _recIsIPv6(r));          \
    } while(0)
#define _recMemSetNhIP(r, in_addr)                              \
    do {                                                        \
        if (skipaddrIsV6(in_addr) == _recIsIPv6(r)) {           \
            /* both are either V4 or V6 */                      \
            memcpy(&((r)->nhIP), (in_addr), sizeof(ipUnion));   \
        } else if (_recIsIPv6(r)) {                             \
            /* convert V4 IP to V6 */                           \
            ipUnion4to6(&((in_addr)->ip_ip), &((r)->nhIP));     \
        } else {                                                \
            /* must convert record to V6 */                     \
            _recConvertToIPv6(r);                               \
            memcpy(&((r)->nhIP), (in_addr), sizeof(ipUnion));   \
        }                                                       \
    } while(0)
#define _recMemCmpNhIP(r, addr)     rwrec_MemCmpNhIP(r, addr)
#define _recApplyMaskNhIP(r, addr)  rwrec_ApplyMaskNhIP(r, addr)
#endif /* SK_ENABLE_IPV6 */

#if RWREC_OPAQUE
#  define rwRecMemGetNhIP(r, out_addr)  rwrec_MemGetNhIP(r, out_addr)
#  define rwRecMemSetNhIP(r, in_addr)   rwrec_MemSetNhIP(r, in_addr)
#  define rwRecMemCmpNhIP(r, addr)      rwrec_MemCmpNhIP(r, addr)
#  define rwRecApplyMaskNhIP(r, addr)   rwrec_ApplyMaskNhIP(r, addr)
#else
#  define rwRecMemGetNhIP(r, out_addr)  _recMemGetNhIP(r, out_addr)
#  define rwRecMemSetNhIP(r, in_addr)   _recMemSetNhIP(r, in_addr)
#  define rwRecMemCmpNhIP(r, addr)      _recMemCmpNhIP(r, addr)
#  define rwRecApplyMaskNhIP(r, addr)   _recApplyMaskNhIP(r, addr)
#endif /* RWREC_OPAQUE */


/***  Source Port (sPort)  ***/

uint16_t rwrec_GetSPort(const rwRec *r);
void     rwrec_SetSPort(rwRec *r, uint16_t in_v);
void     rwrec_MemGetSPort(const rwRec *r, void *out_vp);
void     rwrec_MemSetSPort(rwRec *r, const void *in_vp);
int      rwrec_MemCmpSPort(const rwRec *r, const void *vp);

#define _recGetSPort(r)                         \
    ((r)->sPort)
#define _recSetSPort(r, in_v)                   \
    { ((r)->sPort) = (in_v); }
#define _recMemGetSPort(r, out_vp)                      \
    memcpy((out_vp), &((r)->sPort), RWREC_SIZEOF_SPORT)
#define _recMemSetSPort(r, in_vp)                       \
    memcpy(&((r)->sPort), (in_vp), RWREC_SIZEOF_SPORT)
#define _recMemCmpSPort(r, vp)                          \
    memcmp(&((r)->sPort), (vp), RWREC_SIZEOF_SPORT)

#if RWREC_OPAQUE
#  define rwRecGetSPort(r)  rwrec_GetSPort(r)
#  define rwRecSetSPort(r, in_v)  rwrec_SetSPort(r, in_v)
#  define rwRecMemGetSPort(r, out_vp)  rwrec_MemGetSPort(r, out_vp)
#  define rwRecMemSetSPort(r, in_vp)  rwrec_MemSetSPort(r, in_vp)
#  define rwRecMemCmpSPort(r, vp)  rwrec_MemCmpSPort(r, vp)
#else
#  define rwRecGetSPort(r)  _recGetSPort(r)
#  define rwRecSetSPort(r, in_v)  _recSetSPort(r, in_v)
#  define rwRecMemGetSPort(r, out_vp)  _recMemGetSPort(r, out_vp)
#  define rwRecMemSetSPort(r, in_vp)  _recMemSetSPort(r, in_vp)
#  define rwRecMemCmpSPort(r, vp)  _recMemCmpSPort(r, vp)
#endif /* RWREC_OPAQUE */


/***  Destination Port (dPort)  ***/

uint16_t rwrec_GetDPort(const rwRec *r);
void     rwrec_SetDPort(rwRec *r, uint16_t in_v);
void     rwrec_MemGetDPort(const rwRec *r, void *out_vp);
void     rwrec_MemSetDPort(rwRec *r, const void *in_vp);
int      rwrec_MemCmpDPort(const rwRec *r, const void *vp);

#define _recGetDPort(r)                         \
    ((r)->dPort)
#define _recSetDPort(r, in_v)                   \
    { ((r)->dPort) = (in_v); }
#define _recMemGetDPort(r, out_vp)                      \
    memcpy((out_vp), &((r)->dPort), RWREC_SIZEOF_DPORT)
#define _recMemSetDPort(r, in_vp)                       \
    memcpy(&((r)->dPort), (in_vp), RWREC_SIZEOF_DPORT)
#define _recMemCmpDPort(r, vp)                          \
    memcmp(&((r)->dPort), (vp), RWREC_SIZEOF_DPORT)

#if RWREC_OPAQUE
#  define rwRecGetDPort(r)  rwrec_GetDPort(r)
#  define rwRecSetDPort(r, in_v)  rwrec_SetDPort(r, in_v)
#  define rwRecMemGetDPort(r, out_vp)  rwrec_MemGetDPort(r, out_vp)
#  define rwRecMemSetDPort(r, in_vp)  rwrec_MemSetDPort(r, in_vp)
#  define rwRecMemCmpDPort(r, vp)  rwrec_MemCmpDPort(r, vp)
#else
#  define rwRecGetDPort(r)  _recGetDPort(r)
#  define rwRecSetDPort(r, in_v)  _recSetDPort(r, in_v)
#  define rwRecMemGetDPort(r, out_vp)  _recMemGetDPort(r, out_vp)
#  define rwRecMemSetDPort(r, in_vp)  _recMemSetDPort(r, in_vp)
#  define rwRecMemCmpDPort(r, vp)  _recMemCmpDPort(r, vp)
#endif /* RWREC_OPAQUE */


/***  Protocol  ***/

uint8_t  rwrec_GetProto(const rwRec *r);
void     rwrec_SetProto(rwRec *r, uint8_t in_v);
void     rwrec_MemGetProto(const rwRec *r, void *out_vp);
void     rwrec_MemSetProto(rwRec *r, const void *in_vp);
int      rwrec_MemCmpProto(const rwRec *r, const void *vp);

#define _recGetProto(r)                         \
    ((r)->proto)
#define _recSetProto(r, in_v)                   \
    { ((r)->proto) = (in_v); }
#define _recMemGetProto(r, out_vp)                      \
    memcpy((out_vp), &((r)->proto), RWREC_SIZEOF_PROTO)
#define _recMemSetProto(r, in_vp)                       \
    memcpy(&((r)->proto), (in_vp), RWREC_SIZEOF_PROTO)
#define _recMemCmpProto(r, vp)                          \
    memcmp(&((r)->proto), (vp), RWREC_SIZEOF_PROTO)

#if RWREC_OPAQUE
#  define rwRecGetProto(r)  rwrec_GetProto(r)
#  define rwRecSetProto(r, in_v)  rwrec_SetProto(r, in_v)
#  define rwRecMemGetProto(r, out_vp)  rwrec_MemGetProto(r, out_vp)
#  define rwRecMemSetProto(r, in_vp)  rwrec_MemSetProto(r, in_vp)
#  define rwRecMemCmpProto(r, vp)  rwrec_MemCmpProto(r, vp)
#else
#  define rwRecGetProto(r)  _recGetProto(r)
#  define rwRecSetProto(r, in_v)  _recSetProto(r, in_v)
#  define rwRecMemGetProto(r, out_vp)  _recMemGetProto(r, out_vp)
#  define rwRecMemSetProto(r, in_vp)  _recMemSetProto(r, in_vp)
#  define rwRecMemCmpProto(r, vp)  _recMemCmpProto(r, vp)
#endif /* RWREC_OPAQUE */


/***  Packet Count (pkts)  ***/

uint32_t rwrec_GetPkts(const rwRec *r);
void     rwrec_SetPkts(rwRec *r, uint32_t in_v);
void     rwrec_MemGetPkts(const rwRec *r, void *out_vp);
void     rwrec_MemSetPkts(rwRec *r, const void *in_vp);
int      rwrec_MemCmpPkts(const rwRec *r, const void *vp);

#define _recGetPkts(r)                          \
    ((r)->pkts)
#define _recSetPkts(r, in_v)                    \
    { ((r)->pkts) = (in_v); }
#define _recMemGetPkts(r, out_vp)                       \
    memcpy((out_vp), &((r)->pkts), RWREC_SIZEOF_PKTS)
#define _recMemSetPkts(r, in_vp)                        \
    memcpy(&((r)->pkts), (in_vp), RWREC_SIZEOF_PKTS)
#define _recMemCmpPkts(r, vp)                           \
    memcmp(&((r)->pkts), (vp), RWREC_SIZEOF_PKTS)

#if RWREC_OPAQUE
#  define rwRecGetPkts(r)  rwrec_GetPkts(r)
#  define rwRecSetPkts(r, in_v)  rwrec_SetPkts(r, in_v)
#  define rwRecMemGetPkts(r, out_vp)  rwrec_MemGetPkts(r, out_vp)
#  define rwRecMemSetPkts(r, in_vp)  rwrec_MemSetPkts(r, in_vp)
#  define rwRecMemCmpPkts(r, vp)  rwrec_MemCmpPkts(r, vp)
#else
#  define rwRecGetPkts(r)  _recGetPkts(r)
#  define rwRecSetPkts(r, in_v)  _recSetPkts(r, in_v)
#  define rwRecMemGetPkts(r, out_vp)  _recMemGetPkts(r, out_vp)
#  define rwRecMemSetPkts(r, in_vp)  _recMemSetPkts(r, in_vp)
#  define rwRecMemCmpPkts(r, vp)  _recMemCmpPkts(r, vp)
#endif /* RWREC_OPAQUE */


/***  Byte count  ***/

uint32_t rwrec_GetBytes(const rwRec *r);
void     rwrec_SetBytes(rwRec *r, uint32_t in_v);
void     rwrec_MemGetBytes(const rwRec *r, void *out_vp);
void     rwrec_MemSetBytes(rwRec *r, const void *in_vp);
int      rwrec_MemCmpBytes(const rwRec *r, const void *vp);

#define _recGetBytes(r)                         \
    ((r)->bytes)
#define _recSetBytes(r, in_v)                   \
    { ((r)->bytes) = (in_v); }
#define _recMemGetBytes(r, out_vp)                      \
    memcpy((out_vp), &((r)->bytes), RWREC_SIZEOF_BYTES)
#define _recMemSetBytes(r, in_vp)                       \
    memcpy(&((r)->bytes), (in_vp), RWREC_SIZEOF_BYTES)
#define _recMemCmpBytes(r, vp)                          \
    memcmp(&((r)->bytes), (vp), RWREC_SIZEOF_BYTES)

#if RWREC_OPAQUE
#  define rwRecGetBytes(r)  rwrec_GetBytes(r)
#  define rwRecSetBytes(r, in_v)  rwrec_SetBytes(r, in_v)
#  define rwRecMemGetBytes(r, out_vp)  rwrec_MemGetBytes(r, out_vp)
#  define rwRecMemSetBytes(r, in_vp)  rwrec_MemSetBytes(r, in_vp)
#  define rwRecMemCmpBytes(r, vp)  rwrec_MemCmpBytes(r, vp)
#else
#  define rwRecGetBytes(r)  _recGetBytes(r)
#  define rwRecSetBytes(r, in_v)  _recSetBytes(r, in_v)
#  define rwRecMemGetBytes(r, out_vp)  _recMemGetBytes(r, out_vp)
#  define rwRecMemSetBytes(r, in_vp)  _recMemSetBytes(r, in_vp)
#  define rwRecMemCmpBytes(r, vp)  _recMemCmpBytes(r, vp)
#endif /* RWREC_OPAQUE */


/***  Bitwise OR of TCP Flags on ALL packets in flow  ***/

uint8_t  rwrec_GetFlags(const rwRec *r);
void     rwrec_SetFlags(rwRec *r, uint8_t in_v);
void     rwrec_MemGetFlags(const rwRec *r, void *out_vp);
void     rwrec_MemSetFlags(rwRec *r, const void *in_vp);
int      rwrec_MemCmpFlags(const rwRec *r, const void *vp);

#define _recGetFlags(r)                         \
    ((r)->flags)
#define _recSetFlags(r, in_v)                   \
    { ((r)->flags) = (in_v); }
#define _recMemGetFlags(r, out_vp)                      \
    memcpy((out_vp), &((r)->flags), RWREC_SIZEOF_FLAGS)
#define _recMemSetFlags(r, in_vp)                       \
    memcpy(&((r)->flags), (in_vp), RWREC_SIZEOF_FLAGS)
#define _recMemCmpFlags(r, vp)                          \
    memcmp(&((r)->flags), (vp), RWREC_SIZEOF_FLAGS)

#if RWREC_OPAQUE
#  define rwRecGetFlags(r)  rwrec_GetFlags(r)
#  define rwRecSetFlags(r, in_v)  rwrec_SetFlags(r, in_v)
#  define rwRecMemGetFlags(r, out_vp)  rwrec_MemGetFlags(r, out_vp)
#  define rwRecMemSetFlags(r, in_vp)  rwrec_MemSetFlags(r, in_vp)
#  define rwRecMemCmpFlags(r, vp)  rwrec_MemCmpFlags(r, vp)
#else
#  define rwRecGetFlags(r)  _recGetFlags(r)
#  define rwRecSetFlags(r, in_v)  _recSetFlags(r, in_v)
#  define rwRecMemGetFlags(r, out_vp)  _recMemGetFlags(r, out_vp)
#  define rwRecMemSetFlags(r, in_vp)  _recMemSetFlags(r, in_vp)
#  define rwRecMemCmpFlags(r, vp)  _recMemCmpFlags(r, vp)
#endif /* RWREC_OPAQUE */


/***  TCP Flags seen on initial packet of flow  ***/

uint8_t  rwrec_GetInitFlags(const rwRec *r);
void     rwrec_SetInitFlags(rwRec *r, uint8_t in_v);
void     rwrec_MemGetInitFlags(const rwRec *r, void *out_vp);
void     rwrec_MemSetInitFlags(rwRec *r, const void *in_vp);
int      rwrec_MemCmpInitFlags(const rwRec *r, const void *vp);

#define _recGetInitFlags(r)                     \
    ((r)->init_flags)
#define _recSetInitFlags(r, in_v)               \
    { ((r)->init_flags) = (in_v); }
#define _recMemGetInitFlags(r, out_vp)                                  \
    memcpy((out_vp), &((r)->init_flags), RWREC_SIZEOF_INIT_FLAGS)
#define _recMemSetInitFlags(r, in_vp)                                   \
    memcpy(&((r)->init_flags), (in_vp), RWREC_SIZEOF_INIT_FLAGS)
#define _recMemCmpInitFlags(r, vp)                              \
    memcmp(&((r)->init_flags), (vp), RWREC_SIZEOF_INIT_FLAGS)

#if RWREC_OPAQUE
#  define rwRecGetInitFlags(r)  rwrec_GetInitFlags(r)
#  define rwRecSetInitFlags(r, in_v)  rwrec_SetInitFlags(r, in_v)
#  define rwRecMemGetInitFlags(r, out_vp)  rwrec_MemGetInitFlags(r, out_vp)
#  define rwRecMemSetInitFlags(r, in_vp)  rwrec_MemSetInitFlags(r, in_vp)
#  define rwRecMemCmpInitFlags(r, vp)  rwrec_MemCmpInitFlags(r, vp)
#else
#  define rwRecGetInitFlags(r)  _recGetInitFlags(r)
#  define rwRecSetInitFlags(r, in_v)  _recSetInitFlags(r, in_v)
#  define rwRecMemGetInitFlags(r, out_vp)  _recMemGetInitFlags(r, out_vp)
#  define rwRecMemSetInitFlags(r, in_vp)  _recMemSetInitFlags(r, in_vp)
#  define rwRecMemCmpInitFlags(r, vp)  _recMemCmpInitFlags(r, vp)
#endif /* RWREC_OPAQUE */


/***  Bitwise OR of TCP Flags on all packets in session except first  ***/

uint8_t  rwrec_GetRestFlags(const rwRec *r);
void     rwrec_SetRestFlags(rwRec *r, uint8_t in_v);
void     rwrec_MemGetRestFlags(const rwRec *r, void *out_vp);
void     rwrec_MemSetRestFlags(rwRec *r, const void *in_vp);
int      rwrec_MemCmpRestFlags(const rwRec *r, const void *vp);

#define _recGetRestFlags(r)                     \
    ((r)->rest_flags)
#define _recSetRestFlags(r, in_v)               \
    { ((r)->rest_flags) = (in_v); }
#define _recMemGetRestFlags(r, out_vp)                                  \
    memcpy((out_vp), &((r)->rest_flags), RWREC_SIZEOF_REST_FLAGS)
#define _recMemSetRestFlags(r, in_vp)                                   \
    memcpy(&((r)->rest_flags), (in_vp), RWREC_SIZEOF_REST_FLAGS)
#define _recMemCmpRestFlags(r, vp)                              \
    memcmp(&((r)->rest_flags), (vp), RWREC_SIZEOF_REST_FLAGS)

#if RWREC_OPAQUE
#  define rwRecGetRestFlags(r)  rwrec_GetRestFlags(r)
#  define rwRecSetRestFlags(r, in_v)  rwrec_SetRestFlags(r, in_v)
#  define rwRecMemGetRestFlags(r, out_vp)  rwrec_MemGetRestFlags(r, out_vp)
#  define rwRecMemSetRestFlags(r, in_vp)  rwrec_MemSetRestFlags(r, in_vp)
#  define rwRecMemCmpRestFlags(r, vp)  rwrec_MemCmpRestFlags(r, vp)
#else
#  define rwRecGetRestFlags(r)  _recGetRestFlags(r)
#  define rwRecSetRestFlags(r, in_v)  _recSetRestFlags(r, in_v)
#  define rwRecMemGetRestFlags(r, out_vp)  _recMemGetRestFlags(r, out_vp)
#  define rwRecMemSetRestFlags(r, in_vp)  _recMemSetRestFlags(r, in_vp)
#  define rwRecMemCmpRestFlags(r, vp)  _recMemCmpRestFlags(r, vp)
#endif /* RWREC_OPAQUE */


/***  Start Time as milliseconds since UNIX epoch (sTime)  ***/

sktime_t rwrec_GetStartTime(const rwRec *r);
void     rwrec_SetStartTime(rwRec *r, sktime_t in_v);
void     rwrec_MemGetStartTime(const rwRec *r, void *out_vp);
void     rwrec_MemSetStartTime(rwRec *r, const void *in_vp);
int      rwrec_MemCmpStartTime(const rwRec *r, const void *vp);
uint32_t rwrec_GetStartSeconds(const rwRec *r);
void     rwrec_MemGetStartSeconds(const rwRec *r, void *out_vp);
uint16_t rwrec_GetStartMSec(const rwRec *r);
void     rwrec_MemGetStartMSec(const rwRec *r, void *out_vp);

#define _recGetStartTime(r)                     \
    ((r)->sTime)
#define _recSetStartTime(r, in_v)               \
    { ((r)->sTime = (in_v)); }
#define _recMemGetStartTime(r, out_vp)                  \
    memcpy((out_vp), &((r)->sTime), RWREC_SIZEOF_STIME)
#define _recMemSetStartTime(r, in_vp)                   \
    memcpy(&((r)->sTime), (in_vp), RWREC_SIZEOF_STIME)
#define _recMemCmpStartTime(r, vp)                      \
    memcmp(&((r)->sTime), (vp), RWREC_SIZEOF_STIME)

#define _recGetStartSeconds(r)                  \
    ((uint32_t)((r)->sTime / 1000))
#define _recMemGetStartSeconds(r, out_vp) {     \
        uint32_t _t = _recGetStartSeconds(r);   \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }

#define _recGetStartMSec(r)                     \
    ((uint16_t)((r)->sTime % 1000))
#define _recMemGetStartMSec(r, out_vp) {        \
        uint16_t _t = _recGetStartMSec(r);      \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }

#if RWREC_OPAQUE
#  define rwRecGetStartTime(r)  rwrec_GetStartTime(r)
#  define rwRecSetStartTime(r, in_v)  rwrec_SetStartTime(r, in_v)
#  define rwRecMemGetStartTime(r, out_vp)  rwrec_MemGetStartTime(r, out_vp)
#  define rwRecMemSetStartTime(r, in_vp)  rwrec_MemSetStartTime(r, in_vp)
#  define rwRecMemCmpStartTime(r, vp)  rwrec_MemCmpStartTime(r, vp)
#  define rwRecGetStartSeconds(r)  rwrec_GetStartSeconds(r)
#  define rwRecMemGetStartSeconds(r, out_vp)    \
    rwrec_MemGetStartSeconds(r, out_vp)
#  define rwRecGetStartMSec(r)  rwrec_GetStartMSec(r)
#  define rwRecMemGetStartMSec(r, out_vp)  rwrec_MemGetStartMSec(r, out_vp)
#else
#  define rwRecGetStartTime(r)  _recGetStartTime(r)
#  define rwRecSetStartTime(r, in_v)  _recSetStartTime(r, in_v)
#  define rwRecMemGetStartTime(r, out_vp)  _recMemGetStartTime(r, out_vp)
#  define rwRecMemSetStartTime(r, in_vp)  _recMemSetStartTime(r, in_vp)
#  define rwRecMemCmpStartTime(r, vp)  _recMemCmpStartTime(r, vp)
#  define rwRecGetStartSeconds(r)  _recGetStartSeconds(r)
#  define rwRecMemGetStartSeconds(r, out_vp)  _recMemGetStartSeconds(r, out_vp)
#  define rwRecGetStartMSec(r)  _recGetStartMSec(r)
#  define rwRecMemGetStartMSec(r, out_vp)  _recMemGetStartMSec(r, out_vp)
#endif /* RWREC_OPAQUE */


/***  Elapsed (duration) of the flow, in milliseconds  ***/

uint32_t rwrec_GetElapsed(const rwRec *r);
void     rwrec_SetElapsed(rwRec *r, sktime_t in_v);
void     rwrec_MemGetElapsed(const rwRec *r, void *out_vp);
void     rwrec_MemSetElapsed(rwRec *r, const void *in_vp);
int      rwrec_MemCmpElapsed(const rwRec *r, const void *vp);
uint32_t rwrec_GetElapsedSeconds(const rwRec *r);
void     rwrec_MemGetElapsedSeconds(const rwRec *r, void *out_vp);
uint16_t rwrec_GetElapsedMSec(const rwRec *r);
void     rwrec_MemGetElapsedMSec(const rwRec *r, void *out_vp);

#define _recGetElapsed(r)                       \
    ((r)->elapsed)
#define _recSetElapsed(r, in_v)                 \
    { (r)->elapsed = (uint32_t)(in_v); }
#define _recMemGetElapsed(r, out_vp)                            \
    memcpy((out_vp), &((r)->elapsed), RWREC_SIZEOF_ELAPSED)
#define _recMemSetElapsed(r, in_vp)                             \
    memcpy(&((r)->elapsed), (in_vp), RWREC_SIZEOF_ELAPSED)
#define _recMemCmpElapsed(r, vp)                                \
    memcmp(&((r)->elapsed), (vp), RWREC_SIZEOF_ELAPSED)

#define _recGetElapsedSeconds(r)                \
    ((uint32_t)((r)->elapsed / 1000))
#define _recMemGetElapsedSeconds(r, out_vp) {   \
        uint32_t _t = _recGetElapsedSeconds(r); \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }

#define _recGetElapsedMSec(r)                   \
    ((uint16_t)((r)->elapsed % 1000))
#define _recMemGetElapsedMSec(r, out_vp) {      \
        uint16_t _t = _recGetElapsedMSec(r);    \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }


#if RWREC_OPAQUE
#  define rwRecGetElapsed(r)  rwrec_GetElapsed(r)
#  define rwRecSetElapsed(r, in_v)  rwrec_SetElapsed(r, in_v)
#  define rwRecMemGetElapsed(r, out_vp)  rwrec_MemGetElapsed(r, out_vp)
#  define rwRecMemSetElapsed(r, in_vp)  rwrec_MemSetElapsed(r, in_vp)
#  define rwRecMemCmpElapsed(r, vp)  rwrec_MemCmpElapsed(r, vp)
#  define rwRecGetElapsedSeconds(r)  rwrec_GetElapsedSeconds(r)
#  define rwRecMemGetElapsedSeconds(r, out_vp)  \
    rwrec_MemGetElapsedSeconds(r, out_vp)
#  define rwRecGetElapsedMSec(r)  rwrec_GetElapsedMSec(r)
#  define rwRecMemGetElapsedMSec(r, out_vp)  rwrec_MemGetElapsedMSec(r, out_vp)
#else
#  define rwRecGetElapsed(r)  _recGetElapsed(r)
#  define rwRecSetElapsed(r, in_v)  _recSetElapsed(r, in_v)
#  define rwRecMemGetElapsed(r, out_vp)  _recMemGetElapsed(r, out_vp)
#  define rwRecMemSetElapsed(r, in_vp)  _recMemSetElapsed(r, in_vp)
#  define rwRecMemCmpElapsed(r, vp)  _recMemCmpElapsed(r, vp)
#  define rwRecGetElapsedSeconds(r)  _recGetElapsedSeconds(r)
#  define rwRecMemGetElapsedSeconds(r, out_vp)  \
    _recMemGetElapsedSeconds(r, out_vp)
#  define rwRecGetElapsedMSec(r)  _recGetElapsedMSec(r)
#  define rwRecMemGetElapsedMSec(r, out_vp)  _recMemGetElapsedMSec(r, out_vp)
#endif /* RWREC_OPAQUE */



/***  End Time is derived from the sTime and duration (eTime)  ***/

/* No Set macros/functions since this is a derived field */

sktime_t rwrec_GetEndTime(const rwRec *r);
void     rwrec_MemGetEndTime(const rwRec *r, void *out_vp);
uint32_t rwrec_GetEndSeconds(const rwRec *r);
void     rwrec_MemGetEndSeconds(const rwRec *r, void *out_vp);
uint16_t rwrec_GetEndMSec(const rwRec *r);
void     rwrec_MemGetEndMSec(const rwRec *r, void *out_vp);

#define _recGetEndTime(r)                               \
    ((sktime_t)_recGetStartTime(r) + _recGetElapsed(r))
#define _recMemGetEndTime(r, out_vp) {          \
        sktime_t _t = _recGetEndTime(r);        \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }
#define _recGetEndSeconds(r)                    \
    ((uint32_t)(_recGetEndTime(r) / 1000))
#define _recMemGetEndSeconds(r, out_vp) {       \
        uint32_t _t = rwRecGetEndSeconds(r);    \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }

#define _recGetEndMSec(r)                       \
    ((uint16_t)(_recGetEndTime(r) % 1000))
#define _recMemGetEndMSec(r, out_vp) {          \
        uint16_t _t = _recGetEndMSec(r);        \
        memcpy((out_vp), &_t, sizeof(_t));      \
    }

#if RWREC_OPAQUE
#  define rwRecGetEndTime(r)  rwrec_GetEndTime(r)
#  define rwRecMemGetEndTime(r, out_vp)  rwrec_MemGetEndTime(r, out_vp)
#  define rwRecGetEndSeconds(r)  rwrec_GetEndSeconds(r)
#  define rwRecMemGetEndSeconds(r, out_vp)  rwrec_MemGetEndSeconds(r, out_vp)
#  define rwRecGetEndMSec(r)  rwrec_GetEndMSec(r)
#  define rwRecMemGetEndMSec(r, out_vp)  rwrec_MemGetEndMSec(r, out_vp)
#else
#  define rwRecGetEndTime(r)  _recGetEndTime(r)
#  define rwRecMemGetEndTime(r, out_vp)  _recMemGetEndTime(r, out_vp)
#  define rwRecGetEndSeconds(r)  _recGetEndSeconds(r)
#  define rwRecMemGetEndSeconds(r, out_vp)  _recMemGetEndSeconds(r, out_vp)
#  define rwRecGetEndMSec(r)  _recGetEndMSec(r)
#  define rwRecMemGetEndMSec(r, out_vp)  _recMemGetEndMSec(r, out_vp)
#endif /* RWREC_OPAQUE */


/***  Sensor ID (sID)  ***/

sensorID_t rwrec_GetSensor(const rwRec *r);
void     rwrec_SetSensor(rwRec *r, sensorID_t in_v);
void     rwrec_MemGetSensor(const rwRec *r, void *out_vp);
void     rwrec_MemSetSensor(rwRec *r, const void *in_vp);
int      rwrec_MemCmpSensor(const rwRec *r, const void *vp);

#define _recGetSensor(r)                        \
    ((r)->sID)
#define _recSetSensor(r, in_v)                  \
    { ((r)->sID) = (in_v); }
#define _recMemGetSensor(r, out_vp)                     \
    memcpy((out_vp), &((r)->sID), RWREC_SIZEOF_SID)
#define _recMemSetSensor(r, in_vp)                      \
    memcpy(&((r)->sID), (in_vp), RWREC_SIZEOF_SID)
#define _recMemCmpSensor(r, vp)                 \
    memcmp(&((r)->sID), (vp), RWREC_SIZEOF_SID)

#if RWREC_OPAQUE
#  define rwRecGetSensor(r)  rwrec_GetSensor(r)
#  define rwRecSetSensor(r, in_v)  rwrec_SetSensor(r, in_v)
#  define rwRecMemGetSensor(r, out_vp)  rwrec_MemGetSensor(r, out_vp)
#  define rwRecMemSetSensor(r, in_vp)  rwrec_MemSetSensor(r, in_vp)
#  define rwRecMemCmpSensor(r, vp)  rwrec_MemCmpSensor(r, vp)
#else
#  define rwRecGetSensor(r)  _recGetSensor(r)
#  define rwRecSetSensor(r, in_v)  _recSetSensor(r, in_v)
#  define rwRecMemGetSensor(r, out_vp)  _recMemGetSensor(r, out_vp)
#  define rwRecMemSetSensor(r, in_vp)  _recMemSetSensor(r, in_vp)
#  define rwRecMemCmpSensor(r, vp)  _recMemCmpSensor(r, vp)
#endif /* RWREC_OPAQUE */


/***  FlowType holds Class and Type  ***/

flowtypeID_t rwrec_GetFlowType(const rwRec *r);
void         rwrec_SetFlowType(rwRec *r, flowtypeID_t in_v);
void         rwrec_MemGetFlowType(const rwRec *r, void *out_vp);
void         rwrec_MemSetFlowType(rwRec *r, const void *in_vp);
int          rwrec_MemCmpFlowType(const rwRec *r, const void *vp);

#define _recGetFlowType(r)                      \
    ((r)->flow_type)
#define _recSetFlowType(r, in_v)                \
    { ((r)->flow_type) = (in_v); }
#define _recMemGetFlowType(r, out_vp)                           \
    memcpy((out_vp), &((r)->flow_type), RWREC_SIZEOF_FLOW_TYPE)
#define _recMemSetFlowType(r, in_vp)                            \
    memcpy(&((r)->flow_type), (in_vp), RWREC_SIZEOF_FLOW_TYPE)
#define _recMemCmpFlowType(r, vp)                               \
    memcmp(&((r)->flow_type), (vp), RWREC_SIZEOF_FLOW_TYPE)

#if RWREC_OPAQUE
#  define rwRecGetFlowType(r)  rwrec_GetFlowType(r)
#  define rwRecSetFlowType(r, in_v)  rwrec_SetFlowType(r, in_v)
#  define rwRecMemGetFlowType(r, out_vp)  rwrec_MemGetFlowType(r, out_vp)
#  define rwRecMemSetFlowType(r, in_vp)  rwrec_MemSetFlowType(r, in_vp)
#  define rwRecMemCmpFlowType(r, vp)  rwrec_MemCmpFlowType(r, vp)
#else
#  define rwRecGetFlowType(r)  _recGetFlowType(r)
#  define rwRecSetFlowType(r, in_v)  _recSetFlowType(r, in_v)
#  define rwRecMemGetFlowType(r, out_vp)  _recMemGetFlowType(r, out_vp)
#  define rwRecMemSetFlowType(r, in_vp)  _recMemSetFlowType(r, in_vp)
#  define rwRecMemCmpFlowType(r, vp)  _recMemCmpFlowType(r, vp)
#endif /* RWREC_OPAQUE */


/***  SNMP Input Value (Router incoming/ingress interface)  ***/

uint16_t rwrec_GetInput(const rwRec *r);
void     rwrec_SetInput(rwRec *r, uint16_t in_v);
void     rwrec_MemGetInput(const rwRec *r, void *out_vp);
void     rwrec_MemSetInput(rwRec *r, const void *in_vp);
int      rwrec_MemCmpInput(const rwRec *r, const void *vp);

#define _recGetInput(r)                         \
    ((r)->input)
#define _recSetInput(r, in_v)                   \
    { ((r)->input) = (in_v); }
#define _recMemGetInput(r, out_vp)                      \
    memcpy((out_vp), &((r)->input), RWREC_SIZEOF_INPUT)
#define _recMemSetInput(r, in_vp)                       \
    memcpy(&((r)->input), (in_vp), RWREC_SIZEOF_INPUT)
#define _recMemCmpInput(r, vp)                          \
    memcmp(&((r)->input), (vp), RWREC_SIZEOF_INPUT)

#if RWREC_OPAQUE
#  define rwRecGetInput(r)  rwrec_GetInput(r)
#  define rwRecSetInput(r, in_v)  rwrec_SetInput(r, in_v)
#  define rwRecMemGetInput(r, out_vp)  rwrec_MemGetInput(r, out_vp)
#  define rwRecMemSetInput(r, in_vp)  rwrec_MemSetInput(r, in_vp)
#  define rwRecMemCmpInput(r, vp)  rwrec_MemCmpInput(r, vp)
#else
#  define rwRecGetInput(r)  _recGetInput(r)
#  define rwRecSetInput(r, in_v)  _recSetInput(r, in_v)
#  define rwRecMemGetInput(r, out_vp)  _recMemGetInput(r, out_vp)
#  define rwRecMemSetInput(r, in_vp)  _recMemSetInput(r, in_vp)
#  define rwRecMemCmpInput(r, vp)  _recMemCmpInput(r, vp)
#endif /* RWREC_OPAQUE */


/***  SNMP Output Value  (Router outgoing/egress interface)  ***/

uint16_t rwrec_GetOutput(const rwRec *r);
void     rwrec_SetOutput(rwRec *r, uint16_t in_v);
void     rwrec_MemGetOutput(const rwRec *r, void *out_vp);
void     rwrec_MemSetOutput(rwRec *r, const void *in_vp);
int      rwrec_MemCmpOutput(const rwRec *r, const void *vp);

#define _recGetOutput(r)                        \
    ((r)->output)
#define _recSetOutput(r, in_v)                  \
    { ((r)->output) = (in_v); }
#define _recMemGetOutput(r, out_vp)                             \
    memcpy((out_vp), &((r)->output), RWREC_SIZEOF_OUTPUT)
#define _recMemSetOutput(r, in_vp)                              \
    memcpy(&((r)->output), (in_vp), RWREC_SIZEOF_OUTPUT)
#define _recMemCmpOutput(r, vp)                         \
    memcmp(&((r)->output), (vp), RWREC_SIZEOF_OUTPUT)

#if RWREC_OPAQUE
#  define rwRecGetOutput(r)  rwrec_GetOutput(r)
#  define rwRecSetOutput(r, in_v)  rwrec_SetOutput(r, in_v)
#  define rwRecMemGetOutput(r, out_vp)  rwrec_MemGetOutput(r, out_vp)
#  define rwRecMemSetOutput(r, in_vp)  rwrec_MemSetOutput(r, in_vp)
#  define rwRecMemCmpOutput(r, vp)  rwrec_MemCmpOutput(r, vp)
#else
#  define rwRecGetOutput(r)  _recGetOutput(r)
#  define rwRecSetOutput(r, in_v)  _recSetOutput(r, in_v)
#  define rwRecMemGetOutput(r, out_vp)  _recMemGetOutput(r, out_vp)
#  define rwRecMemSetOutput(r, in_vp)  _recMemSetOutput(r, in_vp)
#  define rwRecMemCmpOutput(r, vp)  _recMemCmpOutput(r, vp)
#endif /* RWREC_OPAQUE */


/***  TCP State  ***/

uint8_t  rwrec_GetTcpState(const rwRec *r);
void     rwrec_SetTcpState(rwRec *r, uint8_t in_v);
void     rwrec_MemGetTcpState(const rwRec *r, void *out_vp);
void     rwrec_MemSetTcpState(rwRec *r, const void *in_vp);
int      rwrec_MemCmpTcpState(const rwRec *r, const void *vp);

#define _recGetTcpState(r)                      \
    ((uint8_t)((r)->tcp_state & 0x7F))
#define _recSetTcpState(r, in_v)                \
    { ((r)->tcp_state) = ((r)->tcp_state & 0x80) | (0x7F & (in_v)); }
#define _recMemGetTcpState(r, out_vp)                   \
    { *((uint8_t*)(out_vp)) = _recGetTcpState(r); }
#define _recMemSetTcpState(r, in_vp)            \
    _recSetTcpState((r), *((uint8_t*)(in_vp)))
#define _recMemCmpTcpState(r, vp)                                       \
    ((int)(_recGetTcpState(r) - (uint8_t)(0x7f & *((uint8_t*)(vp)))))

#if RWREC_OPAQUE
#  define rwRecGetTcpState(r)  rwrec_GetTcpState(r)
#  define rwRecSetTcpState(r, in_v)  rwrec_SetTcpState(r, in_v)
#  define rwRecMemGetTcpState(r, out_vp)  rwrec_MemGetTcpState(r, out_vp)
#  define rwRecMemSetTcpState(r, in_vp)  rwrec_MemSetTcpState(r, in_vp)
#  define rwRecMemCmpTcpState(r, vp)  rwrec_MemCmpTcpState(r, vp)
#else
#  define rwRecGetTcpState(r)  _recGetTcpState(r)
#  define rwRecSetTcpState(r, in_v)  _recSetTcpState(r, in_v)
#  define rwRecMemGetTcpState(r, out_vp)  _recMemGetTcpState(r, out_vp)
#  define rwRecMemSetTcpState(r, in_vp)  _recMemSetTcpState(r, in_vp)
#  define rwRecMemCmpTcpState(r, vp)  _recMemCmpTcpState(r, vp)
#endif /* RWREC_OPAQUE */


/***  Application  ***/

uint16_t rwrec_GetApplication(const rwRec *r);
void     rwrec_SetApplication(rwRec *r, uint16_t in_v);
void     rwrec_MemGetApplication(const rwRec *r, void *out_vp);
void     rwrec_MemSetApplication(rwRec *r, const void *in_vp);
int      rwrec_MemCmpApplication(const rwRec *r, const void *vp);

#define _recGetApplication(r)                   \
    ((r)->application)
#define _recSetApplication(r, in_v)             \
    { ((r)->application) = (in_v); }
#define _recMemGetApplication(r, out_vp)                                \
    memcpy((out_vp), &((r)->application), RWREC_SIZEOF_APPLICATION)
#define _recMemSetApplication(r, in_vp)                                 \
    memcpy(&((r)->application), (in_vp), RWREC_SIZEOF_APPLICATION)
#define _recMemCmpApplication(r, vp)                            \
    memcmp(&((r)->application), (vp), RWREC_SIZEOF_APPLICATION)

#if RWREC_OPAQUE
#  define rwRecGetApplication(r)  rwrec_GetApplication(r)
#  define rwRecSetApplication(r, in_v)  rwrec_SetApplication(r, in_v)
#  define rwRecMemGetApplication(r, out_vp)  rwrec_MemGetApplication(r, out_vp)
#  define rwRecMemSetApplication(r, in_vp)  rwrec_MemSetApplication(r, in_vp)
#  define rwRecMemCmpApplication(r, vp)  rwrec_MemCmpApplication(r, vp)
#else
#  define rwRecGetApplication(r)  _recGetApplication(r)
#  define rwRecSetApplication(r, in_v)  _recSetApplication(r, in_v)
#  define rwRecMemGetApplication(r, out_vp)  _recMemGetApplication(r, out_vp)
#  define rwRecMemSetApplication(r, in_vp)  _recMemSetApplication(r, in_vp)
#  define rwRecMemCmpApplication(r, vp)  _recMemCmpApplication(r, vp)
#endif /* RWREC_OPAQUE */


/***  Memo  ***/

uint16_t rwrec_GetMemo(const rwRec *r);
void     rwrec_SetMemo(rwRec *r, uint16_t in_v);
void     rwrec_MemGetMemo(const rwRec *r, void *out_vp);
void     rwrec_MemSetMemo(rwRec *r, const void *in_vp);
int      rwrec_MemCmpMemo(const rwRec *r, const void *vp);

#define _recGetMemo(r)                          \
    ((r)->memo)
#define _recSetMemo(r, in_v)                    \
    { ((r)->memo) = (in_v); }
#define _recMemGetMemo(r, out_vp)                       \
    memcpy((out_vp), &((r)->memo), RWREC_SIZEOF_MEMO)
#define _recMemSetMemo(r, in_vp)                        \
    memcpy(&((r)->memo), (in_vp), RWREC_SIZEOF_MEMO)
#define _recMemCmpMemo(r, vp)                           \
    memcmp(&((r)->memo), (vp), RWREC_SIZEOF_MEMO)

#if RWREC_OPAQUE
#  define rwRecGetMemo(r)  rwrec_GetMemo(r)
#  define rwRecSetMemo(r, in_v)  rwrec_SetMemo(r, in_v)
#  define rwRecMemGetMemo(r, out_vp)  rwrec_MemGetMemo(r, out_vp)
#  define rwRecMemSetMemo(r, in_vp)  rwrec_MemSetMemo(r, in_vp)
#  define rwRecMemCmpMemo(r, vp)  rwrec_MemCmpMemo(r, vp)
#else
#  define rwRecGetMemo(r)  _recGetMemo(r)
#  define rwRecSetMemo(r, in_v)  _recSetMemo(r, in_v)
#  define rwRecMemGetMemo(r, out_vp)  _recMemGetMemo(r, out_vp)
#  define rwRecMemSetMemo(r, in_vp)  _recMemSetMemo(r, in_vp)
#  define rwRecMemCmpMemo(r, vp)  _recMemCmpMemo(r, vp)
#endif /* RWREC_OPAQUE */


/***  ICMP Type and Code is derived from the DPort  ***/

/*
 *    In NetFlow, Cisco has traditionally encoded the ICMP type and
 *    code in the DESTPORT field as (type << 8 | code); however, some
 *    routers encode it in the SRCPORT field as (code << 8 | type)
 *    (basically byteswapping the 32-bit "SRCPORT DESTPORT" value).
 *
 *    Our current packing code will modify these odd SRCPORT encodings
 *    to be in the more traditional DESTPORT encoding when the record
 *    is initially packed, though these macros will work for the
 *    window before our packing code was modified.  If the DESTPORT is
 *    non-zero, we assume it contains the ICMP values; otherwise, we
 *    look at the SRCPORT values.  Note that these functions/macros do
 *    not check the protocol.
 */
uint8_t  rwrec_GetIcmpType(const rwRec *r);
void     rwrec_SetIcmpType(rwRec *r, uint8_t in_v);
void     rwrec_MemGetIcmpType(const rwRec *r, void *out_vp);
uint8_t  rwrec_GetIcmpCode(const rwRec *r);
void     rwrec_SetIcmpCode(rwRec *r, uint8_t in_v);
void     rwrec_MemGetIcmpCode(const rwRec *r, void *out_vp);

#define  rwrec_GetIcmpTypeAndCode    rwrec_GetDPort
#define  rwrec_SetIcmpTypeAndCode    rwrec_SetDPort
#define  rwrec_MemGetIcmpTypeAndCode rwrec_MemGetDPort
#define  rwrec_MemSetIcmpTypeAndCode rwrec_MemSetDPort
#define  rwrec_MemCmpIcmpTypeAndCode rwrec_MemCmpDPort

#define _recGetIcmpType(r)                      \
    ((uint8_t)(0xFF & ((r)->dPort >> 8)))
#define _recSetIcmpType(r, in_v)                                        \
    { (r)->dPort = ((((in_v) & 0xFF) << 8) | ((r)->dPort & 0x00FF)); }
#define _recMemGetIcmpType(r, out_vp)           \
    { *out_vp = _recGetIcmpType(r); }

#define _recGetIcmpCode(r)                      \
    ((uint8_t)(0xFF & (r)->dPort))
#define _recSetIcmpCode(r, in_v)                                \
    { (r)->dPort = (((r)->dPort & 0xFF00) | ((in_v) & 0xFF)); }
#define _recMemGetIcmpCode(r, out_vp)           \
    { *out_vp = rwRecGetIcmpCode(r); }

#if RWREC_OPAQUE
#  define rwRecGetIcmpType(r)  rwrec_GetIcmpType(r)
#  define rwRecSetIcmpType(r, in_v)  rwrec_SetIcmpType(r, in_v)
#  define rwRecMemGetIcmpType(r, out_vp)  rwrec_MemGetIcmpType(r, out_vp)

#  define rwRecGetIcmpCode(r)  rwrec_GetIcmpCode(r)
#  define rwRecSetIcmpCode(r, in_v)  rwrec_SetIcmpCode(r, in_v)
#  define rwRecMemGetIcmpCode(r, out_vp)  rwrec_MemGetIcmpCode(r, out_vp)

#  define rwRecGetIcmpTypeAndCode(r)  rwrec_GetIcmpTypeAndCode(r)
#  define rwRecSetIcmpTypeAndCode(r, in_v)  rwrec_SetIcmpTypeAndCode(r, in_v)
#  define rwRecMemGetIcmpTypeAndCode(r, out_vp) \
    rwrec_MemGetIcmpTypeAndCode(r, out_vp)
#  define rwRecMemSetIcmpTypeAndCode(r, in_vp)  \
    rwrec_MemSetIcmpTypeAndCode(r, in_vp)
#  define rwRecMemCmpIcmpTypeAndCode(r, vp)     \
    rwrec_MemCmpIcmpTypeAndCode(r, vp)
#else /* RWREC_OPAQUE */
#  define rwRecGetIcmpType(r)  _recGetIcmpType(r)
#  define rwRecSetIcmpType(r, in_v)  _recSetIcmpType(r, in_v)
#  define rwRecMemGetIcmpType(r, out_vp)  _recMemGetIcmpType(r, out_vp)

#  define rwRecGetIcmpCode(r)  _recGetIcmpCode(r)
#  define rwRecSetIcmpCode(r, in_v)  _recSetIcmpCode(r, in_v)
#  define rwRecMemGetIcmpCode(r, out_vp)  _recMemGetIcmpCode(r, out_vp)

#  define rwRecGetIcmpTypeAndCode(r)  _recGetDPort(r)
#  define rwRecSetIcmpTypeAndCode(r, in_v)  _recSetDPort(r, in_v)
#  define rwRecMemGetIcmpTypeAndCode(r, out_vp)  _recMemGetDPort(r, out_vp)
#  define rwRecMemSetIcmpTypeAndCode(r, in_vp)  _recMemSetDPort(r, in_vp)
#  define rwRecMemCmpIcmpTypeAndCode(r, vp)  _recMemCmpDPort(r, vp)
#endif /* RWREC_OPAQUE */


#ifdef __cplusplus
}
#endif
#endif /* _RWREC_H */

/*
** Local Variables:
** mode:c
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 42 bytes --]


-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Happy New Year, Michael!

On Sun, Dec 09, 2012 at 10:35:42PM -0500, Michael Welsh Duggan wrote:
> Here's another case.  

> emacs -Q rwrec.h
> M-x c-toggle-parse-state-debug
> C-v {about 24-27 times}

> This is with current bzr.  
> 111170 rgm@gnu.org-20121210020042-arkhaf5eej6ujgtn

> rwrec.h is attached.

Thanks.  I think the following patch should have sorted out this one.  It
is based on the cc-engine.el in the emacs-24 branch in savannah.  Could
you try it out as usual, please.



=== modified file 'lisp/progmodes/cc-engine.el'
*** lisp/progmodes/cc-engine.el	2012-12-11 19:06:57 +0000
--- lisp/progmodes/cc-engine.el	2013-01-06 21:52:00 +0000
***************
*** 2464,2471 ****
--- 2464,2475 ----
  
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  ;; Variables which keep track of preprocessor constructs.
+ (defvar c-state-old-cpp-beg-marker)
+ (make-variable-buffer-local 'c-state-old-cpp-beg-marker)
  (defvar c-state-old-cpp-beg nil)
  (make-variable-buffer-local 'c-state-old-cpp-beg)
+ (defvar c-state-old-cpp-end-marker)
+ (make-variable-buffer-local 'c-state-old-cpp-end-marker)
  (defvar c-state-old-cpp-end nil)
  (make-variable-buffer-local 'c-state-old-cpp-end)
  ;; These are the limits of the macro containing point at the previous call of
***************
*** 2653,2665 ****
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
!       (let ((bra from) ce		; Positions of "{" and "}".
! 	    new-cons
! 	    (cache-pos (c-state-cache-top-lparen)) ; might be nil.
! 	    (macro-start-or-from
! 	     (progn (goto-char from)
! 		    (c-beginning-of-macro)
! 		    (point))))
  	(or upper-lim (setq upper-lim from))
  
  	;; If we're essentially repeating a fruitless search, just give up.
--- 2657,2678 ----
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
! 
!       (let* (new-cons
! 	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
! 	     (macro-start-or-from
! 	      (progn (goto-char from)
! 		     (c-beginning-of-macro)
! 		     (point)))
! 	     (bra			; Position of "{".
! 	      ;; Don't start scanning in the middle of a CPP construct unless
! 	      ;; it contains HERE - these constructs, in Emacs, are "commented
! 	      ;; out" with category properties.
! 	      (if (eq (c-get-char-property macro-start-or-from 'category)
! 			'c-cpp-delimiter)
! 		    macro-start-or-from
! 		  from))
! 	     ce)			; Position of "}"
  	(or upper-lim (setq upper-lim from))
  
  	;; If we're essentially repeating a fruitless search, just give up.
***************
*** 2899,2905 ****
--- 2912,2920 ----
  		  (point-max)
  		(min (point-max) c-state-old-cpp-beg)))
  	(while (and c-state-cache (>= (c-state-cache-top-lparen) upper-lim))
+ 	  (setq scan-back-pos (car-safe (car c-state-cache)))
  	  (setq c-state-cache (cdr c-state-cache)))
+ 
  	;; If `upper-lim' is inside the last recorded brace pair, remove its
  	;; RBrace and indicate we'll need to search backwards for a previous
  	;; brace pair.
***************
*** 3324,3329 ****
--- 3339,3351 ----
       (c-with-cpps-commented-out
        (c-invalidate-state-cache-1 here)))))
  
+ (defmacro c-state-maybe-marker (place marker)
+   ;; If PLACE is non-nil, return a marker marking it, otherwise nil.
+   ;; We (re)use MARKER.
+   `(and ,place
+ 	(or ,marker (setq ,marker (make-marker)))
+ 	(set-marker ,marker ,place)))
+ 
  (defun c-parse-state ()
    ;; This is a wrapper over `c-parse-state-1'.  See that function for a
    ;; description of the functionality and return value.
***************
*** 3350,3358 ****
  	      (c-parse-state-1))
  	   (c-with-cpps-commented-out
  	    (c-parse-state-1))))
!       (setq c-state-old-cpp-beg (and here-cpp-beg (copy-marker here-cpp-beg t))
! 	    c-state-old-cpp-end (and here-cpp-end (copy-marker here-cpp-end t)))
!       )))
  
  ;; Debug tool to catch cache inconsistencies.  This is called from
  ;; 000tests.el.
--- 3372,3381 ----
  	      (c-parse-state-1))
  	   (c-with-cpps-commented-out
  	    (c-parse-state-1))))
!       (setq c-state-old-cpp-beg
! 	    (c-state-maybe-marker here-cpp-beg c-state-old-cpp-beg-marker)
! 	    c-state-old-cpp-end
! 	    (c-state-maybe-marker here-cpp-end c-state-old-cpp-end-marker)))))
  
  ;; Debug tool to catch cache inconsistencies.  This is called from
  ;; 000tests.el.



> -- 
> Michael Welsh Duggan
> (md5i@md5i.com)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Happy New Year, Michael!
>
> On Sun, Dec 09, 2012 at 10:35:42PM -0500, Michael Welsh Duggan wrote:
>> Here's another case.  
>
>> emacs -Q rwrec.h
>> M-x c-toggle-parse-state-debug
>> C-v {about 24-27 times}
>
>> This is with current bzr.  
>> 111170 rgm@gnu.org-20121210020042-arkhaf5eej6ujgtn
>
>> rwrec.h is attached.
>
> Thanks.  I think the following patch should have sorted out this one.  It
> is based on the cc-engine.el in the emacs-24 branch in savannah.  Could
> you try it out as usual, please.

This fixes this case, indeed.  Unfortunately, here's another.

emacs -Q rwtransfer.c
  M-x c-toggle-parse-state-debug
  M->
Then type M-v until the parse failure happens.  For me it happens at 
32% point in the file.  Please not that this particular problem only
happens when paging through the file in reverse order.

c-parse-state inconsistency at 15885: using cache: (15636 15181 (13849 . 15141)), from scratch: (15636 (15303 . 15493) 15181 (13849 . 15141))
Old state:
(setq c-state-cache '(16334 16271 16248 (16059 . 16193) 15181 (13849 . 15141))  c-state-cache-good-pos 16335  c-state-nonlit-pos-cache '(48473 45473 42473 39473 36473 33473 30473 27378 24378 21260 18015 15015 12015 9015 6015 3015)  c-state-nonlit-pos-cache-limit 48473  c-state-semi-nonlit-pos-cache '(48323 45323 42323 39323 36323 33260 30260 27260 24260 21260 18001 15001 12001 9001 6001 3001)  c-state-semi-nonlit-pos-cache-limit 48323  c-state-brace-pair-desert '(15181 . 17388)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil  c-parse-state-point 16335)
c-parse-state inconsistency at 15885: using cache: (15636 15181 (13849 . 15141)), from scratch: (15636 (15303 . 15493) 15181 (13849 . 15141))

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 48 bytes --]

And here's the file I was supposed to include:


[-- Attachment #2: rwtransfer.c --]
[-- Type: text/plain, Size: 50818 bytes --]

/*
** Copyright (C) 2006-2013 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

/*
**  rwtransfer.c
**
**    This file contains functions that are common to rwsender and
**    rwreceiver, such as options processing and establishing the
**    connection.
**
*/


#include <silk/silk.h>

RCSIDENT("$Id$");

#include <silk/utils.h>
#include <silk/sklog.h>
#include <silk/skdaemon.h>
#include "rwtransfer.h"


/* LOCAL DEFINES AND TYPEDEFS */

#if SK_ENABLE_GNUTLS
#  define UNUSED_NO_GNUTLS(x) x
#else
#  define UNUSED_NO_GNUTLS(x) UNUSED(x)
#endif

/* Illegal ident characters */
#define ILLEGAL_IDENT_CHARS " \t:/\\.,"

/* Define lowest protocol version which we handle */
#define LOW_VERSION  1

/* Version protocol we emit */
#define EMIT_VERISION 2

/* Turn on PKCS12 support */
#define PKCS12 1

/* Environment variable used to turn off keepalive.  Used for
 * debugging. */
#define RWTRANSFER_TURN_OFF_KEEPALIVE "RWTRANSFER_TURN_OFF_KEEPALIVE"

/* Maximum expected size of connection information string*/
#define RWTRANSFER_CONNECTION_TYPE_SIZE_MAX 50

typedef struct connection_msg_data_st {
    const char *name;
    int32_t     size;
} connection_msg_data_t;


/* EXPORTED VARIABLE DEFINITIONS */

int main_retval = EXIT_SUCCESS;


/* LOCAL VARIABLE DEFINITIONS */

/* Mode (client/server) */
static enum {CLIENT, SERVER, NOT_SET} mode;

/* Marks last client/server-specific option*/
static int client_sentinel;
static int server_sentinel;

/* Daemon identity */
static char *identity;

#if SK_ENABLE_GNUTLS
/* Encryption and authentication files */
static char *tls_ca_file     = NULL;
static char *tls_cert_file   = NULL;
static char *tls_key_file    = NULL;
#ifdef PKCS12
static char *tls_pkcs12_file = NULL;
#endif
#endif /* SK_ENABLE_GNUTLS */

/* Message queue */
static sk_msg_queue_t *control;

/* Temporary transfer_t item */
static transfer_t *global_temp_item;

/* Control message thread */
static pthread_t control_thread;
static int       control_thread_valid;

/* Address upon which to listen for incoming connections */
static sk_sockaddr_array_t *listen_address = NULL;
static const char *listen_address_arg = NULL;

/* Locations which can be addressed as return values */
static void *exit_standard   = &exit_standard;
static void *exit_disconnect = &exit_disconnect;
static void *exit_failure    = &exit_failure;

/* Main thread */
static pthread_t main_thread;

/* Detached thread entry/exit control (see comment in serverMain()) */
static uint16_t detached_thread_count = 0;
static pthread_mutex_t detached_thread_mutex = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t  detached_thread_cond  = PTHREAD_COND_INITIALIZER;

typedef int (*connection_fn_t)(
    sk_msg_queue_t *,
    struct sockaddr *,
    socklen_t,
    skm_channel_t *);

typedef struct conn_info_st {
    sk_msg_queue_t *queue;
    skm_channel_t   channel;
    transfer_t     *trnsfr;
    unsigned        tls;
} conn_info_t;

typedef enum {
    /* Global options */
    OPT_MODE, OPT_IDENT
#if SK_ENABLE_GNUTLS
    , OPT_TLS_CA,
    OPT_TLS_CERT,
    OPT_TLS_KEY
#ifdef PKCS12
    , OPT_TLS_PKCS12
#endif
#endif /* SK_ENABLE_GNUTLS */
} appOptionsEnum;

typedef enum {
    /* Client options */
    OPT_SERVER_ADDR,

    /* Unused */
    OPT_CLIENT_UNUSED
} appClientOptionsEnum;

typedef enum {
    /* Server options */
    OPT_SERVER_PORT, OPT_CLIENT_IDENT,

    /* Unused */
    OPT_SERVER_UNUSED
} appServerOptionsEnum;


static struct option appOptions[] = {
    {"mode",                 REQUIRED_ARG, 0, OPT_MODE},
    {"identifier",           REQUIRED_ARG, 0, OPT_IDENT},
#if SK_ENABLE_GNUTLS
    {"tls-ca",               REQUIRED_ARG, 0, OPT_TLS_CA},
    {"tls-cert",             REQUIRED_ARG, 0, OPT_TLS_CERT},
    {"tls-key",              REQUIRED_ARG, 0, OPT_TLS_KEY},
#ifdef PKCS12
    {"tls-pkcs12",           REQUIRED_ARG, 0, OPT_TLS_PKCS12},
#endif
#endif /* SK_ENABLE_GNUTLS */
    {0,0,0,0}           /* sentinel entry */
};

static const char *appHelp[] = {
    ("Whether to run as a client or a server ('client', 'server')"),
    ("Named identifier for the daemon"),
#if SK_ENABLE_GNUTLS
    ("Full path to CA cert PEM file. Def. None. Either --tls-key\n"
     "\tand --tls-key or --tls-pkcs12 must also be specified."),
    ("Full path to encryption cert PEM file. Def. None.  Requires\n"
     "\tthat --tls-ca and --tls-key also be specified."),
    ("Full path to encryption key PEM file. Def. None.  Requires\n"
     "\tthat --tls-ca and --tls-cert also be specified."),
#ifdef PKCS12
    ("Full path to encryption cert and key PKCS#12 file.\n"
     "\tDef. None.  Requires that --tls-ca also be specified."),
#endif
#endif /* SK_ENABLE_GNUTLS */
    (char *)NULL
};

static struct option appClientOptions[] = {
    {"server-address", REQUIRED_ARG, 0, OPT_SERVER_ADDR},
    {0,0,0,0}           /* sentinel entry */
};

static const char *appClientHelp[] = {
    ("A server to which to connect.  Repeatable.\n"
     "\t<ident>:<address>:<port>"),
    (char *)NULL
};

static struct option appServerOptions[] = {
    {"server-port",  REQUIRED_ARG, 0, OPT_SERVER_PORT},
    {"client-ident", REQUIRED_ARG, 0, OPT_CLIENT_IDENT},
    {0,0,0,0}           /* sentinel entry */
};

static const char *appServerHelp[] = {
    ("[HOST:]PORT upon which to listen for incoming client\n"
     "\tconnections"),
    ("The identifier of a client allowed to connect to\n"
     "\tthis server.  Repeatable."),
    (char *)NULL
};


/*
 *  Connection message textual representation and lengths.
 *
 *  Length of -1 indicates a variable length message (use of
 *  sendString() implies variable length).
 */
static connection_msg_data_t conn_msg_data[CONN_NUMBER_OF_CONNECTION_MESSAGES]=
    {
        {"CONN_SENDER_VERSION",    sizeof(uint32_t)},
        {"CONN_RECEIVER_VERSION",  sizeof(uint32_t)},
        {"CONN_IDENT",            -1},
        {"CONN_READY",             0},
        {"CONN_DISCONNECT_RETRY", -1},
        {"CONN_DISCONNECT",       -1},
        {"CONN_NEW_FILE",         -1},
        {"CONN_NEW_FILE_READY",    0},
        {"CONN_FILE_BLOCK",       -1},
        {"CONN_FILE_COMPLETE",     0},
        {"CONN_DUPLICATE_FILE",   -1},
        {"CONN_REJECT_FILE",      -1}
    };


/* LOCAL FUNCTION PROTOTYPES */

static void *clientMain(void *); /* Thread entry point */
static void *serverMain(void *); /* Thread entry point */
static int appOptionsHandler(
    clientData  cData,
    int         opt_index,
    char       *opt_arg);
static int appClientOptionsHandler(
    clientData  cData,
    int         opt_index,
    char       *opt_arg);
static int appServerOptionsHandler(
    clientData  cData,
    int         opt_index,
    char       *opt_arg);
static void parseServerAddress(
    const char *const_addr);
static void addIdent(
    const char *ident);


/* FUNCTION DEFINITIONS */

/* checkIdent();
 *
 *    Check to see if an ident is legal.  If illegal, print an error
 *    message to the error stream, end exit.
 */
int checkIdent(
    const char     *ident)
{
    const char *invalid;
    const char *c;

    if (ident == NULL || ident[0] == '\0') {
        skAppPrintErr("Identifiers must contain at least one character");
        exit(EXIT_FAILURE);
    }
    invalid = strpbrk(ident, ILLEGAL_IDENT_CHARS);
    if (invalid != NULL) {
        skAppPrintErr(("The identifier %s contains the illegal "
                       "character %c"), ident, *invalid);
        exit(EXIT_FAILURE);
    }
    for (c = ident; *c; c++) {
        if (!isprint((int)*c)) {
            skAppPrintErr(("The identifier %s contains the illegal "
                           "nonprintable character 0x%x"), ident, (int)*c);
            exit(EXIT_FAILURE);
        }
    }

    return 0;
}


/*
 * appModeUsage();
 *
 *    Print usage information for the mode named 'mode_str', using the
 *    given 'options' and 'help'.
 */
static void appModeUsage(
    FILE           *fh,
    const char     *mode_str,
    struct option   options[],
    const char     *help[])
{
    unsigned int i;

    fprintf(fh, "\n%s switches:\n", mode_str);
    for (i = 0; options[i].name; i++) {
        if (help[i]) {
            fprintf(fh, "--%s %s. %s\n", options[i].name,
                    SK_OPTION_HAS_ARG(options[i]), help[i]);
        }
    }
}


void transferUsageLong(
    FILE           *fh,
    const char     *usage,
    struct option   options[],
    const char     *help[])
{
    unsigned int i;
    fprintf(fh, "%s %s", skAppName(), usage);

    fprintf(fh, "\nCommon switches:\n");
    skOptionsDefaultUsage(fh);
    for (i = 0; appOptions[i].name; ++i) {
        fprintf(fh, "--%s %s. %s\n", appOptions[i].name,
                SK_OPTION_HAS_ARG(appOptions[i]), appHelp[i]);
    }
    for (i = 0; options[i].name; ++i) {
        fprintf(fh, "--%s %s. %s\n", options[i].name,
                SK_OPTION_HAS_ARG(options[i]), help[i]);
    }
    appModeUsage(fh, "Client", appClientOptions, appClientHelp);
    appModeUsage(fh, "Server", appServerOptions, appServerHelp);
    fprintf(fh, "\nLogging and daemon switches:\n");
    skdaemonOptionsUsage(fh);
}


int transferSetup(
    void)
{
    /* verify that the sizes of options and help match */
    assert((sizeof(appHelp)/sizeof(char*)) ==
           (sizeof(appOptions)/sizeof(struct option)));
    assert((sizeof(appClientHelp)/sizeof(char*)) ==
           (sizeof(appClientOptions)/sizeof(struct option)));
    assert((sizeof(appServerHelp)/sizeof(char*)) ==
           (sizeof(appServerOptions)/sizeof(struct option)));

    mode                  = NOT_SET;
    client_sentinel       = OPT_CLIENT_UNUSED;
    server_sentinel       = OPT_SERVER_UNUSED;
    identity              = NULL;
    global_temp_item      = NULL;
    control_thread_valid  = 0;

    /* register the options and handler */
    if (skOptionsRegister(appOptions, &appOptionsHandler, NULL))
    {
        skAppPrintErr("Unable to transfer application options");
        return -1;
    }

    /* register the client options and handler */
    if (skOptionsRegister(appClientOptions, &appClientOptionsHandler, NULL))
    {
        skAppPrintErr("Unable to register client options");
        return -1;
    }

    /* register the server options and handler */
    if (skOptionsRegister(appServerOptions, &appServerOptionsHandler, NULL))
    {
        skAppPrintErr("Unable to register server options");
        return -1;
    }

    return 0;
}


/*
 *  status = appOptionsHandler(cData, opt_index, opt_arg);
 *
 *    This function is passed to skOptionsRegister(); it will be called
 *    by skOptionsParse() for each user-specified switch that the
 *    application has registered; it should handle the switch as
 *    required---typically by setting global variables---and return 1
 *    if the switch processing failed or 0 if it succeeded.  Returning
 *    a non-zero from from the handler causes skOptionsParse() to return
 *    a negative value.
 *
 *    The clientData in 'cData' is typically ignored; 'opt_index' is
 *    the index number that was specified as the last value for each
 *    struct option in appOptions[]; 'opt_arg' is the user's argument
 *    to the switch for options that have a REQUIRED_ARG or an
 *    OPTIONAL_ARG.
 */
static int appOptionsHandler(
    clientData  UNUSED(cData),
    int         opt_index,
    char       *opt_arg)
{

    switch ((appOptionsEnum)opt_index) {

      case OPT_MODE:
        if (0 == strcmp(opt_arg, "server")) {
            mode = SERVER;
        } else if (0 == strcmp(opt_arg, "client")) {
            mode = CLIENT;
        } else {
            skAppPrintErr("Invalid --%s '%s'",
                          appOptions[opt_index].name, opt_arg);
            return 1;
        }
        break;

      case OPT_IDENT:
        checkIdent(opt_arg);
        identity = opt_arg;
        break;

#if SK_ENABLE_GNUTLS
      case OPT_TLS_CA:
        if (optionsFileCheck(appOptions[opt_index].name, opt_arg)) {
            return 1;
        }
        tls_ca_file = opt_arg;
        break;

      case OPT_TLS_CERT:
        if (optionsFileCheck(appOptions[opt_index].name, opt_arg)) {
            return 1;
        }
        tls_cert_file = opt_arg;
        break;

      case OPT_TLS_KEY:
        if (optionsFileCheck(appOptions[opt_index].name, opt_arg)) {
            return 1;
        }
        tls_key_file = opt_arg;
        break;

#ifdef PKCS12
      case OPT_TLS_PKCS12:
        if (optionsFileCheck(appOptions[opt_index].name, opt_arg)) {
            return 1;
        }
        tls_pkcs12_file = opt_arg;
        break;
#endif
#endif /* SK_ENABLE_GNUTLS */

    }

    return 0;  /* OK */
}


int transferVerifyOptions(
    void)
{
    RBLIST *list;
    transfer_t *item;
    int error_count = 0;

    /* Check mode options */
    if (mode == NOT_SET) {
        skAppPrintErr(("Client or server mode must be chosen "
                       "via the --%s switch"),
                      appOptions[OPT_MODE].name);
        ++error_count;
    }
    if ((mode == CLIENT && (server_sentinel != OPT_SERVER_UNUSED)) ||
        (mode == SERVER && (client_sentinel != OPT_CLIENT_UNUSED)))
    {
        int badopt = (mode == CLIENT) ? server_sentinel : client_sentinel;
        struct option *opts =
            (mode == CLIENT) ? appServerOptions : appClientOptions;
        const char *mode_string = (mode == CLIENT) ? "client" : "server";
        skAppPrintErr("The --%s switch cannot be used in %s mode",
                      opts[badopt].name, mode_string);
        return -1;
    }

    if (identity == NULL) {
        skAppPrintErr("The --%s switch is required",
                      appOptions[OPT_IDENT].name);
        ++error_count;
    }

    if (rbmin(transfers) == NULL && mode != NOT_SET) {
        skAppPrintErr("Must supply at least one --%s switch",
                      (mode == CLIENT) ?
                      appClientOptions[OPT_SERVER_ADDR].name :
                      appServerOptions[OPT_CLIENT_IDENT].name);
        ++error_count;
    }

#if SK_ENABLE_GNUTLS
#ifdef PKCS12
    if (tls_ca_file || tls_cert_file || tls_key_file || tls_pkcs12_file) {
        if (!tls_ca_file) {
            skAppPrintErr("A CA cert file must be specified for "
                          "encryption: --%s", appOptions[OPT_TLS_CA].name);
            ++error_count;
        }
        if (!(tls_cert_file && tls_key_file && !tls_pkcs12_file) &&
            !(!tls_cert_file && !tls_key_file && tls_pkcs12_file))
        {
            skAppPrintErr("When using encryption, you must specify --%s and "
                          "--%s, or just --%s",
                          appOptions[OPT_TLS_CERT].name,
                          appOptions[OPT_TLS_KEY].name,
                          appOptions[OPT_TLS_PKCS12].name);
            ++error_count;
        }
    }
#else /* !PKCS12 */
    if (tls_ca_file || tls_cert_file || tls_key_file) {
        if (!tls_ca_file) {
            skAppPrintErr("A CA cert file must be specified for "
                          "encryption: --%s", appOptions[OPT_TLS_CA]);
            ++error_count;
        }
        if (!(tls_cert_file && tls_key_file))
        {
            skAppPrintErr("When using encryption, you must specify --%s and "
                          "--%s",
                          appOptions[OPT_TLS_CERT].name,
                          appOptions[OPT_TLS_KEY].name);
            ++error_count;
        }
    }
#endif /* PKCS12 */
#endif /* SK_ENABLE_GNUTLS */

    switch (mode) {
      case SERVER:
        if (listen_address == NULL) {
            skAppPrintErr("Must supply a port using --%s in server mode",
                          appServerOptions[OPT_SERVER_PORT].name);
            ++error_count;
        }
        break;

      case CLIENT:
        list = rbopenlist(transfers);
        if (list == NULL) {
            skAppPrintErr("Memory allocation failure verifying options");
            return -1;
        }
        while ((item = (transfer_t *)rbreadlist(list)) != NULL) {
            if (item->address_exists == 0) {
                skAppPrintErr("Ident %s has no address associated with it",
                              item->ident);
                return -1;
            }
        }
        rbcloselist(list);
        break;

      case NOT_SET:
        break;
    }

    if (error_count) {
        return -1;
    }

    main_thread = pthread_self();

    return 0;
}


void transferShutdown(
    void)
{
    RBLIST *iter;
    transfer_t *trnsfr;
    int rv;

    assert(shuttingdown);

    skMsgQueueShutdownAll(control);

    iter = rbopenlist(transfers);
    CHECK_ALLOC(iter);
    while ((trnsfr = (transfer_t *)rbreadlist(iter)) != NULL) {
        rv = transferUnblock(trnsfr);
        if (rv != 0) {
            CRITMSG("Unexpected failure to unblock transfer");
            _exit(EXIT_FAILURE);
        }
    }
    rbcloselist(iter);
}


void transferTeardown(
    void)
{
    /* Wait for transfer threads to end.  In server mode, all these
     * threads are detached, and as such may not be joinable.  */
    if (mode != SERVER) {
        RBLIST *iter;
        transfer_t *trnsfr;

        iter = rbopenlist(transfers);
        CHECK_ALLOC(iter);
        while ((trnsfr = (transfer_t *)rbreadlist(iter)) != NULL) {
            if (trnsfr->thread_exists) {
                DEBUGMSG("Waiting for thread %s to end...", trnsfr->ident);
                pthread_join(trnsfr->thread, NULL);
                DEBUGMSG("Thread %s has ended.", trnsfr->ident);
            }
        }
        rbcloselist(iter);
    }

    /* Wait for control thread to end */
    if (control_thread_valid) {
        DEBUGMSG("Waiting for control thread to end...");
        pthread_join(control_thread, NULL);
        DEBUGMSG("Control thread has ended.");
    }

    /* Wait for detached threads to end */
    DEBUGMSG("Waiting for detached threads to end...");
    pthread_mutex_lock(&detached_thread_mutex);
    while (detached_thread_count) {
        pthread_cond_wait(&detached_thread_cond, &detached_thread_mutex);
    }
    pthread_mutex_unlock(&detached_thread_mutex);
    DEBUGMSG("Detached threads have ended.");

    /* Destroy stuff */
    skMsgQueueDestroy(control);
    if (listen_address) {
        skSockaddrArrayDestroy(listen_address);
        listen_address = NULL;
    }
    if (global_temp_item != NULL) {
        free(global_temp_item);
    }

#if SK_ENABLE_GNUTLS
    skMsgGnuTLSTeardown();
#endif
}


/*
 *  file_exists = optionsFileCheck(opt_name, opt_arg);
 *
 *    Verify that the file in 'opt_arg' exists and that we have a full
 *    path to the file.  Verify that the length is shorter than
 *    PATH_MAX.  If so, return 0; otherwise, print an error that the
 *    option named by 'opt_name' was bad and return -1.
 */
int optionsFileCheck(
    const char *opt_name,
    const char *opt_arg)
{
    if (!opt_arg || !opt_arg[0]) {
        skAppPrintErr("Invalid %s: The argument empty", opt_name);
        return -1;
    }

    if (strlen(opt_arg)+1 >= PATH_MAX) {
        skAppPrintErr("Invalid %s: Path is too long", opt_name);
        return -1;
    }

    if (!skFileExists(opt_arg)) {
        skAppPrintErr("Invalid %s: File '%s' does not exist",
                      opt_name, opt_arg);
        return -1;
    }

    if (opt_arg[0] != '/') {
        skAppPrintErr(("Invalid %s: Must use complete path"
                       " ('%s' does not begin with slash)"),
                      opt_name, opt_arg);
        return -1;
    }

    return 0;
}


static int appClientOptionsHandler(
    clientData  UNUSED(cData),
    int         opt_index,
    char       *opt_arg)
{
    client_sentinel = opt_index;

    switch ((appClientOptionsEnum)opt_index) {
      case OPT_SERVER_ADDR:
        parseServerAddress(opt_arg);
        break;

      default:
        /* Should not get here */
        ASSERT_ABORT(0);
    }

    return 0;  /* OK */
}


static int appServerOptionsHandler(
    clientData  UNUSED(cData),
    int         opt_index,
    char       *opt_arg)
{
    int rv;

    server_sentinel = opt_index;

    switch ((appServerOptionsEnum)opt_index) {
      case OPT_CLIENT_IDENT:
        addIdent(opt_arg);
        break;

      case OPT_SERVER_PORT:
        rv = skStringParseHostPortPair(&listen_address,
                                       opt_arg, PORT_REQUIRED);
        if (rv) {
            skAppPrintErr("Invalid %s '%s': %s",
                          appOptions[opt_index].name, opt_arg,
                          skStringParseStrerror(rv));
            return 1;
        }
        listen_address_arg = opt_arg;
        break;

      default:
        /* Should not get here */
        ASSERT_ABORT(0);
    }

    return 0;  /* OK */
}



/* String compare for receiver rbtree */
static int transferCompare(
    const void *va,
    const void *vb,
    const void *UNUSED(cbdata))
{
    const transfer_t *a = (const transfer_t *)va;
    const transfer_t *b = (const transfer_t *)vb;
    return strcmp(a->ident, b->ident);
}


struct rbtree *transferIdentTreeCreate(
    void)
{
    return rbinit(transferCompare, NULL);
}


/* Create temporary transfer_t objects */
transfer_t *initTemp(
    void)
{
    /* Allocate and/or clear the temporary item */
    if (global_temp_item == NULL) {
        global_temp_item = (transfer_t *)calloc(1, sizeof(*global_temp_item));
    } else {
        memset(global_temp_item, 0, sizeof(*global_temp_item));
    }
    return global_temp_item;
}

/* If a program wishes to keep a temporary transfer object, it should
   call this. */
void clearTemp(
    void)
{
    global_temp_item = NULL;
}


/* Parse a <ident>:<address>:<port> specification */
static void parseServerAddress(
    const char     *const_addr)
{
#define FMT_PARSE_FAILURE                               \
    ("Server address parse failure parsing '%s'\n"      \
     "\tCorrect form is <ident>:<address>:<port>")
#define FMT_MEM_FAILURE                                         \
    "Memory allocation failure when parsing server address %s"

    char *addr = strdup(const_addr);
    char *colon, *next;
    transfer_t *old;
    const void *test;
    int rv;
    transfer_t *temp_item;

    temp_item = initTemp();

    if (addr == NULL || temp_item == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, const_addr);
        exit(EXIT_FAILURE);
    }

    /* First, extract the ident */
    colon = strchr(addr, ':');
    if (colon == NULL) {
        free(addr);
        skAppPrintErr(FMT_PARSE_FAILURE, const_addr);
        exit(EXIT_FAILURE);
    }
    *colon = '\0';
    checkIdent(addr);
    temp_item->ident = addr;


    /* See if it has already been used */
    old = (transfer_t *)rbfind(temp_item, transfers);
    if (old != NULL) {
        if (!old->address_exists) {
            memcpy(temp_item, old, sizeof(*temp_item));
            test = rbdelete(old, transfers);
            assert(test == old);
            temp_item->ident = addr;
            free(old->ident);
            free(old);
        } else {
            free(addr);
            skAppPrintErr("Duplicate ident in server address %s", const_addr);
            exit(EXIT_FAILURE);
        }
    }

    /* Next, extract the address */
    next = colon + 1;
    rv = skStringParseHostPortPair(&temp_item->addr, next,
                                   HOST_REQUIRED | PORT_REQUIRED);
    if (rv < 0) {
        skAppPrintErr("Could not parse address: %s",
                      skStringParseStrerror(rv));
        exit(EXIT_FAILURE);
    }

    /* Add to our server list */
    temp_item->ident = strdup(temp_item->ident);
    free(addr);
    if (temp_item->ident == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, const_addr);
        exit(EXIT_FAILURE);
    }
    test = rbsearch(temp_item, transfers);
    if (test == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, const_addr);
        exit(EXIT_FAILURE);
    }
    temp_item->address_exists = 1;

    assert(test == temp_item);
    clearTemp();

#undef FMT_PARSE_FAILURE
#undef FMT_MEM_FAILURE
}


/* Add a bare ident to the transfer list */
static void addIdent(
    const char     *ident)
{
#define FMT_MEM_FAILURE "Memory allocation failure when parsing ident %s"
    const void *test;
    transfer_t *temp_item;

    checkIdent(ident);
    temp_item = initTemp();
    if (temp_item == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, ident);
        exit(EXIT_FAILURE);
    }
    temp_item->ident = (char *)ident;
    test = rbsearch(temp_item, transfers);
    if (test == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, ident);
        exit(EXIT_FAILURE);
    }
    if (test != temp_item) {
        skAppPrintErr("Duplicate ident %s", ident);
        exit(EXIT_FAILURE);
    }
    temp_item->ident = strdup(ident);
    if (temp_item->ident == NULL) {
        skAppPrintErr(FMT_MEM_FAILURE, ident);
        exit(EXIT_FAILURE);
    }
    clearTemp();
#undef FMT_MEM_FAILURE
}


static void getConnectionInformation(
    sk_msg_queue_t *queue,
    skm_channel_t   channel,
    char           *buffer,
    size_t          buffer_size)
{
    int rv;

    rv = skMsgGetConnectionInformation(queue, channel, buffer, buffer_size);
    if (rv == -1) {
        strncpy(buffer, "<unknown>", buffer_size);
    }
    buffer[buffer_size - 1] = '\0';
}


int handleDisconnect(
    sk_msg_t       *msg,
    const char     *type)
{
    skm_type_t msgtyp = skMsgType(msg);

    if (msgtyp == CONN_DISCONNECT || msgtyp == CONN_DISCONNECT_RETRY) {
        int length = MAX_ERROR_MESSAGE;

        if (skMsgLength(msg) < length) {
            length = skMsgLength(msg);
        }

        INFOMSG("Connection %s disconnected: %.*s",
                type, length, (char *)skMsgMessage(msg));

        return (msgtyp == CONN_DISCONNECT) ? -1 : 1;
    }

    return 0;
}


/*
 *    This function is used by servers and clients.  The function
 *    verifies the connection (version, ident), and then calls the
 *    transferFiles() function defined in rwsender.c or rwreceiver.c.
 *
 *    For a server, this is a THREAD ENTRY POINT.  Entry point for the
 *    "connection" thread, started from serverMain().  This thread is
 *    detached.
 *
 *    For a client, this function is called by startClientConnection()
 *    once the client has connected to a server.
 */
static void *handleConnection(
    void           *vinfo)
{
    conn_info_t *info = (conn_info_t *)vinfo;
    transfer_t target;
    transfer_t *trnsfr = NULL;
    transfer_t *found = NULL;
    uint32_t version;
    skm_channel_t channel;
    sk_msg_queue_t *q;
    enum conn_state {Version, Ident, Ready, Running, Disconnect} state;
    int proto_err;
    int fatal_err = 0;
    const char *ident = "<unassigned>";
    void *retval = exit_failure;
    char connection_type[RWTRANSFER_CONNECTION_TYPE_SIZE_MAX];
    int transferred_file = 0;

    DEBUG_PRINT1("connection thread started");

    q = info->queue;
    channel = info->channel;
    trnsfr = info->trnsfr;
    free(info);

    /* start by sending my version and waiting for remote's version */
    state = Version;
    version = htonl(EMIT_VERISION);
    proto_err = skMsgQueueSendMessage(q, channel, local_version_check,
                                      &version, sizeof(version));

    while (!shuttingdown && !proto_err && !fatal_err && state != Running) {
        int rv;
        sk_msg_t *msg;

        rv = skMsgQueueGetMessage(q, &msg);
        if (rv == -1) {
            ASSERT_ABORT(shuttingdown);
            continue;
        }
        DEBUG_PRINT3("handleConnection() state=%d, got msg type=%d",
                     (int)state, (int)skMsgType(msg));

        rv = handleDisconnect(msg, ident);
        if (rv) {
            proto_err = 1;
            retval = transferred_file ? exit_disconnect : exit_failure;
            state = Disconnect;
        }

        switch (state) {
          case Version:
            /* expecting remote's version. if not valid, close the
             * channel.  if valid, send my ident and wait for remote's
             * ident */
            if ((proto_err = checkMsg(msg, q, remote_version_check))) {
                DEBUG_PRINT2("checkMsg(%s) FAILED",
                             conn_msg_data[remote_version_check].name);
                retval = exit_failure;
                break;
            }
            DEBUG_PRINT2("Received %s",
                         conn_msg_data[remote_version_check].name);
            version = MSG_UINT32(msg);
            if (version < LOW_VERSION) {
                sendString(q, skMsgChannel(msg), EXTERNAL,
                           CONN_DISCONNECT, LOG_WARNING,
                           ("Unsupported version %" PRIu32), version);
                proto_err = 1;
                retval = exit_failure;
                break;
            }
            if (!getenv(RWTRANSFER_TURN_OFF_KEEPALIVE)) {
                rv = skMsgSetKeepalive(q, channel, KEEPALIVE_TIMEOUT);
                assert(rv == 0);
            }
            state = Ident;
            proto_err = skMsgQueueSendMessage(q, channel, CONN_IDENT,
                                              identity, strlen(identity) + 1);
            if (proto_err != 0) {
                retval = exit_failure;
            }
            break;

          case Ident:
            /* expecting remote's ident.  if not valid, close the
             * channel.  if valid, send CONN_READY and wait for remote
             * to say it is ready */
            if ((proto_err = checkMsg(msg, q, CONN_IDENT))) {
                DEBUG_PRINT1("checkMsg(CONN_IDENT) FAILED");
                retval = exit_failure;
                break;
            }
            DEBUG_PRINT1("Received CONN_IDENT");
            target.ident = MSG_CHARP(msg);
            found = (transfer_t *)rbfind(&target, transfers);
            if (found == NULL
                || (trnsfr != NULL && trnsfr != found)
                || (trnsfr == NULL && found->thread_exists))
            {
                const char *reason;
                if (found == NULL) {
                    reason = "Unknown ident";
                } else if (trnsfr != NULL && trnsfr != found) {
                    reason = "Unexpected ident";
                } else {
                    reason = "Duplicate ident";
                }

                sendString(q, skMsgChannel(msg), EXTERNAL,
                           CONN_DISCONNECT, LOG_WARNING,
                           "%s %s", reason, target.ident);
                proto_err = 1;
                retval = exit_failure;
                break;
            }
            ident = found->ident;
            found->thread = pthread_self();
            found->thread_exists = 1;
            found->channel = channel;
            found->channel_exists = 1;
            found->remote_version = version;

            getConnectionInformation(q, channel, connection_type,
                                     sizeof(connection_type));
            INFOMSG("Connected to remote %s (%s, Protocol v%" PRIu32 ")",
                    ident, connection_type, version);
            state = Ready;
            proto_err = skMsgQueueSendMessage(q, channel, CONN_READY, NULL, 0);
            if (proto_err != 0) {
                DEBUG_PRINT1("skMsgQueueSendMessage(CONN_READY) failed");
                retval = exit_failure;
            }
            break;

          case Ready:
            /* expecting remote to say it is ready. if ready, call
             * transferFiles() */
            if ((proto_err = checkMsg(msg, q, CONN_READY))) {
                DEBUG_PRINT1("checkMsg(CONN_READY) FAILED");
                retval = exit_failure;
                break;
            }
            DEBUGMSG("Remote %s is ready for messages", ident);
            state = Running;
            rv = transferFiles(q, channel, found);
            switch (rv) {
              case -1:
                fatal_err = 1;
                break;
              case 1:
                transferred_file = 1;
                break;
              default:
                break;
            }
            break;

          case Disconnect:
            DEBUG_PRINT1("Disconnecting");
            break;

          case Running:
            ASSERT_ABORT(0);
        }

        /* Free message */
        skMsgDestroy(msg);
    }

    if (found) {
        found->channel_exists = 0;
        found->disconnect = 0;
    }

    skMsgQueueDestroy(q);

    /* If running in server mode, this was a detached thread. */
    if (trnsfr == NULL) {
        if (found) {
            found->thread_exists = 0;
        }
        pthread_mutex_lock(&detached_thread_mutex);
        detached_thread_count--;
        pthread_cond_signal(&detached_thread_cond);
        pthread_mutex_unlock(&detached_thread_mutex);
    }

    DEBUG_PRINT2("connection thread ended (status = %s)",
                 ((fatal_err)
                  ? "exit_failure [from transferFiles()]"
                  : ((exit_standard == retval)
                     ? "exit_standard"
                     : ((exit_disconnect == retval)
                        ? "exit_disconnect"
                        : ((exit_failure == retval)
                           ? "exit_failure"
                           : "UNKNOWN")))));

    if (fatal_err) {
        threadExit(EXIT_FAILURE, exit_failure);
    }

    return retval;
}


static int attemptBind(
    const sk_sockaddr_array_t  *addr,
    unsigned                   *UNUSED_NO_GNUTLS(tls),
    const char                **UNUSED_NO_GNUTLS(connection_type))
{
#if SK_ENABLE_GNUTLS
    if (tls_ca_file) {
        *tls = 1;
        *connection_type = "TCP, TLS";
        return skMsgQueueBindTLS(control, addr);
    }
#endif /* SK_ENABLE_GNUTLS */

    return skMsgQueueBindTCP(control, addr);
}


/*
 *    THREAD ENTRY POINT
 *
 *    Entry point for the "server_main" thread, started from
 *    startTransferDaemon().
 */
static void *serverMain(
    void    UNUSED(*dummy))
{
    int rv;
    unsigned tls = 0;
    const char *connection_type = "TCP";

    control_thread_valid = 1;

    DEBUG_PRINT1("server_main() thread started");

    assert(listen_address);

    rv = attemptBind(listen_address, &tls, &connection_type);
    if (rv < 0) {
        CRITMSG("Failed to bind to %s for listening", listen_address_arg);
        threadExit(EXIT_FAILURE, NULL);
    }

    INFOMSG("Bound to %s for listening (%s)",
            listen_address_arg, connection_type);

    while (!shuttingdown) {
        sk_msg_t *msg;
        skm_channel_t channel;
        pthread_t thread;
        conn_info_t *info;
        transfer_t *item;
        RBLIST *list;
        sk_new_channel_info_t *addr_info;
        char buf[PATH_MAX];
        char conn_type[RWTRANSFER_CONNECTION_TYPE_SIZE_MAX];

        rv = skMsgQueueGetMessageFromChannel(control, SKMSG_CHANNEL_CONTROL,
                                             &msg);
        if (rv == -1) {
            ASSERT_ABORT(shuttingdown);
            continue;
        }

        switch (skMsgType(msg)) {

          case SKMSG_CTL_NEW_CONNECTION:
            DEBUG_PRINT1("Received SKMSG_CTL_NEW_CONNECTION");
            channel = SKMSG_CTL_MSG_GET_CHANNEL(msg);
            addr_info = (sk_new_channel_info_t *)skMsgMessage(msg);
            getConnectionInformation(control, channel, conn_type,
                                     sizeof(conn_type));
            if (addr_info->known) {
                skSockaddrString(buf, sizeof(buf), &addr_info->addr);
            }
            INFOMSG("Received connection from %s (%s)",
                    (addr_info->known ? buf :
                     "unknown address"), conn_type);
            info = (conn_info_t *)calloc(1, sizeof(*info));
            if (info == NULL) {
                CRITMSG("Memory allocation failure");
                threadExit(EXIT_FAILURE, NULL);
            }
            info->tls = tls;
            info->trnsfr = NULL;
            rv = skMsgChannelSplit(control, channel, &info->queue);
            if (rv != 0) {
                if (shuttingdown) {
                    break;
                }
                CRITMSG("Failed to split channel");
                threadExit(EXIT_FAILURE, NULL);
            }
            info->channel = channel;

            /* In server mode we don't have one thread per ident.
             * Instead we have one thread per entity that is
             * connecting to us.  Since there is no transfer object to
             * attach the thread to, we create a detached thread
             * instead, and use the detached_thread_mutex and
             * detached_thread_count to know when the threads have
             * ended. */
            pthread_mutex_lock(&detached_thread_mutex);
            rv = skthread_create_detached("connection", &thread,
                                          handleConnection, info);
            if (rv != 0) {
                pthread_mutex_unlock(&detached_thread_mutex);
                CRITMSG("Failed to create connection thread: %s",
                        strerror(rv));
                threadExit(EXIT_FAILURE, NULL);
            }
            detached_thread_count++;
            pthread_mutex_unlock(&detached_thread_mutex);
            break;

          case SKMSG_CTL_CHANNEL_DIED:
            DEBUG_PRINT1("Received SKMSG_CTL_CHANNEL_DIED");
            channel = SKMSG_CTL_MSG_GET_CHANNEL(msg);
            list = rbopenlist(transfers);
            CHECK_ALLOC(list);
            while ((item = (transfer_t *)rbreadlist(list)) != NULL) {
                if (item->channel_exists && channel == item->channel) {
                    INFOMSG("Channel to %s died", item->ident);
                    item->disconnect = 1;
                    rv = transferUnblock(item);
                    if (rv != 0) {
                        threadExit(EXIT_FAILURE, NULL);
                    }
                    break;
                }
            }
            rbcloselist(list);

            if (!shuttingdown) {
                sendString(control, channel, INTERNAL, CONN_DISCONNECT_RETRY,
                           LOG_INFO, "Remote side of channel died");
            }
            break;

          default:
            WARNINGMSG("Received unknown control message %d", skMsgType(msg));
        }

        skMsgDestroy(msg);
    }

    DEBUG_PRINT1("server_main() thread ended");

    return NULL;
}


/*
 *    THREAD ENTRY POINT
 *
 *    Entry point for the "connection" thread, started from
 *    clientMain().
 */
static void *startClientConnection(
    void           *vitem)
{
    transfer_t *item = (transfer_t *)vitem;
    void *exit_status = exit_standard;
    int waitsecs = 0;
    const char *connection_type = "TCP";
    connection_fn_t connection_function;
    socklen_t addrlen;
    int tls = 0;
    char buf[SK_NUM2DOT_STRLEN];


    item->thread_exists = 1;

    DEBUG_PRINT1("client_connection() thread started");

    while (!shuttingdown) {
        size_t i;
        int rv;
        skm_channel_t channel;

        if (waitsecs != 0) {
            int waitcount = waitsecs;

            DEBUG_PRINT2("Failure in connection, "
                         "waiting %d seconds", waitcount);
            while (waitcount-- && !shuttingdown) {
                sleep(1);
            }
            if (shuttingdown) {
                break;
            }
        }

        connection_function = skMsgQueueConnectTCP;
#if SK_ENABLE_GNUTLS
        if (tls_ca_file) {
            tls = 1;
            connection_function = skMsgQueueConnectTLS;
            connection_type = "TCP, TLS";
        }
#endif /* SK_ENABLE_GNUTLS */

        INFOMSG("Attempting to connect to %s (%s)...",
                item->ident, connection_type);

        for (rv = -1, i = 0;
             rv != 0 && i < skSockaddrArraySize(item->addr); i++)
        {
            sk_sockaddr_t *addr = skSockaddrArrayGet(item->addr, i);
            switch (addr->sa.sa_family) {
              case AF_INET:
                addrlen = sizeof(addr->v4);
                break;
              case AF_INET6:
                addrlen = sizeof(addr->v6);
                break;
              default:
                continue;
            }
            skSockaddrString(buf, sizeof(buf), addr);
            DEBUGMSG("Address for %s is %s", item->ident, buf);
            rv = connection_function(control, &addr->sa, addrlen, &channel);
        }

        if (rv != 0) {
            INFOMSG("Attempt to connect to %s failed (%s)",
                    item->ident, connection_type);
            if (waitsecs < 60) {
                waitsecs += 5;
            }
        } else {
            conn_info_t *info;
            char conn_type[RWTRANSFER_CONNECTION_TYPE_SIZE_MAX];

            getConnectionInformation(control, channel, conn_type,
                                     sizeof(conn_type));
            DEBUGMSG("Connected (expecting ident %s) (%s)",
                     item->ident, conn_type);
            info = (conn_info_t *)calloc(1, sizeof(*info));
            if (info == NULL) {
                CRITMSG("Memory allocation failure");
                threadExit(EXIT_FAILURE, exit_failure);
            }
            info->tls = tls;
            info->trnsfr = item;
            rv = skMsgChannelSplit(control, channel, &info->queue);
            if (rv != 0) {
                if (shuttingdown) {
                    break;
                }
                CRITMSG("Failed to split channel");
                threadExit(EXIT_FAILURE, exit_failure);
            }

            info->channel = channel;
            exit_status = handleConnection(info);
            if (exit_status != exit_failure) {
                waitsecs = 0;
            } else if (waitsecs < 60) {
                waitsecs += 5;
            }
        }
    }

    DEBUG_PRINT1("client_connection() thread ended");

    return exit_status;
}


/*
 *    THREAD ENTRY POINT
 *
 *    Entry point for the "client_main" thread, started from
 *    startTransferDaemon()
 */
static void *clientMain(
    void    UNUSED(*dummy))
{
    RBLIST *list;
    transfer_t *item;
    int rv;

    control_thread_valid = 1;

    DEBUG_PRINT1("client_main() thread started");

    list = rbopenlist(transfers);
    if (list == NULL) {
        skAppPrintErr("Memory allocation failure stating client thread");
        threadExit(EXIT_FAILURE, NULL);
    }

    /* Start client threads */
    while ((item = (transfer_t *)rbreadlist(list)) != NULL) {
        rv = skthread_create("connection", &item->thread,
                             startClientConnection, item);
        if (rv != 0) {
            CRITMSG("Failed to create connection thread: %s", strerror(rv));
            threadExit(EXIT_FAILURE, NULL);
        }
    }
    rbcloselist(list);

    /* Start control loop */
    while (!shuttingdown) {
        sk_msg_t *msg;
        skm_channel_t channel;

        rv = skMsgQueueGetMessageFromChannel(control, SKMSG_CHANNEL_CONTROL,
                                             &msg);
        if (rv == -1) {
            assert(shuttingdown);
            continue;
        }

        switch (skMsgType(msg)) {

          case SKMSG_CTL_NEW_CONNECTION:
            /* This can't happen, as we aren't bound */
            ASSERT_ABORT(0);
            break;

          case SKMSG_CTL_CHANNEL_DIED:
            DEBUG_PRINT1("Received SKMSG_CTL_CHANNEL_DIED");
            channel = SKMSG_CTL_MSG_GET_CHANNEL(msg);
            list = rbopenlist(transfers);
            CHECK_ALLOC(list);
            while ((item = (transfer_t *)rbreadlist(list)) != NULL) {
                if (item->channel_exists && channel == item->channel) {
                    INFOMSG("Channel to %s died", item->ident);
                    item->disconnect = 1;
                    rv = transferUnblock(item);
                    if (rv != 0) {
                        threadExit(EXIT_FAILURE, NULL);
                    }
                    break;
                }
            }
            rbcloselist(list);

            sendString(control, channel, INTERNAL, CONN_DISCONNECT_RETRY,
                       LOG_INFO, "Remote side of channel died");
            break;

          default:
            WARNINGMSG("Received unknown control message %d", skMsgType(msg));
        }

        skMsgDestroy(msg);
    }

    DEBUG_PRINT1("client_main() thread ended");
    return NULL;
}


int startTransferDaemon(
    void)
{
    int rv;

    /* Initialize the message queue */
    rv = skMsgQueueCreate(&control);
    if (rv != 0) {
        skAppPrintErr("Failed to initialize message queue");
        exit(EXIT_FAILURE);
    }

#if SK_ENABLE_GNUTLS
    if (tls_ca_file) {
        rv = skMsgQueueAddCA(control, tls_ca_file);
        if (rv != 0) {
            CRITMSG("Invalid Certificate Authority file: %s", tls_ca_file);
            return -1;
        }
#ifdef PKCS12
        if (tls_pkcs12_file) {
            const char *password = getenv(password_env);
            if (password == NULL) {
                password = "";
            }
            rv = skMsgQueueAddPKCS12(control, tls_pkcs12_file, password);
            if (rv != 0) {
                CRITMSG("Invalid encryption cert file: %s", tls_pkcs12_file);
                return -1;
            }
        } else
#endif /* PKCS12 */
        {
            rv = skMsgQueueAddCert(control, tls_cert_file, tls_key_file);
            if (rv != 0) {
                CRITMSG("Invalid encryption cert or key file: %s, %s",
                        tls_cert_file, tls_key_file);
                return -1;
            }
        }
    }
#endif /* SK_ENABLE_GNUTLS */

    switch (mode) {
      case CLIENT:
        rv = skthread_create("client_main", &control_thread,
                             clientMain, NULL);
        if (rv != 0) {
            CRITMSG("Failed to create primary client thread: %s",strerror(rv));
            return -1;
        }
        break;
      case SERVER:
        rv = skthread_create("server_main", &control_thread,
                             serverMain, NULL);
        if (rv != 0) {
            CRITMSG("Failed to create primary server thread: %s",strerror(rv));
            return -1;
        }
        break;
      default:
        ASSERT_ABORT(0);
    }

    return 0;
}


int checkMsg(
    sk_msg_t           *msg,
    sk_msg_queue_t     *q,
    connection_msg_t    type)
{
    skm_type_t t;
    skm_len_t  len;

    assert(msg);
    assert(q);
    assert(type < CONN_NUMBER_OF_CONNECTION_MESSAGES);

    t = skMsgType(msg);

    if (t != type) {
        sendString(q, skMsgChannel(msg), EXTERNAL,
                   CONN_DISCONNECT, LOG_WARNING,
                   "Protocol error: expected %s, got %s (0x%04x)",
                   conn_msg_data[type].name,
                   ((t >= CONN_NUMBER_OF_CONNECTION_MESSAGES)
                    ? "<unknown>"
                    : conn_msg_data[t].name),
                   t);
        return 1;
    }

    if (conn_msg_data[type].size == -1) {
        return 0;
    }

    len = skMsgLength(msg);
    if (len != conn_msg_data[type].size) {
        sendString(q, skMsgChannel(msg), EXTERNAL,
                   CONN_DISCONNECT, LOG_WARNING,
                   "Protocol error: type %s, expected len %" PRId32 ", got %d",
                   conn_msg_data[type].name,
                   conn_msg_data[type].size, len);
        return 1;
    }

    return 0;
}


#undef sendString
int sendString(
    sk_msg_queue_t *q,
    skm_channel_t   channel,
    int             internal,
    skm_type_t      type,
    int             loglevel,
    const char     *fmt,
    ...)
{
    int rv;
    va_list args;
    char msg[MAX_ERROR_MESSAGE];
    int len;

    va_start(args, fmt);
    len = vsnprintf(msg, sizeof(msg), fmt, args);
    if (len >= (int)sizeof(msg)) {
        len = sizeof(msg) - 1;
        msg[len] = '\0';
    }

    if (internal) {
        rv = skMsgQueueInjectMessage(q, channel, type, msg, len + 1);
    } else {
        rv = skMsgQueueSendMessage(q, channel, type, msg, len + 1);
    }

    if (!internal) {
        sklog(loglevel, "Sending \"%s\"", msg);
    }
    return rv;
}


void threadExit(
    int             status,
    void           *retval)
{
    DEBUG_PRINT1("threadExit called");
    main_retval = status;
    pthread_kill(main_thread, SIGQUIT);
    pthread_exit(retval);
}


/*
** Local Variables:
** mode:c
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael.

On Thu, Jan 17, 2013 at 11:27:04AM -0500, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> [This] fixes this case, indeed.  Unfortunately, here's another.

> emacs -Q rwtransfer.c
>   M-x c-toggle-parse-state-debug
>   M->
> Then type M-v until the parse failure happens.  For me it happens at 
> 32% point in the file.  Please not that this particular problem only
> happens when paging through the file in reverse order.

> c-parse-state inconsistency at 15885: using cache: (15636 15181 (13849 . 15141)), from scratch: (15636 (15303 . 15493) 15181 (13849 . 15141))
> Old state:
> (setq c-state-cache '(16334 16271 16248 (16059 . 16193) 15181 (13849 . 15141))  c-state-cache-good-pos 16335  c-state-nonlit-pos-cache '(48473 45473 42473 39473 36473 33473 30473 27378 24378 21260 18015 15015 12015 9015 6015 3015)  c-state-nonlit-pos-cache-limit 48473  c-state-semi-nonlit-pos-cache '(48323 45323 42323 39323 36323 33260 30260 27260 24260 21260 18001 15001 12001 9001 6001 3001)  c-state-semi-nonlit-pos-cache-limit 48323  c-state-brace-pair-desert '(15181 . 17388)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil  c-parse-state-point 16335)
> c-parse-state inconsistency at 15885: using cache: (15636 15181 (13849 . 15141)), from scratch: (15636 (15303 . 15493) 15181 (13849 . 15141))

I think the following patch should fix that glitch.  I've also taken the
opportunity to simplify things a little (part of the global state was
being passed needlessly as a parameter) and to rename some variables for
more consistency.

Would you try this out and confirm it works, as usual, please.

However, this isn't the end of the story - 

(i) Start your test case in rwtransfer.c, and M-v until the top of the
  file.  (This shouldn't display any inconsistency messages)
(ii) M-x goto-char <ret> 20001 <ret>.  This should go to EOL 671, just
  after a "}".
(iii) Type <space> }.  (Two characters)
(iv) C-v, possibly twice.

This displays a message something like:
    c-parse-state inconsistency at 21070: using cache: ((20458 . 20935)),
    from scratch: ((20838 . 20877))
, together with a state dump.

This bug isn't a new one, but I've just come across it.  I'll be working
on it in the meantime.

Here's the patch, based on the savannah emacs-24 branch (which should be
identical to the trunk):



=== modified file 'lisp/progmodes/cc-engine.el'
*** lisp/progmodes/cc-engine.el	2013-01-09 21:33:00 +0000
--- lisp/progmodes/cc-engine.el	2013-01-23 13:36:33 +0000
***************
*** 2545,2558 ****
    ;;
    ;; The return value is a list, one of the following:
    ;;
!   ;; o - ('forward CACHE-POS START-POINT) - scan forward from START-POINT,
!   ;;                                        which is not less than CACHE-POS.
!   ;; o - ('backward CACHE-POS nil) - scan backwards (from HERE).
!   ;; o - ('BOD nil START-POINT) - scan forwards from START-POINT, which is at the
    ;;   top level.
!   ;; o - ('IN-LIT nil nil) - point is inside the literal containing point-min.
!   ;; , where CACHE-POS is the highest position recorded in `c-state-cache' at
!   ;; or below HERE.
    (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
  	BOD-pos		    ; position of 2nd BOD before HERE.
  	strategy	    ; 'forward, 'backward, 'BOD, or 'IN-LIT.
--- 2545,2556 ----
    ;;
    ;; The return value is a list, one of the following:
    ;;
!   ;; o - ('forward START-POINT) - scan forward from START-POINT,
!   ;;	 which is not less than the highest position in `c-state-cache' below here.
!   ;; o - ('backward nil) - scan backwards (from HERE).
!   ;; o - ('BOD START-POINT) - scan forwards from START-POINT, which is at the
    ;;   top level.
!   ;; o - ('IN-LIT nil) - point is inside the literal containing point-min.
    (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
  	BOD-pos		    ; position of 2nd BOD before HERE.
  	strategy	    ; 'forward, 'backward, 'BOD, or 'IN-LIT.
***************
*** 2590,2596 ****
  
      (list
       strategy
-      (and (memq strategy '(forward backward)) cache-pos)
       (and (memq strategy '(forward BOD)) start-point))))
  
  
--- 2588,2593 ----
***************
*** 2657,2663 ****
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
!       (let* (new-cons
  	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
  	     (macro-start-or-from
  	      (progn (goto-char from)
--- 2654,2661 ----
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
!       (let* ((here (point-max))
! 	     new-cons
  	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
  	     (macro-start-or-from
  	      (progn (goto-char from)
***************
*** 2692,2697 ****
--- 2690,2696 ----
  		;; search bound, even though the algorithm below would skip
  		;; over the new paren pair.
  		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
+ 	    (widen)
  	    (narrow-to-region
  		(cond
  		 ((and desert-lim cache-lim)
***************
*** 2711,2726 ****
  		  (while
  		      (and (setq ce (scan-lists bra -1 -1)) ; back past )/]/}; might signal
  			   (setq bra (scan-lists ce -1 1)) ; back past (/[/{; might signal
! 			   (or (> ce upper-lim)
! 			       (not (eq (char-after bra) ?\{))
! 			       (and (goto-char bra)
! 				    (c-beginning-of-macro)
! 				    (< (point) macro-start-or-from))))))
  		(and ce (< ce bra)))
  	    (setq bra ce))	; If we just backed over an unbalanced closing
  					; brace, ignore it.
  
! 	  (if (and ce (< bra ce) (eq (char-after bra) ?\{))
  	      ;; We've found the desired brace-pair.
  	      (progn
  		(setq new-cons (cons bra (1+ ce)))
--- 2710,2727 ----
  		  (while
  		      (and (setq ce (scan-lists bra -1 -1)) ; back past )/]/}; might signal
  			   (setq bra (scan-lists ce -1 1)) ; back past (/[/{; might signal
! 			   (or (> bra here) ;(> ce here)
! 			       (and
! 				(< ce here)
! 				(or (not (eq (char-after bra) ?\{))
! 				    (and (goto-char bra)
! 					 (c-beginning-of-macro)
! 					 (< (point) macro-start-or-from))))))))
  		(and ce (< ce bra)))
  	    (setq bra ce))	; If we just backed over an unbalanced closing
  					; brace, ignore it.
  
! 	  (if (and ce (< ce here) (< bra ce) (eq (char-after bra) ?\{))
  	      ;; We've found the desired brace-pair.
  	      (progn
  		(setq new-cons (cons bra (1+ ce)))
***************
*** 2734,2740 ****
  		 (t (setq c-state-cache (cons new-cons c-state-cache)))))
  
  	    ;; We haven't found a brace pair.  Record this in the cache.
! 	    (setq c-state-brace-pair-desert (cons cache-pos from))))))))
  
  (defsubst c-state-push-any-brace-pair (bra+1 macro-start-or-here)
    ;; If BRA+1 is nil, do nothing.  Otherwise, BRA+1 is the buffer position
--- 2735,2745 ----
  		 (t (setq c-state-cache (cons new-cons c-state-cache)))))
  
  	    ;; We haven't found a brace pair.  Record this in the cache.
! 	    (setq c-state-brace-pair-desert
! 		  (cons (if (and ce (< bra ce) (> ce here)) ; {..} straddling HERE?
! 			    bra
! 			  (point-min))
! 			(min here from)))))))))
  
  (defsubst c-state-push-any-brace-pair (bra+1 macro-start-or-here)
    ;; If BRA+1 is nil, do nothing.  Otherwise, BRA+1 is the buffer position
***************
*** 2852,2870 ****
         (paren+1)
         (t from)))))
  
! (defun c-remove-stale-state-cache (good-pos pps-point)
    ;; Remove stale entries from the `c-cache-state', i.e. those which will
    ;; not be in it when it is amended for position (point-max).
    ;; Additionally, the "outermost" open-brace entry before (point-max)
    ;; will be converted to a cons if the matching close-brace is scanned.
    ;;
!   ;; GOOD-POS is a "maximal" "safe position" - there must be no open
!   ;; parens/braces/brackets between GOOD-POS and (point-max).
    ;;
    ;; As a second thing, calculate the result of parse-partial-sexp at
!   ;; PPS-POINT, w.r.t. GOOD-POS.  The motivation here is that
    ;; `c-state-cache-good-pos' may become PPS-POINT, but the caller may need to
!   ;; adjust it to get outside a string/comment.  (Sorry about this!  The code
    ;; needs to be FAST).
    ;;
    ;; Return a list (GOOD-POS SCAN-BACK-POS PPS-STATE), where
--- 2857,2875 ----
         (paren+1)
         (t from)))))
  
! (defun c-remove-stale-state-cache (start-point pps-point)
    ;; Remove stale entries from the `c-cache-state', i.e. those which will
    ;; not be in it when it is amended for position (point-max).
    ;; Additionally, the "outermost" open-brace entry before (point-max)
    ;; will be converted to a cons if the matching close-brace is scanned.
    ;;
!   ;; START-POINT is a "maximal" "safe position" - there must be no open
!   ;; parens/braces/brackets between START-POINT and (point-max).
    ;;
    ;; As a second thing, calculate the result of parse-partial-sexp at
!   ;; PPS-POINT, w.r.t. START-POINT.  The motivation here is that
    ;; `c-state-cache-good-pos' may become PPS-POINT, but the caller may need to
!   ;; adjust it to get outside a string/comment.	 (Sorry about this!  The code
    ;; needs to be FAST).
    ;;
    ;; Return a list (GOOD-POS SCAN-BACK-POS PPS-STATE), where
***************
*** 2872,2878 ****
    ;;   to be good (we aim for this to be as high as possible);
    ;; o - SCAN-BACK-POS, if not nil, indicates there may be a brace pair
    ;;   preceding POS which needs to be recorded in `c-state-cache'.  It is a
!   ;;   position to scan backwards from.
    ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
    (save-restriction
      (narrow-to-region 1 (point-max))
--- 2877,2885 ----
    ;;   to be good (we aim for this to be as high as possible);
    ;; o - SCAN-BACK-POS, if not nil, indicates there may be a brace pair
    ;;   preceding POS which needs to be recorded in `c-state-cache'.  It is a
!   ;;   position to scan backwards from.  It is the position of the "{" of the
!   ;;   last element to be removed from `c-state-cache', when that elt is a
!   ;;   cons, otherwise nil.
    ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
    (save-restriction
      (narrow-to-region 1 (point-max))
***************
*** 2882,2902 ****
  		(goto-char (point-max))
  		(and (c-beginning-of-macro)
  		     (point))))
! 	     (good-pos-actual-macro-start ; Start of macro containing good-pos
! 					; or nil
! 	      (and (< good-pos (point-max))
  		   (save-excursion
! 		     (goto-char good-pos)
  		     (and (c-beginning-of-macro)
  			  (point)))))
! 	     (good-pos-actual-macro-end	; End of this macro, (maybe
  					; (point-max)), or nil.
! 	      (and good-pos-actual-macro-start
  		   (save-excursion
! 		     (goto-char good-pos-actual-macro-start)
  		     (c-end-of-macro)
  		     (point))))
! 	     pps-state 			; Will be 9 or 10 elements long.
  	     pos
  	     upper-lim	   ; ,beyond which `c-state-cache' entries are removed
  	     scan-back-pos
--- 2889,2909 ----
  		(goto-char (point-max))
  		(and (c-beginning-of-macro)
  		     (point))))
! 	     (start-point-actual-macro-start ; Start of macro containing
! 					     ; start-point or nil
! 	      (and (< start-point (point-max))
  		   (save-excursion
! 		     (goto-char start-point)
  		     (and (c-beginning-of-macro)
  			  (point)))))
! 	     (start-point-actual-macro-end ; End of this macro, (maybe
  					; (point-max)), or nil.
! 	      (and start-point-actual-macro-start
  		   (save-excursion
! 		     (goto-char start-point-actual-macro-start)
  		     (c-end-of-macro)
  		     (point))))
! 	     pps-state			; Will be 9 or 10 elements long.
  	     pos
  	     upper-lim	   ; ,beyond which `c-state-cache' entries are removed
  	     scan-back-pos
***************
*** 2926,2940 ****
  	;; The next loop jumps forward out of a nested level of parens each
  	;; time round; the corresponding elements in `c-state-cache' are
  	;; removed.  `pos' is just after the brace-pair or the open paren at
! 	;; (car c-state-cache).  There can be no open parens/braces/brackets
! 	;; between `good-pos'/`good-pos-actual-macro-start' and (point-max),
  	;; due to the interface spec to this function.
! 	(setq pos (if (and good-pos-actual-macro-end
! 			   (not (eq good-pos-actual-macro-start
  				    in-macro-start)))
! 		      (1+ good-pos-actual-macro-end) ; get outside the macro as
  					; marked by a `category' text property.
! 		    good-pos))
  	(goto-char pos)
  	(while (and c-state-cache
  		    (< (point) (point-max)))
--- 2933,2947 ----
  	;; The next loop jumps forward out of a nested level of parens each
  	;; time round; the corresponding elements in `c-state-cache' are
  	;; removed.  `pos' is just after the brace-pair or the open paren at
! 	;; (car c-state-cache).	 There can be no open parens/braces/brackets
! 	;; between `start-point'/`start-point-actual-macro-start' and (point-max),
  	;; due to the interface spec to this function.
! 	(setq pos (if (and start-point-actual-macro-end
! 			   (not (eq start-point-actual-macro-start
  				    in-macro-start)))
! 		      (1+ start-point-actual-macro-end) ; get outside the macro as
  					; marked by a `category' text property.
! 		    start-point))
  	(goto-char pos)
  	(while (and c-state-cache
  		    (< (point) (point-max)))
***************
*** 2993,3006 ****
  
  	(list pos scan-back-pos pps-state)))))
  
! (defun c-remove-stale-state-cache-backwards (here cache-pos)
    ;; Strip stale elements of `c-state-cache' by moving backwards through the
    ;; buffer, and inform the caller of the scenario detected.
    ;;
    ;; HERE is the position we're setting `c-state-cache' for.
!   ;; CACHE-POS is just after the latest recorded position in `c-state-cache'
!   ;;   before HERE, or a position at or near point-min which isn't in a
!   ;;   literal.
    ;;
    ;; This function must only be called only when (> `c-state-cache-good-pos'
    ;; HERE).  Usually the gap between CACHE-POS and HERE is large.  It is thus
--- 3000,3013 ----
  
  	(list pos scan-back-pos pps-state)))))
  
! (defun c-remove-stale-state-cache-backwards (here)
    ;; Strip stale elements of `c-state-cache' by moving backwards through the
    ;; buffer, and inform the caller of the scenario detected.
    ;;
    ;; HERE is the position we're setting `c-state-cache' for.
!   ;; CACHE-POS (a locally bound variable) is just after the latest recorded
!   ;;   position in `c-state-cache' before HERE, or a position at or near
!   ;;   point-min which isn't in a literal.
    ;;
    ;; This function must only be called only when (> `c-state-cache-good-pos'
    ;; HERE).  Usually the gap between CACHE-POS and HERE is large.  It is thus
***************
*** 3023,3032 ****
    ;; The comments in this defun use "paren" to mean parenthesis or square
    ;; bracket (as contrasted with a brace), and "(" and ")" likewise.
    ;;
!   ;;    .   {..} (..) (..)  ( .. {   }  ) (...)    ( ....          .  ..)
!   ;;    |                   |       |   |     |                    |
!   ;;    CP                  E      here D     C                   good
!   (let ((pos c-state-cache-good-pos)
  	pa ren	       ; positions of "(" and ")"
  	dropped-cons ; whether the last element dropped from `c-state-cache'
  		     ; was a cons (representing a brace-pair)
--- 3030,3040 ----
    ;; The comments in this defun use "paren" to mean parenthesis or square
    ;; bracket (as contrasted with a brace), and "(" and ")" likewise.
    ;;
!   ;;	.   {..} (..) (..)  ( .. {   }	) (...)	   ( ....	   .  ..)
!   ;;	|		    |	    |	|     |			   |
!   ;;	CP		    E	   here D     C			  good
!   (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
! 	(pos c-state-cache-good-pos)
  	pa ren	       ; positions of "(" and ")"
  	dropped-cons ; whether the last element dropped from `c-state-cache'
  		     ; was a cons (representing a brace-pair)
***************
*** 3263,3269 ****
  	 cache-pos	     ; highest position below HERE already existing in
  			     ; cache (or 1).
  	 good-pos
! 	 start-point
  	 bopl-state
  	 res
  	 scan-backward-pos scan-forward-p) ; used for 'backward.
--- 3271,3278 ----
  	 cache-pos	     ; highest position below HERE already existing in
  			     ; cache (or 1).
  	 good-pos
! 	 start-point ; (when scanning forward) a place below HERE where there
! 		     ; are no open parens/braces between it and HERE.
  	 bopl-state
  	 res
  	 scan-backward-pos scan-forward-p) ; used for 'backward.
***************
*** 3274,3281 ****
      ;; Strategy?
      (setq res (c-parse-state-get-strategy here c-state-cache-good-pos)
  	  strategy (car res)
! 	  cache-pos (cadr res)
! 	  start-point (nth 2 res))
  
      (when (eq strategy 'BOD)
        (setq c-state-cache nil
--- 3283,3289 ----
      ;; Strategy?
      (setq res (c-parse-state-get-strategy here c-state-cache-good-pos)
  	  strategy (car res)
! 	  start-point (cadr res))
  
      (when (eq strategy 'BOD)
        (setq c-state-cache nil
***************
*** 3302,3308 ****
  		good-pos)))
  
         ((eq strategy 'backward)
! 	(setq res (c-remove-stale-state-cache-backwards here cache-pos)
  	      good-pos (car res)
  	      scan-backward-pos (cadr res)
  	      scan-forward-p (car (cddr res)))
--- 3310,3316 ----
  		good-pos)))
  
         ((eq strategy 'backward)
! 	(setq res (c-remove-stale-state-cache-backwards here)
  	      good-pos (car res)
  	      scan-backward-pos (cadr res)
  	      scan-forward-p (car (cddr res)))



> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Michael.
>
> On Thu, Jan 17, 2013 at 11:27:04AM -0500, Michael Welsh Duggan wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>> [This] fixes this case, indeed.  Unfortunately, here's another.
>

[...]

> I think the following patch should fix that glitch.  I've also taken the
> opportunity to simplify things a little (part of the global state was
> being passed needlessly as a parameter) and to rename some variables for
> more consistency.
>
> Would you try this out and confirm it works, as usual, please.

I can confirm that this fixes this particular problem.

> However, this isn't the end of the story - 
>
> (i) Start your test case in rwtransfer.c, and M-v until the top of the
>   file.  (This shouldn't display any inconsistency messages)
> (ii) M-x goto-char <ret> 20001 <ret>.  This should go to EOL 671, just
>   after a "}".
> (iii) Type <space> }.  (Two characters)
> (iv) C-v, possibly twice.
>
> This displays a message something like:
>     c-parse-state inconsistency at 21070: using cache: ((20458 . 20935)),
>     from scratch: ((20838 . 20877))
> , together with a state dump.
>
> This bug isn't a new one, but I've just come across it.  I'll be working
> on it in the meantime.

Oh, good.  I haven't reported any "type stuff" errors I have been
getting because they are harder to reproduce.  (Unless it is triggered
the first time you've typed in the buffer, it's hard to replicate the
exact scenario.)  I usually wait until you've fixed the last thing I
sent in before attempting to report another, just in case the thing you
are fixing is related.

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Michael.

On Wed, Jan 23, 2013 at 10:39:34AM -0500, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > However, this isn't the end of the story - 

> > (i) Start your test case in rwtransfer.c, and M-v until the top of the
> >   file.  (This shouldn't display any inconsistency messages)
> > (ii) M-x goto-char <ret> 20001 <ret>.  This should go to EOL 671, just
> >   after a "}".
> > (iii) Type <space> }.  (Two characters)
> > (iv) C-v, possibly twice.

> > This displays a message something like:
> >     c-parse-state inconsistency at 21070: using cache: ((20458 . 20935)),
> >     from scratch: ((20838 . 20877))
> > , together with a state dump.

> > This bug isn't a new one, but I've just come across it.  I'll be working
> > on it in the meantime.

> Oh, good.  I haven't reported any "type stuff" errors I have been
> getting because they are harder to reproduce.  (Unless it is triggered
> the first time you've typed in the buffer, it's hard to replicate the
> exact scenario.)  I usually wait until you've fixed the last thing I
> sent in before attempting to report another, just in case the thing you
> are fixing is related.

OK, I think I've fixed that above bug.  I've also done a moderate amount
of refactoring to simplify things for the next bug, if any.

I've removed the scanning strategy "start at the 2nd previous column-0
beginning-of-defun", because it was returning false results, even in C
Mode.  I'd already disabled it for C++ Mode (because hackers frequently
put things inside namespaces at column 0).  One consequence is that
running with `c-toggle-parse-state-debug' is somewhat slower.  Sorry for
that.

Could you try this new version, please.  The patch is based on the
emacs-24 branch of the savannah bzr repository.  This should be identical
to the trunk.



=== modified file 'lisp/progmodes/cc-engine.el'
*** lisp/progmodes/cc-engine.el	2013-01-23 18:58:11 +0000
--- lisp/progmodes/cc-engine.el	2013-01-29 11:03:35 +0000
***************
*** 2477,2496 ****
  
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  ;; Defuns which analyze the buffer, yet don't change `c-state-cache'.
- (defun c-get-fallback-scan-pos (here)
-   ;; Return a start position for building `c-state-cache' from
-   ;; scratch.  This will be at the top level, 2 defuns back.
-   (save-excursion
-     ;; Go back 2 bods, but ignore any bogus positions returned by
-     ;; beginning-of-defun (i.e. open paren in column zero).
-     (goto-char here)
-     (let ((cnt 2))
-       (while (not (or (bobp) (zerop cnt)))
- 	(c-beginning-of-defun-1)	; Pure elisp BOD.
- 	(if (eq (char-after) ?\{)
- 	    (setq cnt (1- cnt)))))
-     (point)))
- 
  (defun c-state-balance-parens-backwards (here- here+ top)
    ;; Return the position of the opening paren/brace/bracket before HERE- which
    ;; matches the outermost close p/b/b between HERE+ and TOP.  Except when
--- 2477,2482 ----
***************
*** 2548,2594 ****
    ;; o - ('forward START-POINT) - scan forward from START-POINT,
    ;;	 which is not less than the highest position in `c-state-cache' below here.
    ;; o - ('backward nil) - scan backwards (from HERE).
-   ;; o - ('BOD START-POINT) - scan forwards from START-POINT, which is at the
-   ;;   top level.
    ;; o - ('IN-LIT nil) - point is inside the literal containing point-min.
    (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
! 	BOD-pos		    ; position of 2nd BOD before HERE.
! 	strategy	    ; 'forward, 'backward, 'BOD, or 'IN-LIT.
! 	start-point
! 	how-far)			; putative scanning distance.
      (setq good-pos (or good-pos (c-state-get-min-scan-pos)))
      (cond
       ((< here (c-state-get-min-scan-pos))
!       (setq strategy 'IN-LIT
! 	    start-point nil
! 	    cache-pos nil
! 	    how-far 0))
       ((<= good-pos here)
        (setq strategy 'forward
! 	    start-point (max good-pos cache-pos)
! 	    how-far (- here start-point)))
       ((< (- good-pos here) (- here cache-pos)) ; FIXME!!! ; apply some sort of weighting.
!       (setq strategy 'backward
! 	    how-far (- good-pos here)))
       (t
        (setq strategy 'forward
! 	      how-far (- here cache-pos)
! 	      start-point cache-pos)))
! 
!     ;; Might we be better off starting from the top level, two defuns back,
!     ;; instead?  This heuristic no longer works well in C++, where
!     ;; declarations inside namespace brace blocks are frequently placed at
!     ;; column zero.
!     (when (and (not (c-major-mode-is 'c++-mode))
! 	       (> how-far c-state-cache-too-far))
!       (setq BOD-pos (c-get-fallback-scan-pos here)) ; somewhat EXPENSIVE!!!
!       (if (< (- here BOD-pos) how-far)
! 	  (setq strategy 'BOD
! 		start-point BOD-pos)))
! 
!     (list
!      strategy
!      (and (memq strategy '(forward BOD)) start-point))))
  
  
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
--- 2534,2556 ----
    ;; o - ('forward START-POINT) - scan forward from START-POINT,
    ;;	 which is not less than the highest position in `c-state-cache' below here.
    ;; o - ('backward nil) - scan backwards (from HERE).
    ;; o - ('IN-LIT nil) - point is inside the literal containing point-min.
    (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
! 	strategy	    ; 'forward, 'backward, or 'IN-LIT.
! 	start-point)
      (setq good-pos (or good-pos (c-state-get-min-scan-pos)))
      (cond
       ((< here (c-state-get-min-scan-pos))
!       (setq strategy 'IN-LIT))
       ((<= good-pos here)
        (setq strategy 'forward
! 	    start-point (max good-pos cache-pos)))
       ((< (- good-pos here) (- here cache-pos)) ; FIXME!!! ; apply some sort of weighting.
!       (setq strategy 'backward))
       (t
        (setq strategy 'forward
! 	    start-point cache-pos)))
!     (list strategy (and (eq strategy 'forward) start-point))))
  
  
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
***************
*** 2630,2636 ****
  
    (setq c-state-point-min (point-min)))
  
! (defun c-append-lower-brace-pair-to-state-cache (from &optional upper-lim)
    ;; If there is a brace pair preceding FROM in the buffer, at the same level
    ;; of nesting (not necessarily immediately preceding), push a cons onto
    ;; `c-state-cache' to represent it.  FROM must not be inside a literal.  If
--- 2592,2598 ----
  
    (setq c-state-point-min (point-min)))
  
! (defun c-append-lower-brace-pair-to-state-cache (from here &optional upper-lim)
    ;; If there is a brace pair preceding FROM in the buffer, at the same level
    ;; of nesting (not necessarily immediately preceding), push a cons onto
    ;; `c-state-cache' to represent it.  FROM must not be inside a literal.  If
***************
*** 2654,2661 ****
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
!       (let* ((here (point-max))
! 	     new-cons
  	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
  	     (macro-start-or-from
  	      (progn (goto-char from)
--- 2616,2622 ----
    ;; reduce the time wasted in repeated fruitless searches in brace deserts.
    (save-excursion
      (save-restriction
!       (let* (new-cons
  	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
  	     (macro-start-or-from
  	      (progn (goto-char from)
***************
*** 2690,2696 ****
  		;; search bound, even though the algorithm below would skip
  		;; over the new paren pair.
  		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
- 	    (widen)
  	    (narrow-to-region
  		(cond
  		 ((and desert-lim cache-lim)
--- 2651,2656 ----
***************
*** 2698,2704 ****
  		 (desert-lim)
  		 (cache-lim)
  		 ((point-min)))
! 		(point-max)))
  
  	  ;; In the next pair of nested loops, the inner one moves back past a
  	  ;; pair of (mis-)matching parens or brackets; the outer one moves
--- 2658,2666 ----
  		 (desert-lim)
  		 (cache-lim)
  		 ((point-min)))
! 		;; The top limit is EOB to ensure that `bra' is inside the
! 		;; accessible part of the buffer at the next scan operation.
! 		(1+ (buffer-size))))
  
  	  ;; In the next pair of nested loops, the inner one moves back past a
  	  ;; pair of (mis-)matching parens or brackets; the outer one moves
***************
*** 2765,2789 ****
  		      (if (consp (car c-state-cache))
  			  (cdr c-state-cache)
  			c-state-cache)))
! 	;; N.B.  This defsubst codes one method for the simple, normal case,
  	;; and a more sophisticated, slower way for the general case.  Don't
  	;; eliminate this defsubst - it's a speed optimization.
! 	(c-append-lower-brace-pair-to-state-cache (1- bra+1)))))
  
! (defun c-append-to-state-cache (from)
!   ;; Scan the buffer from FROM to (point-max), adding elements into
!   ;; `c-state-cache' for braces etc.  Return a candidate for
!   ;; `c-state-cache-good-pos'.
    ;;
    ;; FROM must be after the latest brace/paren/bracket in `c-state-cache', if
    ;; any.  Typically, it is immediately after it.  It must not be inside a
    ;; literal.
!   (let ((here-bol (c-point 'bol (point-max)))
  	(macro-start-or-here
! 	 (save-excursion (goto-char (point-max))
  			 (if (c-beginning-of-macro)
  			     (point)
! 			   (point-max))))
  	pa+1		      ; pos just after an opening PAren (or brace).
  	(ren+1 from)	      ; usually a pos just after an closing paREN etc.
  			      ; Is actually the pos. to scan for a (/{/[ from,
--- 2727,2750 ----
  		      (if (consp (car c-state-cache))
  			  (cdr c-state-cache)
  			c-state-cache)))
! 	;; N.B.	 This defsubst codes one method for the simple, normal case,
  	;; and a more sophisticated, slower way for the general case.  Don't
  	;; eliminate this defsubst - it's a speed optimization.
! 	(c-append-lower-brace-pair-to-state-cache (1- bra+1) (point-max)))))
  
! (defun c-append-to-state-cache (from here)
!   ;; Scan the buffer from FROM to HERE, adding elements into `c-state-cache'
!   ;; for braces etc.  Return a candidate for `c-state-cache-good-pos'.
    ;;
    ;; FROM must be after the latest brace/paren/bracket in `c-state-cache', if
    ;; any.  Typically, it is immediately after it.  It must not be inside a
    ;; literal.
!   (let ((here-bol (c-point 'bol here))
  	(macro-start-or-here
! 	 (save-excursion (goto-char here)
  			 (if (c-beginning-of-macro)
  			     (point)
! 			   here)))
  	pa+1		      ; pos just after an opening PAren (or brace).
  	(ren+1 from)	      ; usually a pos just after an closing paREN etc.
  			      ; Is actually the pos. to scan for a (/{/[ from,
***************
*** 2796,2870 ****
  	mstart)		      ; start of a macro.
  
      (save-excursion
!       ;; Each time round the following loop, we enter a successively deeper
!       ;; level of brace/paren nesting.  (Except sometimes we "continue at
!       ;; the existing level".)  `pa+1' is a pos inside an opening
!       ;; brace/paren/bracket, usually just after it.
!       (while
! 	  (progn
! 	    ;; Each time round the next loop moves forward over an opening then
! 	    ;; a closing brace/bracket/paren.  This loop is white hot, so it
! 	    ;; plays ugly tricks to go fast.  DON'T PUT ANYTHING INTO THIS
! 	    ;; LOOP WHICH ISN'T ABSOLUTELY NECESSARY!!!  It terminates when a
! 	    ;; call of `scan-lists' signals an error, which happens when there
! 	    ;; are no more b/b/p's to scan.
! 	    (c-safe
! 	      (while t
! 		(setq pa+1 (scan-lists ren+1 1 -1) ; Into (/{/[; might signal
! 		      paren+1s (cons pa+1 paren+1s))
! 		(setq ren+1 (scan-lists pa+1 1 1)) ; Out of )/}/]; might signal
! 		(if (and (eq (char-before pa+1) ?{)) ; Check for a macro later.
! 		    (setq bra+1 pa+1))
! 		(setcar paren+1s ren+1)))
  
! 	    (if (and pa+1 (> pa+1 ren+1))
! 		;; We've just entered a deeper nesting level.
! 		(progn
! 		  ;; Insert the brace pair (if present) and the single open
! 		  ;; paren/brace/bracket into `c-state-cache' It cannot be
! 		  ;; inside a macro, except one around point, because of what
! 		  ;; `c-neutralize-syntax-in-CPP' has done.
! 		  (c-state-push-any-brace-pair bra+1 macro-start-or-here)
! 		  ;; Insert the opening brace/bracket/paren position.
! 		  (setq c-state-cache (cons (1- pa+1) c-state-cache))
! 		  ;; Clear admin stuff for the next more nested part of the scan.
! 		  (setq ren+1 pa+1  pa+1 nil  bra+1 nil  bra+1s nil)
! 		  t)			; Carry on the loop
! 
! 	      ;; All open p/b/b's at this nesting level, if any, have probably
! 	      ;; been closed by matching/mismatching ones.  We're probably
! 	      ;; finished - we just need to check for having found an
! 	      ;; unmatched )/}/], which we ignore.  Such a )/}/] can't be in a
! 	      ;; macro, due the action of `c-neutralize-syntax-in-CPP'.
! 	      (c-safe (setq ren+1 (scan-lists ren+1 1 1)))))) ; acts as loop control.
! 
!       ;; Record the final, innermost, brace-pair if there is one.
!       (c-state-push-any-brace-pair bra+1 macro-start-or-here)
! 
!       ;; Determine a good pos
!       (while (and (setq paren+1 (car paren+1s))
! 		  (> (if (> paren+1 macro-start-or-here)
! 			 paren+1
! 		       (goto-char paren+1)
! 		       (setq mstart (and (c-beginning-of-macro)
! 					 (point)))
! 		       (or mstart paren+1))
! 		     here-bol))
! 	(setq paren+1s (cdr paren+1s)))
!       (cond
!        ((and paren+1 mstart)
! 	(min paren+1 mstart))
!        (paren+1)
!        (t from)))))
  
! (defun c-remove-stale-state-cache (start-point pps-point)
    ;; Remove stale entries from the `c-cache-state', i.e. those which will
!   ;; not be in it when it is amended for position (point-max).
!   ;; Additionally, the "outermost" open-brace entry before (point-max)
!   ;; will be converted to a cons if the matching close-brace is scanned.
    ;;
    ;; START-POINT is a "maximal" "safe position" - there must be no open
!   ;; parens/braces/brackets between START-POINT and (point-max).
    ;;
    ;; As a second thing, calculate the result of parse-partial-sexp at
    ;; PPS-POINT, w.r.t. START-POINT.  The motivation here is that
--- 2757,2833 ----
  	mstart)		      ; start of a macro.
  
      (save-excursion
!       (save-restriction
! 	(narrow-to-region (point-min) here)
! 	;; Each time round the following loop, we enter a successively deeper
! 	;; level of brace/paren nesting.  (Except sometimes we "continue at
! 	;; the existing level".)  `pa+1' is a pos inside an opening
! 	;; brace/paren/bracket, usually just after it.
! 	(while
! 	    (progn
! 	      ;; Each time round the next loop moves forward over an opening then
! 	      ;; a closing brace/bracket/paren.  This loop is white hot, so it
! 	      ;; plays ugly tricks to go fast.  DON'T PUT ANYTHING INTO THIS
! 	      ;; LOOP WHICH ISN'T ABSOLUTELY NECESSARY!!!  It terminates when a
! 	      ;; call of `scan-lists' signals an error, which happens when there
! 	      ;; are no more b/b/p's to scan.
! 	      (c-safe
! 		(while t
! 		  (setq pa+1 (scan-lists ren+1 1 -1) ; Into (/{/[; might signal
! 			paren+1s (cons pa+1 paren+1s))
! 		  (setq ren+1 (scan-lists pa+1 1 1)) ; Out of )/}/]; might signal
! 		  (if (and (eq (char-before pa+1) ?{)) ; Check for a macro later.
! 		      (setq bra+1 pa+1))
! 		  (setcar paren+1s ren+1)))
  
! 	      (if (and pa+1 (> pa+1 ren+1))
! 		  ;; We've just entered a deeper nesting level.
! 		  (progn
! 		    ;; Insert the brace pair (if present) and the single open
! 		    ;; paren/brace/bracket into `c-state-cache' It cannot be
! 		    ;; inside a macro, except one around point, because of what
! 		    ;; `c-neutralize-syntax-in-CPP' has done.
! 		    (c-state-push-any-brace-pair bra+1 macro-start-or-here)
! 		    ;; Insert the opening brace/bracket/paren position.
! 		    (setq c-state-cache (cons (1- pa+1) c-state-cache))
! 		    ;; Clear admin stuff for the next more nested part of the scan.
! 		    (setq ren+1 pa+1  pa+1 nil  bra+1 nil  bra+1s nil)
! 		    t)			; Carry on the loop
! 
! 		;; All open p/b/b's at this nesting level, if any, have probably
! 		;; been closed by matching/mismatching ones.  We're probably
! 		;; finished - we just need to check for having found an
! 		;; unmatched )/}/], which we ignore.  Such a )/}/] can't be in a
! 		;; macro, due the action of `c-neutralize-syntax-in-CPP'.
! 		(c-safe (setq ren+1 (scan-lists ren+1 1 1)))))) ; acts as loop control.
! 
! 	;; Record the final, innermost, brace-pair if there is one.
! 	(c-state-push-any-brace-pair bra+1 macro-start-or-here)
! 
! 	;; Determine a good pos
! 	(while (and (setq paren+1 (car paren+1s))
! 		    (> (if (> paren+1 macro-start-or-here)
! 			   paren+1
! 			 (goto-char paren+1)
! 			 (setq mstart (and (c-beginning-of-macro)
! 					   (point)))
! 			 (or mstart paren+1))
! 		       here-bol))
! 	  (setq paren+1s (cdr paren+1s)))
! 	(cond
! 	 ((and paren+1 mstart)
! 	  (min paren+1 mstart))
! 	 (paren+1)
! 	 (t from))))))
  
! (defun c-remove-stale-state-cache (start-point here pps-point)
    ;; Remove stale entries from the `c-cache-state', i.e. those which will
!   ;; not be in it when it is amended for position HERE.  Additionally, the
!   ;; "outermost" open-brace entry before HERE will be converted to a cons if
!   ;; the matching close-brace is scanned.
    ;;
    ;; START-POINT is a "maximal" "safe position" - there must be no open
!   ;; parens/braces/brackets between START-POINT and HERE.
    ;;
    ;; As a second thing, calculate the result of parse-partial-sexp at
    ;; PPS-POINT, w.r.t. START-POINT.  The motivation here is that
***************
*** 2881,2903 ****
    ;;   last element to be removed from `c-state-cache', when that elt is a
    ;;   cons, otherwise nil.
    ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
!   (save-restriction
!     (narrow-to-region 1 (point-max))
!     (save-excursion
!       (let* ((in-macro-start   ; start of macro containing (point-max) or nil.
  	      (save-excursion
! 		(goto-char (point-max))
  		(and (c-beginning-of-macro)
  		     (point))))
  	     (start-point-actual-macro-start ; Start of macro containing
  					     ; start-point or nil
! 	      (and (< start-point (point-max))
  		   (save-excursion
  		     (goto-char start-point)
  		     (and (c-beginning-of-macro)
  			  (point)))))
  	     (start-point-actual-macro-end ; End of this macro, (maybe
! 					; (point-max)), or nil.
  	      (and start-point-actual-macro-start
  		   (save-excursion
  		     (goto-char start-point-actual-macro-start)
--- 2844,2866 ----
    ;;   last element to be removed from `c-state-cache', when that elt is a
    ;;   cons, otherwise nil.
    ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
!   (save-excursion
!     (save-restriction
!       (narrow-to-region 1 (point-max))
!       (let* ((in-macro-start   ; start of macro containing HERE or nil.
  	      (save-excursion
! 		(goto-char here)
  		(and (c-beginning-of-macro)
  		     (point))))
  	     (start-point-actual-macro-start ; Start of macro containing
  					     ; start-point or nil
! 	      (and (< start-point here)
  		   (save-excursion
  		     (goto-char start-point)
  		     (and (c-beginning-of-macro)
  			  (point)))))
  	     (start-point-actual-macro-end ; End of this macro, (maybe
! 					; HERE), or nil.
  	      (and start-point-actual-macro-start
  		   (save-excursion
  		     (goto-char start-point-actual-macro-start)
***************
*** 2909,2922 ****
  	     scan-back-pos
  	     pair-beg pps-point-state target-depth)
  
! 	;; Remove entries beyond (point-max).  Also remove any entries inside
! 	;; a macro, unless (point-max) is in the same macro.
  	(setq upper-lim
  	      (if (or (null c-state-old-cpp-beg)
! 		      (and (> (point-max) c-state-old-cpp-beg)
! 			   (< (point-max) c-state-old-cpp-end)))
! 		  (point-max)
! 		(min (point-max) c-state-old-cpp-beg)))
  	(while (and c-state-cache (>= (c-state-cache-top-lparen) upper-lim))
  	  (setq scan-back-pos (car-safe (car c-state-cache)))
  	  (setq c-state-cache (cdr c-state-cache)))
--- 2872,2885 ----
  	     scan-back-pos
  	     pair-beg pps-point-state target-depth)
  
! 	;; Remove entries beyond HERE.  Also remove any entries inside
! 	;; a macro, unless HERE is in the same macro.
  	(setq upper-lim
  	      (if (or (null c-state-old-cpp-beg)
! 		      (and (> here c-state-old-cpp-beg)
! 			   (< here c-state-old-cpp-end)))
! 		  here
! 		(min here c-state-old-cpp-beg)))
  	(while (and c-state-cache (>= (c-state-cache-top-lparen) upper-lim))
  	  (setq scan-back-pos (car-safe (car c-state-cache)))
  	  (setq c-state-cache (cdr c-state-cache)))
***************
*** 2934,2940 ****
  	;; time round; the corresponding elements in `c-state-cache' are
  	;; removed.  `pos' is just after the brace-pair or the open paren at
  	;; (car c-state-cache).	 There can be no open parens/braces/brackets
! 	;; between `start-point'/`start-point-actual-macro-start' and (point-max),
  	;; due to the interface spec to this function.
  	(setq pos (if (and start-point-actual-macro-end
  			   (not (eq start-point-actual-macro-start
--- 2897,2903 ----
  	;; time round; the corresponding elements in `c-state-cache' are
  	;; removed.  `pos' is just after the brace-pair or the open paren at
  	;; (car c-state-cache).	 There can be no open parens/braces/brackets
! 	;; between `start-point'/`start-point-actual-macro-start' and HERE,
  	;; due to the interface spec to this function.
  	(setq pos (if (and start-point-actual-macro-end
  			   (not (eq start-point-actual-macro-start
***************
*** 2944,2950 ****
  		    start-point))
  	(goto-char pos)
  	(while (and c-state-cache
! 		    (< (point) (point-max)))
  	  (cond
  	   ((null pps-state)		; first time through
  	    (setq target-depth -1))
--- 2907,2915 ----
  		    start-point))
  	(goto-char pos)
  	(while (and c-state-cache
! 		    (or (numberp (car c-state-cache)) ; Have we a { at all? 
! 			(cdr c-state-cache))
! 		    (< (point) here))
  	  (cond
  	   ((null pps-state)		; first time through
  	    (setq target-depth -1))
***************
*** 2956,2962 ****
  	  ;; Scan!
  	  (setq pps-state
  		(parse-partial-sexp
! 		 (point) (if (< (point) pps-point) pps-point (point-max))
  		 target-depth
  		 nil pps-state))
  
--- 2921,2927 ----
  	  ;; Scan!
  	  (setq pps-state
  		(parse-partial-sexp
! 		 (point) (if (< (point) pps-point) pps-point here)
  		 target-depth
  		 nil pps-state))
  
***************
*** 3209,3215 ****
        ;; Do we need to add in an earlier brace pair, having lopped one off?
        (if (and dropped-cons
  	       (< too-high-pa (+ here c-state-cache-too-far)))
! 	  (c-append-lower-brace-pair-to-state-cache too-high-pa here-bol))
        (setq c-state-cache-good-pos (or (c-state-cache-after-top-paren)
  				       (c-state-get-min-scan-pos)))))
  
--- 3174,3180 ----
        ;; Do we need to add in an earlier brace pair, having lopped one off?
        (if (and dropped-cons
  	       (< too-high-pa (+ here c-state-cache-too-far)))
! 	  (c-append-lower-brace-pair-to-state-cache too-high-pa here here-bol))
        (setq c-state-cache-good-pos (or (c-state-cache-after-top-paren)
  				       (c-state-get-min-scan-pos)))))
  
***************
*** 3285,3331 ****
  	  strategy (car res)
  	  start-point (cadr res))
  
-     (when (eq strategy 'BOD)
-       (setq c-state-cache nil
- 	    c-state-cache-good-pos start-point))
- 
      ;; SCAN!
!     (save-restriction
!       (cond
!        ((memq strategy '(forward BOD))
! 	(narrow-to-region (point-min) here)
! 	(setq res (c-remove-stale-state-cache start-point here-bopl))
! 	(setq cache-pos (car res)
! 	      scan-backward-pos (cadr res)
! 	      bopl-state (car (cddr res))) ; will be nil if (< here-bopl
  					; start-point)
! 	(if scan-backward-pos
! 	    (c-append-lower-brace-pair-to-state-cache scan-backward-pos))
! 	(setq good-pos
! 	      (c-append-to-state-cache cache-pos))
! 	(setq c-state-cache-good-pos
! 	      (if (and bopl-state
! 		       (< good-pos (- here c-state-cache-too-far)))
! 		  (c-state-cache-non-literal-place here-bopl bopl-state)
! 		good-pos)))
! 
!        ((eq strategy 'backward)
! 	(setq res (c-remove-stale-state-cache-backwards here)
! 	      good-pos (car res)
! 	      scan-backward-pos (cadr res)
! 	      scan-forward-p (car (cddr res)))
! 	(if scan-backward-pos
! 	    (c-append-lower-brace-pair-to-state-cache
! 	     scan-backward-pos))
! 	(setq c-state-cache-good-pos
! 	      (if scan-forward-p
! 		  (progn (narrow-to-region (point-min) here)
! 			 (c-append-to-state-cache good-pos))
! 		good-pos)))
  
!        (t ; (eq strategy 'IN-LIT)
! 	(setq c-state-cache nil
! 	      c-state-cache-good-pos nil)))))
  
    c-state-cache)
  
--- 3250,3288 ----
  	  strategy (car res)
  	  start-point (cadr res))
  
      ;; SCAN!
!     (cond
!      ((eq strategy 'forward)
!       (setq res (c-remove-stale-state-cache start-point here here-bopl))
!       (setq cache-pos (car res)
! 	    scan-backward-pos (cadr res)
! 	    bopl-state (car (cddr res))) ; will be nil if (< here-bopl
  					; start-point)
!       (if scan-backward-pos
! 	  (c-append-lower-brace-pair-to-state-cache scan-backward-pos here))
!       (setq good-pos
! 	    (c-append-to-state-cache cache-pos here))
!       (setq c-state-cache-good-pos
! 	    (if (and bopl-state
! 		     (< good-pos (- here c-state-cache-too-far)))
! 		(c-state-cache-non-literal-place here-bopl bopl-state)
! 	      good-pos)))
! 
!      ((eq strategy 'backward)
!       (setq res (c-remove-stale-state-cache-backwards here)
! 	    good-pos (car res)
! 	    scan-backward-pos (cadr res)
! 	    scan-forward-p (car (cddr res)))
!       (if scan-backward-pos
! 	  (c-append-lower-brace-pair-to-state-cache scan-backward-pos here))
!       (setq c-state-cache-good-pos
! 	    (if scan-forward-p
! 		(c-append-to-state-cache good-pos here)
! 	      good-pos)))
  
!      (t					; (eq strategy 'IN-LIT)
!       (setq c-state-cache nil
! 	    c-state-cache-good-pos nil))))
  
    c-state-cache)
  
***************
*** 8559,8566 ****
     ))
  
  (defun c-looking-at-special-brace-list (&optional lim)
!   ;; If we're looking at the start of a pike-style list, ie `({ })',
!   ;; `([ ])', `(< >)' etc, a cons of a cons of its starting and ending
    ;; positions and its entry in c-special-brace-lists is returned, nil
    ;; otherwise.  The ending position is nil if the list is still open.
    ;; LIM is the limit for forward search.  The point may either be at
--- 8516,8523 ----
     ))
  
  (defun c-looking-at-special-brace-list (&optional lim)
!   ;; If we're looking at the start of a pike-style list, ie `({ })',
!   ;; `([ ])', `(< >)' etc, a cons of a cons of its starting and ending
    ;; positions and its entry in c-special-brace-lists is returned, nil
    ;; otherwise.  The ending position is nil if the list is still open.
    ;; LIM is the limit for forward search.  The point may either be at


> -- 
> Michael Welsh Duggan
> (mwd@cert.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Alan Mackenzie <acm@muc.de> writes:

> Hi, Michael.

I've been running with this for a few days, and have yet to run into a
problem.  I'll let you know when/if I do in the future.


> On Wed, Jan 23, 2013 at 10:39:34AM -0500, Michael Welsh Duggan wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>> > However, this isn't the end of the story - 
>
>> > (i) Start your test case in rwtransfer.c, and M-v until the top of the
>> >   file.  (This shouldn't display any inconsistency messages)
>> > (ii) M-x goto-char <ret> 20001 <ret>.  This should go to EOL 671, just
>> >   after a "}".
>> > (iii) Type <space> }.  (Two characters)
>> > (iv) C-v, possibly twice.
>
>> > This displays a message something like:
>> >     c-parse-state inconsistency at 21070: using cache: ((20458 . 20935)),
>> >     from scratch: ((20838 . 20877))
>> > , together with a state dump.
>
>> > This bug isn't a new one, but I've just come across it.  I'll be working
>> > on it in the meantime.
>
>> Oh, good.  I haven't reported any "type stuff" errors I have been
>> getting because they are harder to reproduce.  (Unless it is triggered
>> the first time you've typed in the buffer, it's hard to replicate the
>> exact scenario.)  I usually wait until you've fixed the last thing I
>> sent in before attempting to report another, just in case the thing you
>> are fixing is related.
>
> OK, I think I've fixed that above bug.  I've also done a moderate amount
> of refactoring to simplify things for the next bug, if any.
>
> I've removed the scanning strategy "start at the 2nd previous column-0
> beginning-of-defun", because it was returning false results, even in C
> Mode.  I'd already disabled it for C++ Mode (because hackers frequently
> put things inside namespaces at column 0).  One consequence is that
> running with `c-toggle-parse-state-debug' is somewhat slower.  Sorry for
> that.
>
> Could you try this new version, please.  The patch is based on the
> emacs-24 branch of the savannah bzr repository.  This should be identical
> to the trunk.
>
>
>
> === modified file 'lisp/progmodes/cc-engine.el'
> *** lisp/progmodes/cc-engine.el	2013-01-23 18:58:11 +0000
> --- lisp/progmodes/cc-engine.el	2013-01-29 11:03:35 +0000
> ***************
> *** 2477,2496 ****
>   
>   ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>   ;; Defuns which analyze the buffer, yet don't change `c-state-cache'.
> - (defun c-get-fallback-scan-pos (here)
> -   ;; Return a start position for building `c-state-cache' from
> -   ;; scratch.  This will be at the top level, 2 defuns back.
> -   (save-excursion
> -     ;; Go back 2 bods, but ignore any bogus positions returned by
> -     ;; beginning-of-defun (i.e. open paren in column zero).
> -     (goto-char here)
> -     (let ((cnt 2))
> -       (while (not (or (bobp) (zerop cnt)))
> - 	(c-beginning-of-defun-1)	; Pure elisp BOD.
> - 	(if (eq (char-after) ?\{)
> - 	    (setq cnt (1- cnt)))))
> -     (point)))
> - 
>   (defun c-state-balance-parens-backwards (here- here+ top)
>     ;; Return the position of the opening paren/brace/bracket before HERE- which
>     ;; matches the outermost close p/b/b between HERE+ and TOP.  Except when
> --- 2477,2482 ----
> ***************
> *** 2548,2594 ****
>     ;; o - ('forward START-POINT) - scan forward from START-POINT,
>     ;;	 which is not less than the highest position in `c-state-cache' below here.
>     ;; o - ('backward nil) - scan backwards (from HERE).
> -   ;; o - ('BOD START-POINT) - scan forwards from START-POINT, which is at the
> -   ;;   top level.
>     ;; o - ('IN-LIT nil) - point is inside the literal containing point-min.
>     (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
> ! 	BOD-pos		    ; position of 2nd BOD before HERE.
> ! 	strategy	    ; 'forward, 'backward, 'BOD, or 'IN-LIT.
> ! 	start-point
> ! 	how-far)			; putative scanning distance.
>       (setq good-pos (or good-pos (c-state-get-min-scan-pos)))
>       (cond
>        ((< here (c-state-get-min-scan-pos))
> !       (setq strategy 'IN-LIT
> ! 	    start-point nil
> ! 	    cache-pos nil
> ! 	    how-far 0))
>        ((<= good-pos here)
>         (setq strategy 'forward
> ! 	    start-point (max good-pos cache-pos)
> ! 	    how-far (- here start-point)))
>        ((< (- good-pos here) (- here cache-pos)) ; FIXME!!! ; apply some sort of weighting.
> !       (setq strategy 'backward
> ! 	    how-far (- good-pos here)))
>        (t
>         (setq strategy 'forward
> ! 	      how-far (- here cache-pos)
> ! 	      start-point cache-pos)))
> ! 
> !     ;; Might we be better off starting from the top level, two defuns back,
> !     ;; instead?  This heuristic no longer works well in C++, where
> !     ;; declarations inside namespace brace blocks are frequently placed at
> !     ;; column zero.
> !     (when (and (not (c-major-mode-is 'c++-mode))
> ! 	       (> how-far c-state-cache-too-far))
> !       (setq BOD-pos (c-get-fallback-scan-pos here)) ; somewhat EXPENSIVE!!!
> !       (if (< (- here BOD-pos) how-far)
> ! 	  (setq strategy 'BOD
> ! 		start-point BOD-pos)))
> ! 
> !     (list
> !      strategy
> !      (and (memq strategy '(forward BOD)) start-point))))
>   
>   
>   ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> --- 2534,2556 ----
>     ;; o - ('forward START-POINT) - scan forward from START-POINT,
>     ;;	 which is not less than the highest position in `c-state-cache' below here.
>     ;; o - ('backward nil) - scan backwards (from HERE).
>     ;; o - ('IN-LIT nil) - point is inside the literal containing point-min.
>     (let ((cache-pos (c-get-cache-scan-pos here))	; highest position below HERE in cache (or 1)
> ! 	strategy	    ; 'forward, 'backward, or 'IN-LIT.
> ! 	start-point)
>       (setq good-pos (or good-pos (c-state-get-min-scan-pos)))
>       (cond
>        ((< here (c-state-get-min-scan-pos))
> !       (setq strategy 'IN-LIT))
>        ((<= good-pos here)
>         (setq strategy 'forward
> ! 	    start-point (max good-pos cache-pos)))
>        ((< (- good-pos here) (- here cache-pos)) ; FIXME!!! ; apply some sort of weighting.
> !       (setq strategy 'backward))
>        (t
>         (setq strategy 'forward
> ! 	    start-point cache-pos)))
> !     (list strategy (and (eq strategy 'forward) start-point))))
>   
>   
>   ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> ***************
> *** 2630,2636 ****
>   
>     (setq c-state-point-min (point-min)))
>   
> ! (defun c-append-lower-brace-pair-to-state-cache (from &optional upper-lim)
>     ;; If there is a brace pair preceding FROM in the buffer, at the same level
>     ;; of nesting (not necessarily immediately preceding), push a cons onto
>     ;; `c-state-cache' to represent it.  FROM must not be inside a literal.  If
> --- 2592,2598 ----
>   
>     (setq c-state-point-min (point-min)))
>   
> ! (defun c-append-lower-brace-pair-to-state-cache (from here &optional upper-lim)
>     ;; If there is a brace pair preceding FROM in the buffer, at the same level
>     ;; of nesting (not necessarily immediately preceding), push a cons onto
>     ;; `c-state-cache' to represent it.  FROM must not be inside a literal.  If
> ***************
> *** 2654,2661 ****
>     ;; reduce the time wasted in repeated fruitless searches in brace deserts.
>     (save-excursion
>       (save-restriction
> !       (let* ((here (point-max))
> ! 	     new-cons
>   	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
>   	     (macro-start-or-from
>   	      (progn (goto-char from)
> --- 2616,2622 ----
>     ;; reduce the time wasted in repeated fruitless searches in brace deserts.
>     (save-excursion
>       (save-restriction
> !       (let* (new-cons
>   	     (cache-pos (c-state-cache-top-lparen)) ; might be nil.
>   	     (macro-start-or-from
>   	      (progn (goto-char from)
> ***************
> *** 2690,2696 ****
>   		;; search bound, even though the algorithm below would skip
>   		;; over the new paren pair.
>   		(cache-lim (and cache-pos (< cache-pos from) cache-pos)))
> - 	    (widen)
>   	    (narrow-to-region
>   		(cond
>   		 ((and desert-lim cache-lim)
> --- 2651,2656 ----
> ***************
> *** 2698,2704 ****
>   		 (desert-lim)
>   		 (cache-lim)
>   		 ((point-min)))
> ! 		(point-max)))
>   
>   	  ;; In the next pair of nested loops, the inner one moves back past a
>   	  ;; pair of (mis-)matching parens or brackets; the outer one moves
> --- 2658,2666 ----
>   		 (desert-lim)
>   		 (cache-lim)
>   		 ((point-min)))
> ! 		;; The top limit is EOB to ensure that `bra' is inside the
> ! 		;; accessible part of the buffer at the next scan operation.
> ! 		(1+ (buffer-size))))
>   
>   	  ;; In the next pair of nested loops, the inner one moves back past a
>   	  ;; pair of (mis-)matching parens or brackets; the outer one moves
> ***************
> *** 2765,2789 ****
>   		      (if (consp (car c-state-cache))
>   			  (cdr c-state-cache)
>   			c-state-cache)))
> ! 	;; N.B.  This defsubst codes one method for the simple, normal case,
>   	;; and a more sophisticated, slower way for the general case.  Don't
>   	;; eliminate this defsubst - it's a speed optimization.
> ! 	(c-append-lower-brace-pair-to-state-cache (1- bra+1)))))
>   
> ! (defun c-append-to-state-cache (from)
> !   ;; Scan the buffer from FROM to (point-max), adding elements into
> !   ;; `c-state-cache' for braces etc.  Return a candidate for
> !   ;; `c-state-cache-good-pos'.
>     ;;
>     ;; FROM must be after the latest brace/paren/bracket in `c-state-cache', if
>     ;; any.  Typically, it is immediately after it.  It must not be inside a
>     ;; literal.
> !   (let ((here-bol (c-point 'bol (point-max)))
>   	(macro-start-or-here
> ! 	 (save-excursion (goto-char (point-max))
>   			 (if (c-beginning-of-macro)
>   			     (point)
> ! 			   (point-max))))
>   	pa+1		      ; pos just after an opening PAren (or brace).
>   	(ren+1 from)	      ; usually a pos just after an closing paREN etc.
>   			      ; Is actually the pos. to scan for a (/{/[ from,
> --- 2727,2750 ----
>   		      (if (consp (car c-state-cache))
>   			  (cdr c-state-cache)
>   			c-state-cache)))
> ! 	;; N.B.	 This defsubst codes one method for the simple, normal case,
>   	;; and a more sophisticated, slower way for the general case.  Don't
>   	;; eliminate this defsubst - it's a speed optimization.
> ! 	(c-append-lower-brace-pair-to-state-cache (1- bra+1) (point-max)))))
>   
> ! (defun c-append-to-state-cache (from here)
> !   ;; Scan the buffer from FROM to HERE, adding elements into `c-state-cache'
> !   ;; for braces etc.  Return a candidate for `c-state-cache-good-pos'.
>     ;;
>     ;; FROM must be after the latest brace/paren/bracket in `c-state-cache', if
>     ;; any.  Typically, it is immediately after it.  It must not be inside a
>     ;; literal.
> !   (let ((here-bol (c-point 'bol here))
>   	(macro-start-or-here
> ! 	 (save-excursion (goto-char here)
>   			 (if (c-beginning-of-macro)
>   			     (point)
> ! 			   here)))
>   	pa+1		      ; pos just after an opening PAren (or brace).
>   	(ren+1 from)	      ; usually a pos just after an closing paREN etc.
>   			      ; Is actually the pos. to scan for a (/{/[ from,
> ***************
> *** 2796,2870 ****
>   	mstart)		      ; start of a macro.
>   
>       (save-excursion
> !       ;; Each time round the following loop, we enter a successively deeper
> !       ;; level of brace/paren nesting.  (Except sometimes we "continue at
> !       ;; the existing level".)  `pa+1' is a pos inside an opening
> !       ;; brace/paren/bracket, usually just after it.
> !       (while
> ! 	  (progn
> ! 	    ;; Each time round the next loop moves forward over an opening then
> ! 	    ;; a closing brace/bracket/paren.  This loop is white hot, so it
> ! 	    ;; plays ugly tricks to go fast.  DON'T PUT ANYTHING INTO THIS
> ! 	    ;; LOOP WHICH ISN'T ABSOLUTELY NECESSARY!!!  It terminates when a
> ! 	    ;; call of `scan-lists' signals an error, which happens when there
> ! 	    ;; are no more b/b/p's to scan.
> ! 	    (c-safe
> ! 	      (while t
> ! 		(setq pa+1 (scan-lists ren+1 1 -1) ; Into (/{/[; might signal
> ! 		      paren+1s (cons pa+1 paren+1s))
> ! 		(setq ren+1 (scan-lists pa+1 1 1)) ; Out of )/}/]; might signal
> ! 		(if (and (eq (char-before pa+1) ?{)) ; Check for a macro later.
> ! 		    (setq bra+1 pa+1))
> ! 		(setcar paren+1s ren+1)))
>   
> ! 	    (if (and pa+1 (> pa+1 ren+1))
> ! 		;; We've just entered a deeper nesting level.
> ! 		(progn
> ! 		  ;; Insert the brace pair (if present) and the single open
> ! 		  ;; paren/brace/bracket into `c-state-cache' It cannot be
> ! 		  ;; inside a macro, except one around point, because of what
> ! 		  ;; `c-neutralize-syntax-in-CPP' has done.
> ! 		  (c-state-push-any-brace-pair bra+1 macro-start-or-here)
> ! 		  ;; Insert the opening brace/bracket/paren position.
> ! 		  (setq c-state-cache (cons (1- pa+1) c-state-cache))
> ! 		  ;; Clear admin stuff for the next more nested part of the scan.
> ! 		  (setq ren+1 pa+1  pa+1 nil  bra+1 nil  bra+1s nil)
> ! 		  t)			; Carry on the loop
> ! 
> ! 	      ;; All open p/b/b's at this nesting level, if any, have probably
> ! 	      ;; been closed by matching/mismatching ones.  We're probably
> ! 	      ;; finished - we just need to check for having found an
> ! 	      ;; unmatched )/}/], which we ignore.  Such a )/}/] can't be in a
> ! 	      ;; macro, due the action of `c-neutralize-syntax-in-CPP'.
> ! 	      (c-safe (setq ren+1 (scan-lists ren+1 1 1)))))) ; acts as loop control.
> ! 
> !       ;; Record the final, innermost, brace-pair if there is one.
> !       (c-state-push-any-brace-pair bra+1 macro-start-or-here)
> ! 
> !       ;; Determine a good pos
> !       (while (and (setq paren+1 (car paren+1s))
> ! 		  (> (if (> paren+1 macro-start-or-here)
> ! 			 paren+1
> ! 		       (goto-char paren+1)
> ! 		       (setq mstart (and (c-beginning-of-macro)
> ! 					 (point)))
> ! 		       (or mstart paren+1))
> ! 		     here-bol))
> ! 	(setq paren+1s (cdr paren+1s)))
> !       (cond
> !        ((and paren+1 mstart)
> ! 	(min paren+1 mstart))
> !        (paren+1)
> !        (t from)))))
>   
> ! (defun c-remove-stale-state-cache (start-point pps-point)
>     ;; Remove stale entries from the `c-cache-state', i.e. those which will
> !   ;; not be in it when it is amended for position (point-max).
> !   ;; Additionally, the "outermost" open-brace entry before (point-max)
> !   ;; will be converted to a cons if the matching close-brace is scanned.
>     ;;
>     ;; START-POINT is a "maximal" "safe position" - there must be no open
> !   ;; parens/braces/brackets between START-POINT and (point-max).
>     ;;
>     ;; As a second thing, calculate the result of parse-partial-sexp at
>     ;; PPS-POINT, w.r.t. START-POINT.  The motivation here is that
> --- 2757,2833 ----
>   	mstart)		      ; start of a macro.
>   
>       (save-excursion
> !       (save-restriction
> ! 	(narrow-to-region (point-min) here)
> ! 	;; Each time round the following loop, we enter a successively deeper
> ! 	;; level of brace/paren nesting.  (Except sometimes we "continue at
> ! 	;; the existing level".)  `pa+1' is a pos inside an opening
> ! 	;; brace/paren/bracket, usually just after it.
> ! 	(while
> ! 	    (progn
> ! 	      ;; Each time round the next loop moves forward over an opening then
> ! 	      ;; a closing brace/bracket/paren.  This loop is white hot, so it
> ! 	      ;; plays ugly tricks to go fast.  DON'T PUT ANYTHING INTO THIS
> ! 	      ;; LOOP WHICH ISN'T ABSOLUTELY NECESSARY!!!  It terminates when a
> ! 	      ;; call of `scan-lists' signals an error, which happens when there
> ! 	      ;; are no more b/b/p's to scan.
> ! 	      (c-safe
> ! 		(while t
> ! 		  (setq pa+1 (scan-lists ren+1 1 -1) ; Into (/{/[; might signal
> ! 			paren+1s (cons pa+1 paren+1s))
> ! 		  (setq ren+1 (scan-lists pa+1 1 1)) ; Out of )/}/]; might signal
> ! 		  (if (and (eq (char-before pa+1) ?{)) ; Check for a macro later.
> ! 		      (setq bra+1 pa+1))
> ! 		  (setcar paren+1s ren+1)))
>   
> ! 	      (if (and pa+1 (> pa+1 ren+1))
> ! 		  ;; We've just entered a deeper nesting level.
> ! 		  (progn
> ! 		    ;; Insert the brace pair (if present) and the single open
> ! 		    ;; paren/brace/bracket into `c-state-cache' It cannot be
> ! 		    ;; inside a macro, except one around point, because of what
> ! 		    ;; `c-neutralize-syntax-in-CPP' has done.
> ! 		    (c-state-push-any-brace-pair bra+1 macro-start-or-here)
> ! 		    ;; Insert the opening brace/bracket/paren position.
> ! 		    (setq c-state-cache (cons (1- pa+1) c-state-cache))
> ! 		    ;; Clear admin stuff for the next more nested part of the scan.
> ! 		    (setq ren+1 pa+1  pa+1 nil  bra+1 nil  bra+1s nil)
> ! 		    t)			; Carry on the loop
> ! 
> ! 		;; All open p/b/b's at this nesting level, if any, have probably
> ! 		;; been closed by matching/mismatching ones.  We're probably
> ! 		;; finished - we just need to check for having found an
> ! 		;; unmatched )/}/], which we ignore.  Such a )/}/] can't be in a
> ! 		;; macro, due the action of `c-neutralize-syntax-in-CPP'.
> ! 		(c-safe (setq ren+1 (scan-lists ren+1 1 1)))))) ; acts as loop control.
> ! 
> ! 	;; Record the final, innermost, brace-pair if there is one.
> ! 	(c-state-push-any-brace-pair bra+1 macro-start-or-here)
> ! 
> ! 	;; Determine a good pos
> ! 	(while (and (setq paren+1 (car paren+1s))
> ! 		    (> (if (> paren+1 macro-start-or-here)
> ! 			   paren+1
> ! 			 (goto-char paren+1)
> ! 			 (setq mstart (and (c-beginning-of-macro)
> ! 					   (point)))
> ! 			 (or mstart paren+1))
> ! 		       here-bol))
> ! 	  (setq paren+1s (cdr paren+1s)))
> ! 	(cond
> ! 	 ((and paren+1 mstart)
> ! 	  (min paren+1 mstart))
> ! 	 (paren+1)
> ! 	 (t from))))))
>   
> ! (defun c-remove-stale-state-cache (start-point here pps-point)
>     ;; Remove stale entries from the `c-cache-state', i.e. those which will
> !   ;; not be in it when it is amended for position HERE.  Additionally, the
> !   ;; "outermost" open-brace entry before HERE will be converted to a cons if
> !   ;; the matching close-brace is scanned.
>     ;;
>     ;; START-POINT is a "maximal" "safe position" - there must be no open
> !   ;; parens/braces/brackets between START-POINT and HERE.
>     ;;
>     ;; As a second thing, calculate the result of parse-partial-sexp at
>     ;; PPS-POINT, w.r.t. START-POINT.  The motivation here is that
> ***************
> *** 2881,2903 ****
>     ;;   last element to be removed from `c-state-cache', when that elt is a
>     ;;   cons, otherwise nil.
>     ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
> !   (save-restriction
> !     (narrow-to-region 1 (point-max))
> !     (save-excursion
> !       (let* ((in-macro-start   ; start of macro containing (point-max) or nil.
>   	      (save-excursion
> ! 		(goto-char (point-max))
>   		(and (c-beginning-of-macro)
>   		     (point))))
>   	     (start-point-actual-macro-start ; Start of macro containing
>   					     ; start-point or nil
> ! 	      (and (< start-point (point-max))
>   		   (save-excursion
>   		     (goto-char start-point)
>   		     (and (c-beginning-of-macro)
>   			  (point)))))
>   	     (start-point-actual-macro-end ; End of this macro, (maybe
> ! 					; (point-max)), or nil.
>   	      (and start-point-actual-macro-start
>   		   (save-excursion
>   		     (goto-char start-point-actual-macro-start)
> --- 2844,2866 ----
>     ;;   last element to be removed from `c-state-cache', when that elt is a
>     ;;   cons, otherwise nil.
>     ;; o - PPS-STATE is the parse-partial-sexp state at PPS-POINT.
> !   (save-excursion
> !     (save-restriction
> !       (narrow-to-region 1 (point-max))
> !       (let* ((in-macro-start   ; start of macro containing HERE or nil.
>   	      (save-excursion
> ! 		(goto-char here)
>   		(and (c-beginning-of-macro)
>   		     (point))))
>   	     (start-point-actual-macro-start ; Start of macro containing
>   					     ; start-point or nil
> ! 	      (and (< start-point here)
>   		   (save-excursion
>   		     (goto-char start-point)
>   		     (and (c-beginning-of-macro)
>   			  (point)))))
>   	     (start-point-actual-macro-end ; End of this macro, (maybe
> ! 					; HERE), or nil.
>   	      (and start-point-actual-macro-start
>   		   (save-excursion
>   		     (goto-char start-point-actual-macro-start)
> ***************
> *** 2909,2922 ****
>   	     scan-back-pos
>   	     pair-beg pps-point-state target-depth)
>   
> ! 	;; Remove entries beyond (point-max).  Also remove any entries inside
> ! 	;; a macro, unless (point-max) is in the same macro.
>   	(setq upper-lim
>   	      (if (or (null c-state-old-cpp-beg)
> ! 		      (and (> (point-max) c-state-old-cpp-beg)
> ! 			   (< (point-max) c-state-old-cpp-end)))
> ! 		  (point-max)
> ! 		(min (point-max) c-state-old-cpp-beg)))
>   	(while (and c-state-cache (>= (c-state-cache-top-lparen) upper-lim))
>   	  (setq scan-back-pos (car-safe (car c-state-cache)))
>   	  (setq c-state-cache (cdr c-state-cache)))
> --- 2872,2885 ----
>   	     scan-back-pos
>   	     pair-beg pps-point-state target-depth)
>   
> ! 	;; Remove entries beyond HERE.  Also remove any entries inside
> ! 	;; a macro, unless HERE is in the same macro.
>   	(setq upper-lim
>   	      (if (or (null c-state-old-cpp-beg)
> ! 		      (and (> here c-state-old-cpp-beg)
> ! 			   (< here c-state-old-cpp-end)))
> ! 		  here
> ! 		(min here c-state-old-cpp-beg)))
>   	(while (and c-state-cache (>= (c-state-cache-top-lparen) upper-lim))
>   	  (setq scan-back-pos (car-safe (car c-state-cache)))
>   	  (setq c-state-cache (cdr c-state-cache)))
> ***************
> *** 2934,2940 ****
>   	;; time round; the corresponding elements in `c-state-cache' are
>   	;; removed.  `pos' is just after the brace-pair or the open paren at
>   	;; (car c-state-cache).	 There can be no open parens/braces/brackets
> ! 	;; between `start-point'/`start-point-actual-macro-start' and (point-max),
>   	;; due to the interface spec to this function.
>   	(setq pos (if (and start-point-actual-macro-end
>   			   (not (eq start-point-actual-macro-start
> --- 2897,2903 ----
>   	;; time round; the corresponding elements in `c-state-cache' are
>   	;; removed.  `pos' is just after the brace-pair or the open paren at
>   	;; (car c-state-cache).	 There can be no open parens/braces/brackets
> ! 	;; between `start-point'/`start-point-actual-macro-start' and HERE,
>   	;; due to the interface spec to this function.
>   	(setq pos (if (and start-point-actual-macro-end
>   			   (not (eq start-point-actual-macro-start
> ***************
> *** 2944,2950 ****
>   		    start-point))
>   	(goto-char pos)
>   	(while (and c-state-cache
> ! 		    (< (point) (point-max)))
>   	  (cond
>   	   ((null pps-state)		; first time through
>   	    (setq target-depth -1))
> --- 2907,2915 ----
>   		    start-point))
>   	(goto-char pos)
>   	(while (and c-state-cache
> ! 		    (or (numberp (car c-state-cache)) ; Have we a { at all? 
> ! 			(cdr c-state-cache))
> ! 		    (< (point) here))
>   	  (cond
>   	   ((null pps-state)		; first time through
>   	    (setq target-depth -1))
> ***************
> *** 2956,2962 ****
>   	  ;; Scan!
>   	  (setq pps-state
>   		(parse-partial-sexp
> ! 		 (point) (if (< (point) pps-point) pps-point (point-max))
>   		 target-depth
>   		 nil pps-state))
>   
> --- 2921,2927 ----
>   	  ;; Scan!
>   	  (setq pps-state
>   		(parse-partial-sexp
> ! 		 (point) (if (< (point) pps-point) pps-point here)
>   		 target-depth
>   		 nil pps-state))
>   
> ***************
> *** 3209,3215 ****
>         ;; Do we need to add in an earlier brace pair, having lopped one off?
>         (if (and dropped-cons
>   	       (< too-high-pa (+ here c-state-cache-too-far)))
> ! 	  (c-append-lower-brace-pair-to-state-cache too-high-pa here-bol))
>         (setq c-state-cache-good-pos (or (c-state-cache-after-top-paren)
>   				       (c-state-get-min-scan-pos)))))
>   
> --- 3174,3180 ----
>         ;; Do we need to add in an earlier brace pair, having lopped one off?
>         (if (and dropped-cons
>   	       (< too-high-pa (+ here c-state-cache-too-far)))
> ! 	  (c-append-lower-brace-pair-to-state-cache too-high-pa here here-bol))
>         (setq c-state-cache-good-pos (or (c-state-cache-after-top-paren)
>   				       (c-state-get-min-scan-pos)))))
>   
> ***************
> *** 3285,3331 ****
>   	  strategy (car res)
>   	  start-point (cadr res))
>   
> -     (when (eq strategy 'BOD)
> -       (setq c-state-cache nil
> - 	    c-state-cache-good-pos start-point))
> - 
>       ;; SCAN!
> !     (save-restriction
> !       (cond
> !        ((memq strategy '(forward BOD))
> ! 	(narrow-to-region (point-min) here)
> ! 	(setq res (c-remove-stale-state-cache start-point here-bopl))
> ! 	(setq cache-pos (car res)
> ! 	      scan-backward-pos (cadr res)
> ! 	      bopl-state (car (cddr res))) ; will be nil if (< here-bopl
>   					; start-point)
> ! 	(if scan-backward-pos
> ! 	    (c-append-lower-brace-pair-to-state-cache scan-backward-pos))
> ! 	(setq good-pos
> ! 	      (c-append-to-state-cache cache-pos))
> ! 	(setq c-state-cache-good-pos
> ! 	      (if (and bopl-state
> ! 		       (< good-pos (- here c-state-cache-too-far)))
> ! 		  (c-state-cache-non-literal-place here-bopl bopl-state)
> ! 		good-pos)))
> ! 
> !        ((eq strategy 'backward)
> ! 	(setq res (c-remove-stale-state-cache-backwards here)
> ! 	      good-pos (car res)
> ! 	      scan-backward-pos (cadr res)
> ! 	      scan-forward-p (car (cddr res)))
> ! 	(if scan-backward-pos
> ! 	    (c-append-lower-brace-pair-to-state-cache
> ! 	     scan-backward-pos))
> ! 	(setq c-state-cache-good-pos
> ! 	      (if scan-forward-p
> ! 		  (progn (narrow-to-region (point-min) here)
> ! 			 (c-append-to-state-cache good-pos))
> ! 		good-pos)))
>   
> !        (t ; (eq strategy 'IN-LIT)
> ! 	(setq c-state-cache nil
> ! 	      c-state-cache-good-pos nil)))))
>   
>     c-state-cache)
>   
> --- 3250,3288 ----
>   	  strategy (car res)
>   	  start-point (cadr res))
>   
>       ;; SCAN!
> !     (cond
> !      ((eq strategy 'forward)
> !       (setq res (c-remove-stale-state-cache start-point here here-bopl))
> !       (setq cache-pos (car res)
> ! 	    scan-backward-pos (cadr res)
> ! 	    bopl-state (car (cddr res))) ; will be nil if (< here-bopl
>   					; start-point)
> !       (if scan-backward-pos
> ! 	  (c-append-lower-brace-pair-to-state-cache scan-backward-pos here))
> !       (setq good-pos
> ! 	    (c-append-to-state-cache cache-pos here))
> !       (setq c-state-cache-good-pos
> ! 	    (if (and bopl-state
> ! 		     (< good-pos (- here c-state-cache-too-far)))
> ! 		(c-state-cache-non-literal-place here-bopl bopl-state)
> ! 	      good-pos)))
> ! 
> !      ((eq strategy 'backward)
> !       (setq res (c-remove-stale-state-cache-backwards here)
> ! 	    good-pos (car res)
> ! 	    scan-backward-pos (cadr res)
> ! 	    scan-forward-p (car (cddr res)))
> !       (if scan-backward-pos
> ! 	  (c-append-lower-brace-pair-to-state-cache scan-backward-pos here))
> !       (setq c-state-cache-good-pos
> ! 	    (if scan-forward-p
> ! 		(c-append-to-state-cache good-pos here)
> ! 	      good-pos)))
>   
> !      (t					; (eq strategy 'IN-LIT)
> !       (setq c-state-cache nil
> ! 	    c-state-cache-good-pos nil))))
>   
>     c-state-cache)
>   
> ***************
> *** 8559,8566 ****
>      ))
>   
>   (defun c-looking-at-special-brace-list (&optional lim)
> !   ;; If we're looking at the start of a pike-style list, ie `({ })',
> !   ;; `([ ])', `(< >)' etc, a cons of a cons of its starting and ending
>     ;; positions and its entry in c-special-brace-lists is returned, nil
>     ;; otherwise.  The ending position is nil if the list is still open.
>     ;; LIM is the limit for forward search.  The point may either be at
> --- 8516,8523 ----
>      ))
>   
>   (defun c-looking-at-special-brace-list (&optional lim)
> !   ;; If we're looking at the start of a pike-style list, ie `({ })',
> !   ;; `([ ])', `(< >)' etc, a cons of a cons of its starting and ending
>     ;; positions and its entry in c-special-brace-lists is returned, nil
>     ;; otherwise.  The ending position is nil if the list is still open.
>     ;; LIM is the limit for forward search.  The point may either be at
>
>
>> -- 
>> Michael Welsh Duggan
>> (mwd@cert.org)

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Kim Storm]

On 2013-02-01 23:18, Michael Welsh Duggan wrote:
> Alan Mackenzie <acm@muc.de> writes:
>
>> Hi, Michael.
> I've been running with this for a few days, and have yet to run into a
> problem.  I'll let you know when/if I do in the future.
>
>
Hi Alan

I have been running with the 24.2.91 pretest since it came out - and I 
have not run
into the problems I originally reported with the wrong-type-argument 
error in plain C sources.

So I assume that problem have been fixed  - or Murphy will tell me 
otherwise in 5 mintes :-).

Thank you
Kim

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Kim and Michael,

On Sat, Feb 02, 2013 at 12:50:39AM +0100, Kim Storm wrote:
> On 2013-02-01 23:18, Michael Welsh Duggan wrote:

> > I've been running with this for a few days, and have yet to run into a
> > problem.  I'll let you know when/if I do in the future.

It made my weekend, reading that.  :-)  Many thanks for all the testing
work.

> Hi Alan

> I have been running with the 24.2.91 pretest since it came out - and I 
> have not run
> into the problems I originally reported with the wrong-type-argument 
> error in plain C sources.

> So I assume that problem have been fixed  - or Murphy will tell me 
> otherwise in 5 minutes :-).

No, Murphy will wait until just after 24.3 has been released.  ;-(

Anyhow, I've closed bug#11749 as fixed.  Here's hoping that piece of CC
Mode code now has a modicum of stability about it.

> Thank you

Appreciated!

> Kim

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

Michael Welsh Duggan <mwd@cert.org> writes:

>>> There is generally no way to re-create it, so I've stopped reporting
>>> these.  I wish there were some way to record all actions in c-mode
>>> buffers such that they could be saved and re-played when this type of
>>> problem happens.  If there were some sort of debug flag I could turn
>>> on, I would turn it on by default and hopefully be able to catch some
>>> useful information.
>>
>> M-x c-toggle-parse-state-debug
>>
>> (or (c-toggle-parse-state-debug 1) in your .emacs).  Warning: variable
>> `c-debug-parse-state' is not buffer local.  I wrote this on 19th October
>> last year to help sort out the bug you reported a little earlier.  :-)
>
> I will turn this on.

Okay, I have something repeatable.  This is with an older checkout
("109494 dmantipov@yandex.ru-20120807112841-k0pyiinoxi2llcmu"), so I'm
updating my sources to see if this holds in the current bzr.  In the
meantime, here's a recipe that works in the above version.

1) emacs -Q
2) C-x C-f <attached-file>
3) M-x c-toggle-parse-state-debug RET
4) C-v until you get a state inconsistency message (10 times for me)

c-parse-state inconsistency at 9942: using cache: (9866 9516 5134 5090), from scratch: (9866 9516 (7654 . 9439) 5134 5090)
Old state:
(setq c-state-cache '(9516 5134 5090)  c-state-cache-good-pos 10231  c-state-nonlit-pos-cache '(15063 12049 9049 6049 3049)  c-state-nonlit-pos-cache-limit 15063  c-state-brace-pair-desert '(9516 . 9818)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)
c-parse-state inconsistency at 9942: using cache: (9866 9516 5134 5090), from scratch: (9866 9516 (7654 . 9439) 5134 5090)
Old state:
(setq c-state-cache '(9866 9516 5134 5090)  c-state-cache-good-pos 9867  c-state-nonlit-pos-cache '(15063 12049 9049 6049 3049)  c-state-nonlit-pos-cache-limit 15063  c-state-brace-pair-desert '(9516 . 9818)  c-state-point-min 1  c-state-point-min-lit-type nil  c-state-point-min-lit-start nil  c-state-min-scan-pos 1  c-state-old-cpp-beg nil  c-state-old-cpp-end nil)


[-- Attachment #2: conn_memory.lzz --]
[-- Type: text/plain, Size: 17995 bytes --]

/*
** Copyright (C) 2012 by Carnegie Mellon University.
**
** @OPENSOURCE_HEADER_START@
**
** Use of the SILK system and related source code is subject to the terms
** of the following licenses:
**
** GNU Public License (GPL) Rights pursuant to Version 2, June 1991
** Government Purpose License Rights (GPLR) pursuant to DFARS 252.227.7013
**
** NO WARRANTY
**
** ANY INFORMATION, MATERIALS, SERVICES, INTELLECTUAL PROPERTY OR OTHER
** PROPERTY OR RIGHTS GRANTED OR PROVIDED BY CARNEGIE MELLON UNIVERSITY
** PURSUANT TO THIS LICENSE (HEREINAFTER THE "DELIVERABLES") ARE ON AN
** "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY
** KIND, EITHER EXPRESS OR IMPLIED AS TO ANY MATTER INCLUDING, BUT NOT
** LIMITED TO, WARRANTY OF FITNESS FOR A PARTICULAR PURPOSE,
** MERCHANTABILITY, INFORMATIONAL CONTENT, NONINFRINGEMENT, OR ERROR-FREE
** OPERATION. CARNEGIE MELLON UNIVERSITY SHALL NOT BE LIABLE FOR INDIRECT,
** SPECIAL OR CONSEQUENTIAL DAMAGES, SUCH AS LOSS OF PROFITS OR INABILITY
** TO USE SAID INTELLECTUAL PROPERTY, UNDER THIS LICENSE, REGARDLESS OF
** WHETHER SUCH PARTY WAS AWARE OF THE POSSIBILITY OF SUCH DAMAGES.
** LICENSEE AGREES THAT IT WILL NOT MAKE ANY WARRANTY ON BEHALF OF
** CARNEGIE MELLON UNIVERSITY, EXPRESS OR IMPLIED, TO ANY PERSON
** CONCERNING THE APPLICATION OF OR THE RESULTS TO BE OBTAINED WITH THE
** DELIVERABLES UNDER THIS LICENSE.
**
** Licensee hereby agrees to defend, indemnify, and hold harmless Carnegie
** Mellon University, its trustees, officers, employees, and agents from
** all claims or demands made against them (and any related losses,
** expenses, or attorney's fees) arising out of, or relating to Licensee's
** and/or its sub licensees' negligent use or willful misuse of or
** negligent conduct or willful misconduct regarding the Software,
** facilities, or other rights or assistance granted by Carnegie Mellon
** University under this License, including, but not limited to, any
** claims of product liability, personal injury, death, damage to
** property, or violation of any laws or regulations.
**
** Carnegie Mellon University Software Engineering Institute authored
** documents are sponsored by the U.S. Department of Defense under
** Contract FA8721-05-C-0003. Carnegie Mellon University retains
** copyrights in all material produced under this contract. The U.S.
** Government retains a non-exclusive, royalty-free license to publish or
** reproduce these documents, or allow others to do so, for U.S.
** Government purposes only pursuant to the copyright license under the
** contract clause at 252.227.7013.
**
** @OPENSOURCE_HEADER_END@
*/

#hdr
#include <silk/silk.h>

RCSIDENTVAR(rcsID_CONN_MEMORY_H, "$Id$");

#include "connector.hh"
#end

#src
RCSIDENT("$Id$");

#include <silk/utils.h>

#ifdef SKCONNECTOR_TRACE_LEVEL
#define TRACEMSG_LEVEL SKCONNECTOR_TRACE_LEVEL
#endif
#define TRACEMSG(lvl, msg) TRACEMSG_TO_TRACEMSGLVL(lvl, msg)
#include <silk/sktracemsg.h>

/*
 *
 *    Tracing macros
 *
 */

/* Levels at which certain types of tracing are emitted */
#define FUNCTION_LEVEL 3        /* Main function tracing */
#define CHUNK_LEVEL    4        /* Chunk tracing */

/*
 *    TRACE_FUNC;
 *
 *    Emits a trace message for the containing function.  The pointer
 *    emitted is the connection object.
 */
#define TRACE_FUNC                                                      \
    TRACEMSG(FUNCTION_LEVEL, ("Function: (%p) %s", conn, __func__))


/*
 *    TRACE_MEM_CHUNK(chunk);
 *
 *    Emits trace info for a conn_mem_chunk_t data structure.
 */
#define TRACE_MEM_CHUNK(chunk)                                  \
    TRACEMSG(CHUNK_LEVEL,                                       \
             (("%s:%d Chunk: %p, writer_pos: %zd"               \
               "  reader_pos: %zd  max_reader_pos: %zd"),       \
              __FILE__, __LINE__,                               \
              chunk, chunk->writer_pos, chunk->reader_pos,      \
              chunk->max_reader_pos));


/* TYPEDEFS AND DEFINES */

/* minimum size we allow for a chunk is 4k */
#define CONN_MEM_MIN_CHUNK_SIZE     0x1000

/* standard chunk size is half a meg */
#define CONN_MEM_STD_CHUNK_SIZE     0x80000

/* standard number of chunks the connector norally has---used to
 * compute the maximum allocation */
#define CONN_MEM_STD_NUMBER_CHUNKS  3

/*
 *
 *    Macros common to all connectors
 *
 */

/* when the writer wraps, ensure this much space between the end of
 * the writer's block and the start of the reader's block */
#define WRAP_GAP  sizeof(uint64_t)

/* total space required for a block that holds 'bts_size' bytes of
 * memory in its 'data' member; includes the 'block_size' member and
 * ensures 'data' is 64-bit aligned */
#define BLOCK_TOTAL_SIZE(bts_size)                              \
    (((bts_size) + 2 * sizeof(uint64_t) - 1) & ~(UINT64_C(7)))

/* max size a caller may request; we require at least 3 blocks per
 * chunk; allow for overhead within the chunk */
#define BLOCK_MAX_SIZE_FROM_CHUNK_SIZE(bms_chunk_size)  \
    (((bms_chunk_size) - 4 * WRAP_GAP) / 3)

#end

namespace silk {

class MemoryConnector : public Connector
{
  private:
    struct conn_mem_chunk_t {
        /* pointer to the next chunk */
        conn_mem_chunk_t   *next;

        /* the data used for conn_block_t, which 'writer_pos' and
         * 'reader_pos' are offsets into */
        uint8_t            *blocks;

        /* offset to the location for the upstream writer */
        size_t              writer_pos;
        /* offset to the location for the downstream reader */
        size_t              reader_pos;

        /* total number of bytes in 'blocks'; this space includes blocks
         * the callers access and any overhead */
        size_t              capacity;

        /* last valid byte that can be read in 'blocks'; used to determine
         * when the reader_pos needs to be reset to 0 */
        size_t              max_reader_pos;
    };


  public:

    const char *get_type() const
    {
        return CONN_MEM_NAME;
    }

  private:

    static void setup_class();

    static void teardown_class(
        void *dummy);

    MemoryConnector();

    virtual ~MemoryConnector();

    conn_mem_chunk_t *chunk_alloc()
    {
        conn_mem_chunk_t *chunk;

        chunk = new conn_mem_chunk_t;
        chunk->blocks = new uint8_t[chunk_size];
        chunk->capacity = chunk_size;
        return chunk;
    }

    void chunk_free(
        conn_mem_chunk_t   *chunk)
    {
        if (chunk) {
            if (chunk->blocks) {
                delete[] chunk->blocks;
            }
            delete chunk;
        }
    }

    void chunk_pop()
    {
        conn_mem_chunk_t *chunk;

        assert(reader_chunk);
        assert(reader_chunk->next);

        if (!spare_chunk) {
            spare_chunk = reader_chunk;
            reader_chunk = reader_chunk->next;
        } else {
            chunk = reader_chunk;
            reader_chunk = reader_chunk->next;
            chunk_free(chunk);
        }
    }

    void chunk_push()
    {
        if (spare_chunk) {
            spare_chunk->next = NULL;
            spare_chunk->writer_pos = 0;
            spare_chunk->reader_pos = 0;
            spare_chunk->max_reader_pos = 0;
            writer_chunk->next = spare_chunk;
            writer_chunk = writer_chunk->next;
            spare_chunk = NULL;
        } else {
            writer_chunk->next = chunk_alloc();
            writer_chunk = writer_chunk->next;
        }
    }

    int _get_read_block(
        const conn_block_t    **reader_block,
        size_t                  size,
        bool                    no_wait)
    {
        conn_mem_chunk_t *chunk;

        assert(reader_block);
        assert(size);

        for (;;) {
            chunk = reader_chunk;
            if (chunk) {
                if (stopped) {
                    return -1;
                }

                if (chunk->reader_pos != chunk->writer_pos) {
                    if (chunk->max_reader_pos
                        && (chunk->reader_pos == chunk->max_reader_pos))
                    {
                        /* wrap the reader if the writer has wrapped */
                        chunk->reader_pos = 0;
                        chunk->max_reader_pos = 0;
                        continue;
                    }

                    /* there is data to return */
                    break;
                }
                /* else, there is no data in this block */
                if (chunk->next) {
                    /* I don't think this ever gets called, since we
                     * should have moved to the next block when the reader
                     * returned its previous block */

                    /* free this block and move to the next */
                    chunk_pop();
                    continue;
                }
            }
            /* else no data is available */

            /* check the stopping condition */
            if (stopped || no_more_data) {
                return -1;
            }

            /* wait for data unless no_wait was specified */
            empty = true;
            if (no_wait) {
                return 1;
            }
            pthread_cond_wait(&cond, &mutex);
        }

        TRACE_FUNC;

        /* we get here when there is data to return */
        *reader_block = (conn_block_t*)&chunk->blocks[chunk->reader_pos];
        return 0;
    }

    void _return_read_block(
        const conn_block_t *reader_block)
    {
        conn_mem_chunk_t *chunk;

        assert(reader_block);

        chunk = reader_chunk;
        assert(reader_block==(conn_block_t*)&chunk->blocks[chunk->reader_pos]);

        /* move to next reader position */
        chunk->reader_pos += BLOCK_TOTAL_SIZE(reader_block->block_size);
        if (chunk->reader_pos == chunk->max_reader_pos) {
            chunk->reader_pos = 0;
            chunk->max_reader_pos = 0;
        }
        TRACE_MEM_CHUNK(chunk);

        /* check whether we need to move to the next chunk */
        if (chunk->reader_pos == chunk->writer_pos
            && chunk->next)
        {
            /* free this block and move to the next */
            chunk_pop();
        }
    }

    int _get_write_block(
        const conn_block_t    **writer_block,
        size_t                  size,
        bool                    no_wait)
    {
        conn_mem_chunk_t *chunk;
        size_t required_size;

        assert(writer_block);
        assert(size);

        *writer_block = NULL;

        if (block_max_size < size) {
            throw sk_error_conn_block_size_too_large_t();
        }

        /* account for overhead */
        required_size = BLOCK_TOTAL_SIZE(size);

        /* loop until there is space available */
        for (;;) {
            /* handle stopped condition */
            if (stopped) {
                return -1;
            }

            if (total_used + size > max_allocation) {
                /* we are full; wait for space unless no_wait was
                 * specified */
                full = true;
                if (no_wait) {
                    return 1;
                }
                pthread_cond_wait(&cond, &mutex);
                continue;
            }

            chunk = writer_chunk;
            if (NULL == chunk) {
                /* create the initial chunk */
                chunk = writer_chunk = reader_chunk = chunk_alloc();
            }

            if (chunk->writer_pos < chunk->reader_pos) {
                /* the writer has wrapped; look for space between the
                 * writer_pos and the reader_pos */
                if ((chunk->reader_pos - chunk->writer_pos)
                    >= (required_size + WRAP_GAP))
                {
                    /* space is available */
                    break;
                }
            } else {
                if ((chunk->capacity - chunk->writer_pos) >= required_size) {
                    /* space is available at end of the buffer */
                    break;
                }
                /* else, see if space is available at the front */
                if (chunk->reader_pos >= required_size + WRAP_GAP) {
                    /* space is available; wrap around */
                    chunk->max_reader_pos = chunk->writer_pos;
                    chunk->writer_pos = 0;
                    TRACE_MEM_CHUNK(chunk);
                    break;
                }
            }

            /* space is not available; allocate new block */
            chunk_push();
        }

        TRACE_FUNC;

        /* when we get here, space is available */
        *writer_block = (conn_block_t*)&chunk->blocks[chunk->writer_pos];
        ((conn_block_t*)&chunk->blocks[chunk->writer_pos])->block_size
            = required_size - sizeof(uint64_t);

        return  0;
    }

    void _return_write_block(
        const conn_block_t *writer_block,
        size_t              size)
    {
        conn_mem_chunk_t *chunk;

        assert(writer_block);
        assert(size > 0);

        chunk = writer_chunk;
        assert(writer_block==(conn_block_t*)&chunk->blocks[chunk->writer_pos]);

        ((conn_block_t*)&chunk->blocks[chunk->writer_pos])->block_size = size;
        chunk->writer_pos += BLOCK_TOTAL_SIZE(size);
        TRACE_MEM_CHUNK(chunk);
    }

    void _set_param(
        const Definition& defn,
        const std::vector<std::string>& values) throw (Error)
    {
        uint64_t u64;
        int rv;

        switch (*(param_id_t *)(defn.get_context())) {
          case CONN_MEM_CHUNK_SIZE:
            rv = skStringParseUint64(&u64, values[0].c_str(), 0, 0);
            if (rv) {
                goto PARSE_ERROR;
            }
            chunk_size = u64;
            break;

          case CONN_MEM_MAX_ALLOCATION:
            rv = skStringParseUint64(&u64, values[0].c_str(), 0, 0);
            if (rv) {
                goto PARSE_ERROR;
            }
            max_allocation = u64;
            break;
        }
        return;

      PARSE_ERROR:
        std::ostringstream os;
        os << "Invalid " << defn.get_name() << " '" << values[0] << "': "
           << skStringParseStrerror(rv);
        throw sk_error_conn_param_value_t(os.str());
    }


    void _init()
    {
        if (chunk_size) {
            if (chunk_size < CONN_MEM_MIN_CHUNK_SIZE) {
                std::ostringstream os;
                os << "Specified chunk size is below minimum of "
                   << CONN_MEM_MIN_CHUNK_SIZE;
                throw sk_error_conn_param_value_t(os.str());
            }
            if (0 == max_allocation) {
                /* use caller's chunk size, and the default multiple of it
                 * for the maximum allocation */
                max_allocation = CONN_MEM_STD_NUMBER_CHUNKS * chunk_size;
            } else if (chunk_size > max_allocation) {
                /* chunk size is larger than max allocation */
                throw sk_error_conn_param_value_t(
                    ("Specified chunk size greater than"
                     " specified max allocation"));
            }
            /* else both values specified and are valid */
        } else if (max_allocation) {
            if (max_allocation < CONN_MEM_MIN_CHUNK_SIZE) {
                std::ostringstream os;
                os << "Specified max allocation is below minimum of "
                   << CONN_MEM_MIN_CHUNK_SIZE;
                throw sk_error_conn_param_value_t(os.str());
            }

            /* determine the chunk size */
            if (max_allocation >= CONN_MEM_STD_CHUNK_SIZE) {
                /* use standard chunk size */
                chunk_size = CONN_MEM_STD_CHUNK_SIZE;
            } else {
                chunk_size = (max_allocation / CONN_MEM_STD_NUMBER_CHUNKS);
                if (chunk_size < CONN_MEM_MIN_CHUNK_SIZE) {
                    chunk_size = CONN_MEM_MIN_CHUNK_SIZE;
                }
            }
        } else {
            /* use the default sizes */
            chunk_size = CONN_MEM_STD_CHUNK_SIZE;
            max_allocation = CONN_MEM_STD_NUMBER_CHUNKS * chunk_size;
        }

        block_max_size = BLOCK_MAX_SIZE_FROM_CHUNK_SIZE(chunk_size);

        initialized = true;
    }

    /* linked list of chunks of memory */
    conn_mem_chunk_t   *writer_chunk;
    conn_mem_chunk_t   *reader_chunk;
    conn_mem_chunk_t   *spare_chunk;

    /* size of an individual chunk of memory; may be specified by the
     * caller */
    size_t              chunk_size;

    /* maximum block size a module may request; calculated as
     * roughly 1/3 of the chunk_size */
    size_t              block_max_size;

    enum param_id_t {
        CONN_MEM_CHUNK_SIZE,
        CONN_MEM_MAX_ALLOCATION
    };

    struct class_params_t {
        const char    *key;
        int            val;
        const char    *desciption;
    };

    static MemoryConnector::class_params_t class_params[] = {
        {"chunk_size",     CONN_MEM_CHUNK_SIZE,
         "Size of an individual chunk of memory"},
        {"max_allocation", CONN_MEM_MAX_ALLOCATION,
         ("Maximum amount of memory that may be allocated across all blocks"
          " on all chunks; does not include internal overhead")},
        {0, 0, 0}          /* sentinel */
    };

#if 0
    static const std::set<Parameterized::Definition>& create_param_defs()
    {
        size_t i;
        std::set<Parameterized::Definition> defs;

        for (i = 0; class_params[i].key != NULL; ++i) {
            assert(i == (size_t)class_params[i].val);
            Parameterized::Definition d(class_params[i].key,
                                        class_params[i].desciption,
                                        (void*)&class_params[i].val, 1, 1);
            assert(defs.find(d) = defs.end());
            defs.insert(d);
        }
        return defs;
    }
#endif

};

}

/*
** Local Variables:
** mode:c++
** indent-tabs-mode:nil
** c-basic-offset:4
** End:
*/

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Michael Welsh Duggan <mwd@cert.org> writes:

> Michael Welsh Duggan <mwd@cert.org> writes:
>
> Okay, I have something repeatable.  This is with an older checkout
> ("109494 dmantipov@yandex.ru-20120807112841-k0pyiinoxi2llcmu"), so I'm
> updating my sources to see if this holds in the current bzr.  

It indeed works in bzr version "109966
rgm@gnu.org-20120910101811-dxrpeq7h0vulwpr4", to which I have just
updated.

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Michael Welsh Duggan]

Michael Welsh Duggan <mwd@cert.org> writes:

> Michael Welsh Duggan <mwd@cert.org> writes:
>
>> Michael Welsh Duggan <mwd@cert.org> writes:
>>
>> Okay, I have something repeatable.  This is with an older checkout
>> ("109494 dmantipov@yandex.ru-20120807112841-k0pyiinoxi2llcmu"), so I'm
>> updating my sources to see if this holds in the current bzr.  
>
> It indeed works in bzr version "109966
> rgm@gnu.org-20120910101811-dxrpeq7h0vulwpr4", to which I have just
> updated.

And by works, I mean the recipe works, and the warning is triggered.

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#11749: I also have this issue

[Denis Zalevskiy]

In GNU Emacs 24.1.1 (x86_64-redhat-linux-gnu, GTK+ Version 2.24.11)
 of 2012-08-13 on buildvm-11.phx2.fedoraproject.org

*Messages*
...
c-inside-bracelist-p: Wrong type argument: number-or-marker-p, (4212 . 4415)
...

it occurs sometimes. 

I think it has some relation to bug 4590. At least, symptoms are the same and 
4590 is reproducible on my debian installation with the same version of emacs.

In c++ I can use C-c C-s to see all the code now is topmost-intro (like in bug 
4590), here it just end up in error message pointed above.

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Glenn Morris]

If you'd like to see this in the next pretest, please install this in
emacs-24 as soon as possible.

FWIW, here is some sanity checking from someone who knows nothing about
cc-mode (other than the terror that it brings). Just ignore any of these
comments if they are obviously wrong.

Alan Mackenzie wrote:

> + (defvar c-state-old-cpp-beg-marker)

I guess you wanted to give that an actual value, ie

(defvar c-state-old-cpp-beg-marker nil)

> + (defvar c-state-old-cpp-end-marker)

Likewise.

> + (defmacro c-state-maybe-marker (place marker)
> +   ;; If PLACE is non-nil, return a marker marking it, otherwise nil.
> +   ;; We (re)use MARKER.
> +   `(and ,place
> + 	(or ,marker (setq ,marker (make-marker)))
> + 	(set-marker ,marker ,place)))
> + 
[...]
> !       (setq c-state-old-cpp-beg (and here-cpp-beg (copy-marker here-cpp-beg t))
> ! 	    c-state-old-cpp-end (and here-cpp-end (copy-marker here-cpp-end t)))
> !       )))
>   
>   ;; Debug tool to catch cache inconsistencies.  This is called from
>   ;; 000tests.el.
> --- 3372,3381 ----
>   	      (c-parse-state-1))
>   	   (c-with-cpps-commented-out
>   	    (c-parse-state-1))))
> !       (setq c-state-old-cpp-beg
> ! 	    (c-state-maybe-marker here-cpp-beg c-state-old-cpp-beg-marker)
> ! 	    c-state-old-cpp-end
> ! 	    (c-state-maybe-marker here-cpp-end c-state-old-cpp-end-marker)))))

Looks like the marker insertion type has changed?

bug#11749: Acknowledgement (24.1; C-mode indentation gives wrong-type-argument error.)

[Alan Mackenzie]

Hi, Glenn.

On Tue, Jan 08, 2013 at 02:10:58PM -0500, Glenn Morris wrote:

> If you'd like to see this in the next pretest, please install this in
> emacs-24 as soon as possible.

DONE.  I'll be looking forward to the new pretest ?tomorrow.

> FWIW, here is some sanity checking from someone who knows nothing about
> cc-mode (other than the terror that it brings). Just ignore any of these
> comments if they are obviously wrong.

> Alan Mackenzie wrote:

> > + (defvar c-state-old-cpp-beg-marker)

> I guess you wanted to give that an actual value, ie

> (defvar c-state-old-cpp-beg-marker nil)

Sugar!  I must stop being caught out by this.  That construct merely
interns a symbol, leaving its value unbound.  Thanks very much for
catching this.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#11749: 24.1; C-mode indentation gives wrong-type-argument error.

[Alan Mackenzie]

Bug fixed.

bug#13385: 24.1; TAB in C file causes type error

[Julian Stecklina]

Hello,

While editing C files, I sometimes stumble upon the following error. I
cannot reliably reproduce its first occurence, but when it occurs it
will break indenting code until I kill the buffer and revisit the file.

I removed all cc-mode customization from my .emacs.el, but it doesn't
make a difference.

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p (51932 . 51982))
  c-looking-at-inexpr-block((51932 . 51982) (51932 . 51982))
  c-inside-bracelist-p(52112 ((51932 . 51982) 54476 (52112 . 54355) 54476 (52112 . 54355) 52112 (51932 . 51982) 52112 (51932 . 51982) 52112 (46073 . 46349) (51932 . 51982) 52112 (46073 . 46349) (51932 . 51982) 52112 (51932 . 51982) 52112 (51932 . 51982) 52112 (51932 . 51982) 52112 (51932 . 51982) 52112 (46073 . 46349) (51932 . 51982) 52112 (51932 . 51982) 52112 (46073 . 46349) (51932 . 51982) 52112 (51932 . 51982) 52112 (46073 . 46349) (51932 . 51982) 51213 51180 50702 (50307 . 50564)))
  c-guess-basic-syntax()
  c-indent-line()
  #[nil "\302\303 !\210\b\304>\203.^@\212\305 \210\306\307x\210\305 \210\310 )i\310 X\203'^@\311	!\202,^@\212\311	!))\207\b \207" [indent-line-function column syntax-propertize line-end-position (indent-relative indent-relative-maybe) beginning-of-line "\n 	" nil current-indentation indent-line-to] 2 2125346 nil]()
  c-indent-command(nil)
  c-indent-line-or-region(nil nil)
  call-interactively(c-indent-line-or-region nil nil)

bug#13385: closed (Re: 24.1; C-mode indentation gives wrong-type-argument error.)

[Julian Stecklina]

Is there a way to work around that bug in the latest release?