Discussion:
Sysadmin tools, microformats, and data exchange
(too old to reply)
Luke Kanies
2005-11-30 17:48:28 UTC
Permalink
Hi all,

I've been working on Puppet[1] for a while now, and while it's coming along
nicely and I'm starting to collect clients who will use it, I'm also
thinking about how Puppet will interact with the rest of the network.

One of the things that's always driven me nuts about sysadmin tools is that
they don't talk to each other. I've thought a lot about this in Puppet's
development, and initially it seemed like I'd end up writing a lot of hooks
into the different tools, so that, for instance, Puppet would know how to
talk to Nagios and RT and a few others.

In thinking about it, though, that sounds like a pain, because I would never
be able to support all of the tools out there, and no one else could reuse
the connecting work I did.

So, I've been thinking about how I can decouple things so that I get the
end result of being able to integrate with these tools but in a way that's
reusable and also allows others to implement one side of a hook (e.g.,
support for a specific application) in a way that I can use with Puppet.

I started thinking about the kind of data I want to exchange, and most if it
is pretty darn simple: log messages, metrics, alerts, trouble tickets,
things like that. Basically, if two sysadmin tools _should_ be talking, it
seems like it is almost always pretty easy to come up with a simple data
concept for that talking. For instance, Puppet will be producing metrics,
notifying Nagios that service X on machine Y should be monitored, and will
be opening trouble tickets when it encounters a failure.

So, can we come up with a standard way of talking about those little data
formats, such that all of our sysadmin apps can support some subset of these
objects and accept them over the wire?

Initially, it looked like Microformats[2] was the answer, but it turns out
that those really only exist for web browser documents, and their existence
is driven by the fact that HTTP header tags are limited. Or something. The
point is, they aren't really useful for non-HTML data exchange.

So, what else is there? Well, we in the Unix world are already constantly
using a simple messaging format, SMTP, so I am thinking maybe we could
produce formats akin to microformats but built to ride on SMTP instead of
HTTP. It's generally pretty damn easy to configure Unix systems to hand
messages off to an external program -- I know Postfix and Sendmail support
this out of the box, and I'm sure QMail does, also. So the question is,
could we create something like a little microformats router, in that the
SMTP server accepts the objects, sends them to our router, and then our
router passes them off to Nagios, or RT, or whatever.

It might be that these apps should thus learn to speak SMTP natively, but,
well, that doesn't really seem necessary. SMTP is well-known, supports
everything I can think we would need (authentication, certificates, etc.),
and is already very fast and very stable. Best of all, there are multiple
fast and stable servers out there, and they all know how to hand data off to
an external program, which is all we need here.

That leaves me with some summary questions, then:

First, would something like this be useful?

Second, would you use it, and potentially even contribute to it?

Third, how should we go about creating it? Should the formats be special
SMTP messages, just SMTP attachments, something entirely different, or what?

I'll contact a few of the developers for key related projects, like RT and
hopefully Nagios, but if anyone else can think of developers who should be
informed, please ping them with this discussion. I'm also going to ping the
microformats people and see if they're interested in trying to do SMTP based
formats, or supporting both SMTP and HTTP, so that I can reuse their
structure.

Really, I'm serious here -- I'm going to do something here, and I'm willing
to stand up and lead the effort, but I'd like some feedback.

Thanks,
Luke

1 - http://reductivelabs.com/projects/puppet
2 - http://www.microformats.org
--
I object to doing things that computers can do.
--Olin Shivers
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-11-30 18:13:34 UTC
Permalink
Well, if I think of it in terms of how other tools do it then I have
to answer a modules API. Most things handle this in terms of dynamically
loadable .so files, but that might be overkill here, or not fit inside
the idiom you're working in. Still, having an API to define modules
that people could write to arbitrarily extend to other things via
message passing or whatever would be useful. Then you could provide
some examples like RT and Nagios and people could run with it.
maybe all modules must have a <modulename>_init method that defines
callbacks or whatever.
(e.g. Apache, php, tcl, perl, gaim, etc)
There's a lot of examples done 'right' out there.
Modules don't pass over the wire, though. No amount of module loading in
Puppet will ever allow me to talk to, say, Nagios, which doesn't provide a
network API for configuration.

This is the key here: I want a simple format that I can pass, over the
wire, from one tool to another, and I want a simple message bus for passing
that data.
--
Today at work an ethernet switch decided to take the 'N' out of NVRAM
-- Richard Letts
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Doug Hughes
2005-11-30 18:17:22 UTC
Permalink
Post by Luke Kanies
Well, if I think of it in terms of how other tools do it then I have
to answer a modules API. Most things handle this in terms of dynamically
loadable .so files, but that might be overkill here, or not fit inside
the idiom you're working in. Still, having an API to define modules
that people could write to arbitrarily extend to other things via
message passing or whatever would be useful. Then you could provide
some examples like RT and Nagios and people could run with it.
maybe all modules must have a <modulename>_init method that defines
callbacks or whatever.
(e.g. Apache, php, tcl, perl, gaim, etc)
There's a lot of examples done 'right' out there.
Modules don't pass over the wire, though. No amount of module loading in
Puppet will ever allow me to talk to, say, Nagios, which doesn't provide a
network API for configuration.
This is the key here: I want a simple format that I can pass, over the
wire, from one tool to another, and I want a simple message bus for passing
that data.
The module can/should take care of all of that for you.
e.g. your nagios module sends an SNMP trap.. your RT module might
use CLI to manipulate tickets, your mysql module injects the
method into a DB log table, etc.

Message busses are hard. (though you might have a module that works
with the Tibco Rendevous bus - a LOT of commercial tools work with
that, and some free)

Doug
Luke Kanies
2005-11-30 18:21:43 UTC
Permalink
Post by Doug Hughes
The module can/should take care of all of that for you.
e.g. your nagios module sends an SNMP trap.. your RT module might
use CLI to manipulate tickets, your mysql module injects the
method into a DB log table, etc.
It sounds like you're saying here that my module literally does these things
(modify a table, send a trap), which I most definitely do not want. That's
not a network API, which is specifically what I'm looking for. System-level
abstractions are not sufficient, I want network-level abstractions. I don't
want to care what kind of system is on the other end of the API, I just want
to care what kind of data it can accept. Can you take trouble tickets?
Great, here's one. Can't handle metrics? OK, who can?
Post by Doug Hughes
Message busses are hard. (though you might have a module that works
with the Tibco Rendevous bus - a LOT of commercial tools work with
that, and some free)
Yeah, um, if I sell some tool as a sysadmin simplification tool but it
requires Tibco, um, please shoot me.

I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
--
It's a small world, but I wouldn't want to paint it.
-- Stephen Wright
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Doug Hughes
2005-11-30 18:39:08 UTC
Permalink
Post by Luke Kanies
Post by Doug Hughes
The module can/should take care of all of that for you.
e.g. your nagios module sends an SNMP trap.. your RT module might
use CLI to manipulate tickets, your mysql module injects the
method into a DB log table, etc.
It sounds like you're saying here that my module literally does these things
(modify a table, send a trap), which I most definitely do not want. That's
not a network API, which is specifically what I'm looking for. System-level
abstractions are not sufficient, I want network-level abstractions. I don't
want to care what kind of system is on the other end of the API, I just want
to care what kind of data it can accept. Can you take trouble tickets?
Great, here's one. Can't handle metrics? OK, who can?
not 'your' module per se. Any module. You just provide the API
for people to get stuff in and out of puppet. Very flexible.
Post by Luke Kanies
Post by Doug Hughes
Message busses are hard. (though you might have a module that works
with the Tibco Rendevous bus - a LOT of commercial tools work with
that, and some free)
Yeah, um, if I sell some tool as a sysadmin simplification tool but it
requires Tibco, um, please shoot me.
I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
I think the difficulty of this approach is that you are going to have
to then develop message bus handlers on both sides for every application.
Now, what if somebody else develops their own message bus (there are
several out there). Now every 3rd party app has to support multiple
message busses.

If you just develop a generic module API that allows for taking things
out of and putting things into puppet, then you can leave the details
of all of this to the module writers. That doesn't mean that you can't
build a reuse library in your module API for handling similar things
(something like DBI for perl). But, each third party app can use
more native means for communication. In fact, your SMTP bus solution
could itself be a module.

I think the custom bus system may be fine for free stuff where you can
hack into the existing app and do your bus receiver, but it's not
as flexible as a module API. For instance, say I want to inject messages
into netcool about things that puppet found (that's our EMS). I have
a socket probe or snmp trap probe or syslog probe that can handle the
events. Ideally I'd use the socket probe and just format the messages
and send them to the failover socket collector agents for injection
into netcool. With a SMTP message bus, I'd have to write a deamon or something
to handle the SMTP message injected from your bus and re-translate them
and either run an SMTP server on the netcool box for intercepting
or hack into my existing smtp server to extract the stuff and then,
finally, send them to the collector agent. It's much less direct.

Doug
Matt S Trout
2005-11-30 19:28:12 UTC
Permalink
Yeah, but like all things I expect there will be one or two "standard"
busses. For instance, I could see SMTP and XMLRPC being considered the
common buses, and then everything else, not so much.
Much maligned though it might be, SOAP already has not only defined standards
for transport over SMTP and HTTP but a good general (if a little heavyweight)
means of describing sequences, correlation values, encryption etc. through
the various extensions to the standards. It also already has at least some
support in most languages.
--
Matt S Trout Offering custom development, consultancy and support
Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
Luke Kanies
2005-11-30 19:31:37 UTC
Permalink
Post by Matt S Trout
Much maligned though it might be, SOAP already has not only defined standards
for transport over SMTP and HTTP but a good general (if a little heavyweight)
means of describing sequences, correlation values, encryption etc. through
the various extensions to the standards. It also already has at least some
support in most languages.
I've used SOAP a bit and I agree with the movements that put it out of favor
compared to things like REST and and XMLRPC. It's larger than it needs to
be, and it shows.

It also doesn't provide anything like a message bus, and as was mentioned, I
don't want to intermix the format and the transport. If I use SOAP messages
as the format, then I am committed to essentially requiring everyone to
speak SOAP, which will likely never be the case.
--
A person's maturity consists in having found again the seriousness one
had as a child, at play. --Friedrich Nietzsche
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-11-30 19:11:32 UTC
Permalink
Post by Doug Hughes
Post by Luke Kanies
It sounds like you're saying here that my module literally does these things
(modify a table, send a trap), which I most definitely do not want. That's
not a network API, which is specifically what I'm looking for. System-level
abstractions are not sufficient, I want network-level abstractions. I don't
want to care what kind of system is on the other end of the API, I just want
to care what kind of data it can accept. Can you take trouble tickets?
Great, here's one. Can't handle metrics? OK, who can?
not 'your' module per se. Any module. You just provide the API
for people to get stuff in and out of puppet. Very flexible.
Flexible, yes, but not what I'm looking for. Again, the point is network
abstraction, and modules just don't provide that.
Post by Doug Hughes
Post by Luke Kanies
Yeah, um, if I sell some tool as a sysadmin simplification tool but it
requires Tibco, um, please shoot me.
I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
I think the difficulty of this approach is that you are going to have
to then develop message bus handlers on both sides for every application.
Now, what if somebody else develops their own message bus (there are
several out there). Now every 3rd party app has to support multiple
message busses.
Yeah, but like all things I expect there will be one or two "standard"
busses. For instance, I could see SMTP and XMLRPC being considered the
common buses, and then everything else, not so much.
Post by Doug Hughes
If you just develop a generic module API that allows for taking things
out of and putting things into puppet, then you can leave the details
of all of this to the module writers. That doesn't mean that you can't
build a reuse library in your module API for handling similar things
(something like DBI for perl). But, each third party app can use
more native means for communication. In fact, your SMTP bus solution
could itself be a module.
Again, while I agree that modules are a solution to some problems, I just
don't see how they provide network abstraction, which is explicitly what I
want. Modules would just make it easy to have things on the same nodes, but
it would not make it any easier for different apps to talk to each other,
unless we then wrote modules which themselves had their own standards.

How, specifically, does a module API allow an app on one server to talk to
an app on another server?
Post by Doug Hughes
I think the custom bus system may be fine for free stuff where you can
hack into the existing app and do your bus receiver, but it's not
as flexible as a module API. For instance, say I want to inject messages
into netcool about things that puppet found (that's our EMS). I have
a socket probe or snmp trap probe or syslog probe that can handle the
events. Ideally I'd use the socket probe and just format the messages
and send them to the failover socket collector agents for injection
into netcool. With a SMTP message bus, I'd have to write a deamon or something
to handle the SMTP message injected from your bus and re-translate them
and either run an SMTP server on the netcool box for intercepting
or hack into my existing smtp server to extract the stuff and then,
finally, send them to the collector agent. It's much less direct.
Well, someone somewhere has to write that converter, yes, just like someone
somewhere would have to write the module that you're talking about, so there
isn't a significant amount less work to do.

However, the big difference here is that these simplistic formats and their
simple bus provide two conversions, first from my custom format to a generic
one, and then from the generic one to your custom format, whereas modules
provide just one conversion, using a per-conversion module for doing the
work.

However, the big benefit of the conversions is that I can write a single,
central routing daemon that can do all of the conversions for me, in one
centrally managed place. If I use modules, then I haven't reduced the
complexity of the existing cruft at all, other than provided an API.

For Netcool, I would have a single routing daemon running on my network, and
this routing daemon would have a callback loaded for certain formats, such
that they would get converted to Netcool's formats and sent along. It's not
a significant amount more work on anyone's parts -- things still have to get
converted from Puppet and to Netcool -- but it provides a heckuva lot more
functionality.
--
I object to doing things that computers can do.
--Olin Shivers
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Doug Hughes
2005-11-30 19:29:55 UTC
Permalink
Post by Luke Kanies
Post by Doug Hughes
Post by Luke Kanies
It sounds like you're saying here that my module literally does these things
(modify a table, send a trap), which I most definitely do not want. That's
not a network API, which is specifically what I'm looking for. System-level
abstractions are not sufficient, I want network-level abstractions. I don't
want to care what kind of system is on the other end of the API, I just want
to care what kind of data it can accept. Can you take trouble tickets?
Great, here's one. Can't handle metrics? OK, who can?
not 'your' module per se. Any module. You just provide the API
for people to get stuff in and out of puppet. Very flexible.
Flexible, yes, but not what I'm looking for. Again, the point is network
abstraction, and modules just don't provide that.
Post by Doug Hughes
Post by Luke Kanies
Yeah, um, if I sell some tool as a sysadmin simplification tool but it
requires Tibco, um, please shoot me.
I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
I think the difficulty of this approach is that you are going to have
to then develop message bus handlers on both sides for every application.
Now, what if somebody else develops their own message bus (there are
several out there). Now every 3rd party app has to support multiple
message busses.
Yeah, but like all things I expect there will be one or two "standard"
busses. For instance, I could see SMTP and XMLRPC being considered the
common buses, and then everything else, not so much.
Post by Doug Hughes
If you just develop a generic module API that allows for taking things
out of and putting things into puppet, then you can leave the details
of all of this to the module writers. That doesn't mean that you can't
build a reuse library in your module API for handling similar things
(something like DBI for perl). But, each third party app can use
more native means for communication. In fact, your SMTP bus solution
could itself be a module.
Again, while I agree that modules are a solution to some problems, I just
don't see how they provide network abstraction, which is explicitly what I
want. Modules would just make it easy to have things on the same nodes, but
it would not make it any easier for different apps to talk to each other,
unless we then wrote modules which themselves had their own standards.
How, specifically, does a module API allow an app on one server to talk to
an app on another server?
It doesn't in a generic sense. It solves the particular problem of
allowing your app to talk to everything else through a common API.
I'd thought that solving everybody's problems might be a little bit
beyond scope, but it might be an interesting generic tool if it
ever caught on. Getting buy-in for connecting app-x to app-y when
you're not the author of each is always so politically charged. (NIH)
Post by Luke Kanies
Post by Doug Hughes
I think the custom bus system may be fine for free stuff where you can
hack into the existing app and do your bus receiver, but it's not
as flexible as a module API. For instance, say I want to inject messages
into netcool about things that puppet found (that's our EMS). I have
a socket probe or snmp trap probe or syslog probe that can handle the
events. Ideally I'd use the socket probe and just format the messages
and send them to the failover socket collector agents for injection
into netcool. With a SMTP message bus, I'd have to write a deamon or something
to handle the SMTP message injected from your bus and re-translate them
and either run an SMTP server on the netcool box for intercepting
or hack into my existing smtp server to extract the stuff and then,
finally, send them to the collector agent. It's much less direct.
Well, someone somewhere has to write that converter, yes, just like someone
somewhere would have to write the module that you're talking about, so there
isn't a significant amount less work to do.
maybe. What you're doing is basicaly just moving the translation
module to a routing place instead of as a hook into puppet, it seems
to me.
Post by Luke Kanies
However, the big difference here is that these simplistic formats and their
simple bus provide two conversions, first from my custom format to a generic
one, and then from the generic one to your custom format, whereas modules
provide just one conversion, using a per-conversion module for doing the
work.
However, the big benefit of the conversions is that I can write a single,
central routing daemon that can do all of the conversions for me, in one
centrally managed place. If I use modules, then I haven't reduced the
complexity of the existing cruft at all, other than provided an API.
My suggestion would be to have a modules API and your communication
bus as one of the modules. The extra module abstraction gives people
a lot more flexibility to fit in things that don't fit in the normal
SMTP message passing idiom. Then people can do generic extensions
like something that writes to a DB as well as sending messages to
things that somebody writes the bus module for.
Post by Luke Kanies
For Netcool, I would have a single routing daemon running on my network, and
this routing daemon would have a callback loaded for certain formats, such
that they would get converted to Netcool's formats and sent along. It's not
a significant amount more work on anyone's parts -- things still have to get
converted from Puppet and to Netcool -- but it provides a heckuva lot more
functionality.
Personally, I'd rather have puppet (or whatever) send to netcool directly
(and other things) than have a specialized routing daemon out there that
has to be maintained and managed separately (from a process and failover
perspective). I've already got failover built into my collectors.
If there was a modular API I could use, despite the existance of
the routing thing, I'd use the API becaue it allows me to take advantage
of my failover without introducting more dependencies on SMTP working
and on the routing server working, and on mail delays during other
incidents.

Doug
Luke Kanies
2005-11-30 21:10:58 UTC
Permalink
Post by Doug Hughes
Post by Luke Kanies
How, specifically, does a module API allow an app on one server to talk to
an app on another server?
It doesn't in a generic sense. It solves the particular problem of
allowing your app to talk to everything else through a common API.
I'd thought that solving everybody's problems might be a little bit
beyond scope, but it might be an interesting generic tool if it
ever caught on. Getting buy-in for connecting app-x to app-y when
you're not the author of each is always so politically charged. (NIH)
I'm nothing if not ambitious. :) I figure if I'm trying to build an
operating system abstraction layer (which is part of what Puppet is), I
might as well build the backbone for a sysadmin tool mesh.
Post by Doug Hughes
Post by Luke Kanies
Well, someone somewhere has to write that converter, yes, just like someone
somewhere would have to write the module that you're talking about, so there
isn't a significant amount less work to do.
maybe. What you're doing is basicaly just moving the translation
module to a routing place instead of as a hook into puppet, it seems
to me.
Right, that's exactly what I'm doing, but I'm taking the additional step of
providing a generic networkable mechanism for communicating to that routing
place, which is the key. What if you use cfengine, not Puppet? It's highly
unlikely that Mark Burgess is going to ever write to any API that does this,
but someone can pretty easily write a simple converter that takes cfengine
log files, converts them, and sends them on their way.
Post by Doug Hughes
My suggestion would be to have a modules API and your communication
bus as one of the modules. The extra module abstraction gives people
a lot more flexibility to fit in things that don't fit in the normal
SMTP message passing idiom. Then people can do generic extensions
like something that writes to a DB as well as sending messages to
things that somebody writes the bus module for.
Well, I will certainly hope to provide libraries and hooks and such for
major languages, but, well, I'll be writing everything I do in Ruby, and I
don't think people would be very interested if I tried to publish a Ruby API
as the "standard" API.

I think it makes a lot of sense to have this central routing thing support
an API that supports simple modules, but that's different because that's an
implementation, not a spec.
Post by Doug Hughes
Personally, I'd rather have puppet (or whatever) send to netcool directly
(and other things) than have a specialized routing daemon out there that
has to be maintained and managed separately (from a process and failover
perspective). I've already got failover built into my collectors.
If there was a modular API I could use, despite the existance of
the routing thing, I'd use the API becaue it allows me to take advantage
of my failover without introducting more dependencies on SMTP working
and on the routing server working, and on mail delays during other
incidents.
And I think that you'll be able to come very close to that. At the worst,
you'd have to have a simple converter on your systems, so the tools that
spoke this format would pass it to a local converter which then converted
it and sent the data on its way.

I just know for a fact that one-off communication mechanisms will never make
any real progress, and I also know that a network API and format is the only
thing that's going to provide the kind of functionality I really need, which
is a way for all of the different tools to easily talk to each other,
preferably without knowing who they're talking to.

As I mentioned in my response to Ian's email, I don't even want my nodes to
have to know who they're talking to, and I don't want to make them
responsible for duplicating messages that multiple tools need. You might
have both a trending system and an alerting system that consume metrics;
with your system, every node on my network would have to send duplicate
metric messages, but with mine each node would forward the metric to the
router, and the router would duplicate and forward as necessary.
--
Between two evils, I always pick the one I never tried before. -- Mae West
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Doug Hughes
2005-11-30 21:39:16 UTC
Permalink
Post by Luke Kanies
I'm nothing if not ambitious. :) I figure if I'm trying to build an
operating system abstraction layer (which is part of what Puppet is), I
might as well build the backbone for a sysadmin tool mesh.
amibitious, masochistic, other synomyms.. ;)
Post by Luke Kanies
Post by Doug Hughes
maybe. What you're doing is basicaly just moving the translation
module to a routing place instead of as a hook into puppet, it seems
to me.
Right, that's exactly what I'm doing, but I'm taking the additional step of
providing a generic networkable mechanism for communicating to that routing
place, which is the key. What if you use cfengine, not Puppet? It's highly
unlikely that Mark Burgess is going to ever write to any API that does this,
but someone can pretty easily write a simple converter that takes cfengine
log files, converts them, and sends them on their way.
When I put on my parochial hat, I say solving cfengine's issues is somebody
else's problem. ;)

(I write a custom parser that takes cfengine output and sends it
where it needs to be sent)
Post by Luke Kanies
Post by Doug Hughes
My suggestion would be to have a modules API and your communication
bus as one of the modules. The extra module abstraction gives people
a lot more flexibility to fit in things that don't fit in the normal
SMTP message passing idiom. Then people can do generic extensions
like something that writes to a DB as well as sending messages to
things that somebody writes the bus module for.
Well, I will certainly hope to provide libraries and hooks and such for
major languages, but, well, I'll be writing everything I do in Ruby, and I
don't think people would be very interested if I tried to publish a Ruby API
as the "standard" API.
I wouldn't be so sure. At least then it's there. (and it has the advantage of
being idiomatic)
Post by Luke Kanies
I think it makes a lot of sense to have this central routing thing support
an API that supports simple modules, but that's different because that's an
implementation, not a spec.
I think, to be fair, you have to admit that you need both a spec for
your routing protocol and an API for the bus manipulation. No free lunch.
Post by Luke Kanies
Post by Doug Hughes
Personally, I'd rather have puppet (or whatever) send to netcool directly
(and other things) than have a specialized routing daemon out there that
has to be maintained and managed separately (from a process and failover
perspective). I've already got failover built into my collectors.
If there was a modular API I could use, despite the existance of
the routing thing, I'd use the API becaue it allows me to take advantage
of my failover without introducting more dependencies on SMTP working
and on the routing server working, and on mail delays during other
incidents.
And I think that you'll be able to come very close to that. At the worst,
you'd have to have a simple converter on your systems, so the tools that
spoke this format would pass it to a local converter which then converted
it and sent the data on its way.
I just know for a fact that one-off communication mechanisms will never make
any real progress, and I also know that a network API and format is the only
thing that's going to provide the kind of functionality I really need, which
is a way for all of the different tools to easily talk to each other,
preferably without knowing who they're talking to.
somebody else brought up the bidirectional problem already, I think.
Doesn't that mean that you need to have specialized SMTP servers
to handle receipt and parsing of messages for every tool? And you'd
need to open a new session for the reverse communication which makes
things really asynchronous unless you overload or define new error
codes (which now isn't really SMTP anymore anyway, it's just another
protocol using port 25.. like 80 is getting overloaded).

xmlrpc/soap is sounding better the more I think about it and see
the discussions.
Post by Luke Kanies
As I mentioned in my response to Ian's email, I don't even want my nodes to
have to know who they're talking to, and I don't want to make them
responsible for duplicating messages that multiple tools need. You might
have both a trending system and an alerting system that consume metrics;
with your system, every node on my network would have to send duplicate
metric messages, but with mine each node would forward the metric to the
router, and the router would duplicate and forward as necessary.
I see where you're going, but I'm becoming more convinced that SMTP
is probably not the best answer (whether or not a modular API is
the answer for puppet)
David Lutterkort
2005-11-30 22:40:30 UTC
Permalink
Post by Doug Hughes
Post by Luke Kanies
Well, I will certainly hope to provide libraries and hooks and such for
major languages, but, well, I'll be writing everything I do in Ruby, and I
don't think people would be very interested if I tried to publish a Ruby API
as the "standard" API.
I wouldn't be so sure. At least then it's there. (and it has the advantage of
being idiomatic)
To show and prove what can be done with this kind of integration, a Ruby
API within puppet would be completely sufficient. Eventually, there
needs to be a language-agnostic protocol, e.g., XML, that does the same
thing as the Ruby API.
Post by Doug Hughes
Post by Luke Kanies
I think it makes a lot of sense to have this central routing thing support
an API that supports simple modules, but that's different because that's an
implementation, not a spec.
I think, to be fair, you have to admit that you need both a spec for
your routing protocol and an API for the bus manipulation. No free lunch.
xmlrpc/soap is sounding better the more I think about it and see
the discussions.
It also gets much more attention as a generic communication mechanism
and has better tool support than SMTP.
Luke Kanies
2005-11-30 22:47:59 UTC
Permalink
Post by David Lutterkort
To show and prove what can be done with this kind of integration, a Ruby
API within puppet would be completely sufficient. Eventually, there
needs to be a language-agnostic protocol, e.g., XML, that does the same
thing as the Ruby API.
At the least, I'll be implementing the formats and a router. Puppet will
almost definitely be able to talk directly to the router, and I will do
everything possible to make the code that does so independently
distributable.
Post by David Lutterkort
Post by Doug Hughes
xmlrpc/soap is sounding better the more I think about it and see
the discussions.
It also gets much more attention as a generic communication mechanism
and has better tool support than SMTP.
Yeah, there is some benefit, and I'm already using XMLRPC in Puppet (I
started with SOAP, but, well, the SOAP libs in Ruby caused me some trouble).

If we were to decide on XMLRPC (or SOAP) then we have to come up with the
complete stack; I'm using webrick (a free web server distributed as part of
the stdlib in Ruby) with Puppet, but from what I've heard it won't scale
well, so it probably shouldn't be used as the heart of the router.
--
I happen to feel that the degree of a person's intelligence is directly
reflected by the number of conflicting attitudes she can bring to bear on the
same topic. -- Lisa Alther
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-11-30 22:41:29 UTC
Permalink
Post by Doug Hughes
Post by Luke Kanies
I'm nothing if not ambitious. :) I figure if I'm trying to build an
operating system abstraction layer (which is part of what Puppet is), I
might as well build the backbone for a sysadmin tool mesh.
amibitious, masochistic, other synomyms.. ;)
Yeah well, I've been waiting for years for someone else to fix these
problems.
Post by Doug Hughes
When I put on my parochial hat, I say solving cfengine's issues is somebody
else's problem. ;)
But I've got $10 that says there's some tool you use that will not be
updated to support this format, like, say, syslog.
Post by Doug Hughes
Post by Luke Kanies
Well, I will certainly hope to provide libraries and hooks and such for
major languages, but, well, I'll be writing everything I do in Ruby, and I
don't think people would be very interested if I tried to publish a Ruby API
as the "standard" API.
I wouldn't be so sure. At least then it's there. (and it has the advantage of
being idiomatic)
There will still be such an API; it seems likely that the router thing
itself will be written in Ruby and will have a cleanish API to enable ease
of writing plugins and such.
Post by Doug Hughes
I think, to be fair, you have to admit that you need both a spec for
your routing protocol and an API for the bus manipulation. No free lunch.
Yeah, I'm thinking the same thing.
Post by Doug Hughes
somebody else brought up the bidirectional problem already, I think.
Doesn't that mean that you need to have specialized SMTP servers
to handle receipt and parsing of messages for every tool? And you'd
need to open a new session for the reverse communication which makes
things really asynchronous unless you overload or define new error
codes (which now isn't really SMTP anymore anyway, it's just another
protocol using port 25.. like 80 is getting overloaded).
xmlrpc/soap is sounding better the more I think about it and see
the discussions.
Well, ideally the transport protocol won't matter at all, but either way, it
looks like SMTP is basically not going to be used.
--
The Washington Bullets are changing their name. The owners no longer
want their team's name to be associated with crime. So from now on the
team will be known as The Bullets.
-- Paul Harvey, quoting Argus Hamilton
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
James Neal
2005-11-30 18:53:25 UTC
Permalink
Post by Luke Kanies
I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
It also has the advantages of being proxiable (store-and-forward),
and passes through most firewalls. It's also pretty resliant to network
failures.

Some downsides:
- It can be slow. Sendmail by default will wait four hours before
retrying a failed send. You could do some reconfiguration on your
nodes, but it's possible your messages could be sent through some
system that you don't have administrative control over.

- It's non-ordered. Your monitoring system could get the "problem resolved"
alert before the "problem detected" alert. This can be dealt with some
by timestamps and/or sequence numbers.

- Imperfect countermeasures could pollute your stream. Because SMTP
has been so widely abused, there's all sorts of really crappy
countermeasures in place that could interfere with your message
stream. Your messages could be blocked, delayed, rerouted,
editted, or corrupted.

- SMTP's authentication and certificates don't help you a whole lot, when
your messages pass through multiple servers (which is one of the important
advantages of an SMTP message-bus architecture). You'll still need to
do S/MIME or some other inside-the-message work.

(In case this came off as too negative, I really, really, really like
SMTP as a messaging bus for interprocess communication. I've used it
at past jobs, and am implementing a similar system at my current place,
with SMTP, Jabber, or HTTP transports).

-James
Luke Kanies
2005-11-30 19:15:54 UTC
Permalink
Post by James Neal
It also has the advantages of being proxiable (store-and-forward),
and passes through most firewalls. It's also pretty resliant to network
failures.
Yup.
Post by James Neal
- It can be slow. Sendmail by default will wait four hours before
retrying a failed send. You could do some reconfiguration on your
nodes, but it's possible your messages could be sent through some
system that you don't have administrative control over.
Yeah, but this will mostly (at least for now) be used on LANs, where
resending isn't nearly as common.
Post by James Neal
- It's non-ordered. Your monitoring system could get the "problem resolved"
alert before the "problem detected" alert. This can be dealt with some
by timestamps and/or sequence numbers.
Yeah, that could be a problem.
Post by James Neal
- Imperfect countermeasures could pollute your stream. Because SMTP
has been so widely abused, there's all sorts of really crappy
countermeasures in place that could interfere with your message
stream. Your messages could be blocked, delayed, rerouted,
editted, or corrupted.
Yeah, this is definitely a problem. OTOH, we're mostly talking about
internal services that seldom carry any actual mail traffic, so you should
be able to get away with it.
Post by James Neal
- SMTP's authentication and certificates don't help you a whole lot, when
your messages pass through multiple servers (which is one of the important
advantages of an SMTP message-bus architecture). You'll still need to
do S/MIME or some other inside-the-message work.
Yeah, some sort of signing is definitely necessary.
Post by James Neal
(In case this came off as too negative, I really, really, really like
SMTP as a messaging bus for interprocess communication. I've used it
at past jobs, and am implementing a similar system at my current place,
with SMTP, Jabber, or HTTP transports).
Do tell. Even better, publish. :)

Would jabber be a good protocol for this? I think the store-and-forward
nature of SMTP is really beneficial for a lot of this stuff, but it seems
like we should support both an asynchronous and a synchronous form.
--
The whole secret of life is to be interested in one thing profoundly and in a
thousand things well. -- Horace Walpole
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
James Neal
2005-11-30 20:07:59 UTC
Permalink
Post by Luke Kanies
Yeah, but this will mostly (at least for now) be used on LANs, where
resending isn't nearly as common.
At least with sendmail, it'll stop accepting connections when the load
crosses a certain boundry. That can definitely happen if you're doing
a lot of mail processing with scripts.

(This is just from experience with a pretty heavy "data dumping over SMTP
system" that was in use at a previous company. It was sending gigs of
data, and doing a lot of processing though.)
Post by Luke Kanies
Yeah, this is definitely a problem. OTOH, we're mostly talking about
internal services that seldom carry any actual mail traffic, so you should
be able to get away with it.
I'm assuming these messages will be passing through one's normal mail
infrastructure. You could instead have these go through a subdomain
that has a different mail processing infrastructure. smtpmb.example.com,
or somesuch.
Post by Luke Kanies
Post by James Neal
(In case this came off as too negative, I really, really, really like
SMTP as a messaging bus for interprocess communication. I've used it
at past jobs, and am implementing a similar system at my current place,
with SMTP, Jabber, or HTTP transports).
Do tell. Even better, publish. :)
It's not very far along, and it's more focused on event management, and
less on RPC, which seems more like where you're going.

On the other hand, with YAML/microformats, you _could_ have one system
for both events and RPC...
Post by Luke Kanies
Would jabber be a good protocol for this? I think the store-and-forward
nature of SMTP is really beneficial for a lot of this stuff, but it seems
like we should support both an asynchronous and a synchronous form.
Jabber is also store-and-forward once it hits a jabber server, and
I _believe_ it's ordered delivery, but I'm not positive on that.

-James
Ian Blenke
2005-11-30 20:28:15 UTC
Permalink
Post by James Neal
At least with sendmail, it'll stop accepting connections when the load
crosses a certain boundry. That can definitely happen if you're doing
a lot of mail processing with scripts.
(This is just from experience with a pretty heavy "data dumping over SMTP
system" that was in use at a previous company. It was sending gigs of
data, and doing a lot of processing though.)
A "solution" is to use lockfiles so that the scripts aren't run in
parallel. A better solution would be to use a pool of processes or
threads to handle N number of messages simultaneously to keep the load down.
Post by James Neal
I'm assuming these messages will be passing through one's normal mail
infrastructure. You could instead have these go through a subdomain
that has a different mail processing infrastructure. smtpmb.example.com,
or somesuch.
That is actually for the best. If you forward your bulk mail through
your normal mail system, you will probably end up putting load on your
spam and virus scanners.
Post by James Neal
Post by Luke Kanies
Do tell. Even better, publish. :)
It's not very far along, and it's more focused on event management, and
less on RPC, which seems more like where you're going.
On the other hand, with YAML/microformats, you _could_ have one system
for both events and RPC...
One system to bind them all..
Post by James Neal
Post by Luke Kanies
Would jabber be a good protocol for this? I think the store-and-forward
nature of SMTP is really beneficial for a lot of this stuff, but it seems
like we should support both an asynchronous and a synchronous form.
Jabber is also store-and-forward once it hits a jabber server, and
I _believe_ it's ordered delivery, but I'm not positive on that.
Yes, XMPP (Jabber) is guaranteed ordered delivery, as it is store and
forward. The complexity of XMPP is a bit of a turnoff, however.

Granted, XMPP is a forerunner as a message bus.

Looking at the local Borders, developers seem to be heading toward an
Enterprise Service Bus (ESB) as an event driven engine for
Service-Oriented Architecture (SOA). The O'Reilly book hurt my head a
bit in the high-level diagrams, but the ideas seemed cogent to our problem.

From a systems perspective, we want something fairly simple to run and
troubleshoot. From a developer's perspective, they want something fairly
simple and reliable to develop to. The easier the latter, the easier to
develop connectors for the various subsystems and specific engines that
provide those subsystems.

- Ian C. Blenke <ian-LXTTh4l1wdzQT0dZR+***@public.gmane.org> http://ian.blenke.com/
Luke Kanies
2005-11-30 21:27:59 UTC
Permalink
[...]
Post by Ian Blenke
Post by James Neal
It's not very far along, and it's more focused on event management, and
less on RPC, which seems more like where you're going.
On the other hand, with YAML/microformats, you _could_ have one system
for both events and RPC...
One system to bind them all..
At this point we should probably somewhat clearly separate into discussing
the abstraction and the transport. I am probably not qualified to discuss
either side, but I am more clearly unqualified to discuss the transport.
I've used a few of the ones mentioned, and have written my own when
necessary, but it sounds like there are others with far more practical and
thorough experience.
Post by Ian Blenke
Yes, XMPP (Jabber) is guaranteed ordered delivery, as it is store and
forward. The complexity of XMPP is a bit of a turnoff, however.
Complexity is bad, but if it can be hidden in libraries, it might be
acceptable.
Post by Ian Blenke
Granted, XMPP is a forerunner as a message bus.
Looking at the local Borders, developers seem to be heading toward an
Enterprise Service Bus (ESB) as an event driven engine for
Service-Oriented Architecture (SOA). The O'Reilly book hurt my head a
bit in the high-level diagrams, but the ideas seemed cogent to our problem.
From a systems perspective, we want something fairly simple to run and
troubleshoot. From a developer's perspective, they want something fairly
simple and reliable to develop to. The easier the latter, the easier to
develop connectors for the various subsystems and specific engines that
provide those subsystems.
Yeah, exactly. I do want to add one caveat: We are developing this, at
least for now, mostly for sysadmins-as-developers. Look at most of the
tools we use -- at least in numbers, they are simplistic tools that were
developed by one or two sysadmins, and few if any of them have are
maintained full time by a "real" developer.

It definitely shouldn't cause actual development shops to vomit, but I'm
much more interested in sysadmins being able to use it to hook their tools
together and to develop other ad-hoc tools than I am in using it to hook up
a bunch of expensive tools. I've been at way too many shops that have spent
tons of money on these large-scale network tools but can't use them because
they're too complicated. I don't want to follow that mentality; let's start
as small as we can, and get bigger only when we must.
--
Q: How many surrealists does it take to screw in a lightbulb?
A: Two. One to hold the giraffe and the other to fill the
bathtub with brightly colored machine tools.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-11-30 21:31:11 UTC
Permalink
On Wed, 30 Nov 2005, Ian Blenke wrote:

[...]
Post by Ian Blenke
Looking at the local Borders, developers seem to be heading toward an
Enterprise Service Bus (ESB) as an event driven engine for
Service-Oriented Architecture (SOA). The O'Reilly book hurt my head a
bit in the high-level diagrams, but the ideas seemed cogent to our problem.
By the way, in looking this up online, ESB seems exactly like CIM: A huge,
vague buzzword.

See this article for an example:

http://www.ebizq.net/hot_topics/esb/features/6132.html
--
Q: How many surrealists does it take to screw in a lightbulb?
A: Two. One to hold the giraffe and the other to fill the
bathtub with brightly colored machine tools.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
John Rouillard
2005-11-30 20:29:32 UTC
Permalink
Post by Luke Kanies
Post by James Neal
It also has the advantages of being proxiable (store-and-forward),
and passes through most firewalls. It's also pretty resliant to network
failures.
Yup.
Post by James Neal
- It can be slow. Sendmail by default will wait four hours before
retrying a failed send. You could do some reconfiguration on your
nodes, but it's possible your messages could be sent through some
system that you don't have administrative control over.
Yeah, but this will mostly (at least for now) be used on LANs, where
resending isn't nearly as common.
I am looking at using puppet (or config or cfengine) to manage
multiple servers seperated by thousands of miles. We do have smtp
delays and queuing of messages.

As far as reusing the data, I like an API that allows you to export
the data in a defined form (XML maybe). That way you can perform a
query for building a nagios config that says:

show me all hosts running apache at the boston site

and get:

host1 host2 host3 host5

Then I can take that data and set up nagios groups to perform the
monitoring by defining each host to have the appropriate attributes
(member of hostgroups "apache" and "boston") in nagios.

That's one of the reasons 'config' is run off of a "database"
file. Change the host entry in the database file and it automatically
generates the appropriate classes to distribute the needed files.

Also it can export as CSV format as well for data interchange.

The primary underlying mechanism for config was rdist, so update
information wcould be sent via email to a majordomo mailing list.

I look forward to meeting you at LISA next week.
--
-- rouilj

John Rouillard
System Administrator
Renesys Corporation
603-643-9300 x 111
Luke Kanies
2005-11-30 22:52:53 UTC
Permalink
Post by John Rouillard
I am looking at using puppet (or config or cfengine) to manage
multiple servers seperated by thousands of miles. We do have smtp
delays and queuing of messages.
Okay, okay, we won't use SMTP. :)
Post by John Rouillard
As far as reusing the data, I like an API that allows you to export
the data in a defined form (XML maybe). That way you can perform a
show me all hosts running apache at the boston site
host1 host2 host3 host5
Then I can take that data and set up nagios groups to perform the
monitoring by defining each host to have the appropriate attributes
(member of hostgroups "apache" and "boston") in nagios.
In this case you would want to create a tool that subscribed to the data and
stored the output in a database. I plan on keeping the formats as simple
key/value pairs, so they should be downright trivial to convert to
databases.
Post by John Rouillard
That's one of the reasons 'config' is run off of a "database"
file. Change the host entry in the database file and it automatically
generates the appropriate classes to distribute the needed files.
Is this 'config' a tool that I don't know about? If not, I'm confused as to
what you mean.
Post by John Rouillard
Also it can export as CSV format as well for data interchange.
Data interchange should be really easy for this.
Post by John Rouillard
The primary underlying mechanism for config was rdist, so update
information wcould be sent via email to a majordomo mailing list.
I look forward to meeting you at LISA next week.
Great. Maybe we should organize a BoF to hack on this some. I've already
got a Puppet BoF (and a config-mgmt BoF), but this might be fun, too.
--
There is nothing worse than aggressive stupidity.
-- Johann Wolfgang von Goethe
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Ian Blenke
2005-11-30 19:26:05 UTC
Permalink
Post by Luke Kanies
Post by Doug Hughes
The module can/should take care of all of that for you.
e.g. your nagios module sends an SNMP trap.. your RT module might
use CLI to manipulate tickets, your mysql module injects the
method into a DB log table, etc.
It sounds like you're saying here that my module literally does these things
(modify a table, send a trap), which I most definitely do not want. That's
not a network API, which is specifically what I'm looking for. System-level
abstractions are not sufficient, I want network-level abstractions. I don't
want to care what kind of system is on the other end of the API, I just want
to care what kind of data it can accept. Can you take trouble tickets?
Great, here's one. Can't handle metrics? OK, who can?
Code can do anything. Sure, you can make a pluggable module extension
for any engine, and have the code do the actual manipulation between
disparate systems... but this locks your system into that programming
language, and that engine. Developers hate being locked into a
language, or being forced to deal with the code of a system they didn't
write.

What we're really dealing with here is transforming data. Data output
from one component in the system fed as data into another alien system.
Sure, you can write data translators between any two systems, but that
doesn't address the other problems like transport.

I think what Luke is looking for is a generic way to transport data
between different engines running in a system over a network.

There are two problems here: data abstraction, and data transport. A
message bus lets you enforce both.
Post by Luke Kanies
I understand that message busses are hard, which is why I was positing the
use of SMTP as a message bus, since it's already done, absolutely pervasive,
fast, and stable.
- Transport -

Message busses have various properties that make this a complex problem.
Sequential delivery is usually important for messages. Reliable delivery
of messages is usually another.

SMTP is often not bi-directional. I have systems distributed all over
the planet, often behind customer firewalls. This means those systems
can send email out via some relay, but return messages _to_ those
systems will never make it.

If I cant ACK messages received at the other end, how do I know I can
safely delete the messages from the bus sending queue after they are
sent via SMTP? If I haven't received a message within a timeout window,
how do I tell the other side to re-send a message from the sending queue?

In cases like this, you can work things out with a "touch" directory
mirroring the queue files representing each message, and some contrived
method of rsync over ssh _back_ to the system in question to mark
messages as having been delivered.

SMTP, if email delivery is bidirectional, can be made into a two-way
message bus. If unidirectional, you end up with a unreliable (no ACK, no
resends) "fire and forget" message delivery method, very UDP like.

On the other hand, I can ssh into any server in the field from trusted
management IPs. Were the message bus to open up ssh sessions (nailed up
with ssh4 connection sharing) to connect to a remote "message bus agent"
and stream across objects that way, I could get reliable sequential
delivery. Scaling would then be a matter of setting up a series of
trusted interim "message routers" with ssh sockets open to every server
on the bus.

This is why a message bus must be transport agnostic if it is to gain
full support by the largest audience.

Don't get me wrong, SMTP is a wonderful message transport, if you can
get it to work.

- Abstraction -

The other problem we are solving with this is the actual message data
abstraction.

Each message should have a "message type" as well as a "type version"
(as the message types will usually gain fields over time as new
requirements are found). Ideally, you would permit user-defined fields
as well, should they want to add their own "X-*" style fields to the
messages without making that change formal.

Each message type will be a data abstraction of some system event or
another, be it a syslog message, apache log snippet, system stats, or
netflow data.

The message needs to know how to be marshalled at each end. You could
make it raw binary, XML, a microformat, YAML, or some other simplistic
data representation (which I think is what Luke is looking to do).

Each message should have a destination queue address (something beyond
the underlying message transport endpoints), as well as a sequence
number for reliable and sequential delivery.

Some messages are more important than others, so you probably want a
priority level and some terms of service for the underlying transport.

- Publish/Subscribe -

Another common problem is "pubsub" (publish/subscribe), where you want
many systems to subscribe to specific message feeds for distribution of
data in a large cluster. With SMTP it's easy enough to add some aliases
or setup a list server.

This could go on and on...

- Ian C. Blenke <ian-LXTTh4l1wdzQT0dZR+***@public.gmane.org> http://ian.blenke.com/
Luke Kanies
2005-11-30 21:02:30 UTC
Permalink
Post by Ian Blenke
Code can do anything. Sure, you can make a pluggable module extension
for any engine, and have the code do the actual manipulation between
disparate systems... but this locks your system into that programming
language, and that engine. Developers hate being locked into a
language, or being forced to deal with the code of a system they didn't
write.
What we're really dealing with here is transforming data. Data output
from one component in the system fed as data into another alien system.
Sure, you can write data translators between any two systems, but that
doesn't address the other problems like transport.
I think what Luke is looking for is a generic way to transport data
between different engines running in a system over a network.
Basically, yes, although that summary implies that I desire this for generic
data where the rest of the email makes clear this is not what you mean.
Post by Ian Blenke
There are two problems here: data abstraction, and data transport. A
message bus lets you enforce both.
[...]
- Transport -
Message busses have various properties that make this a complex problem.
Sequential delivery is usually important for messages. Reliable delivery
of messages is usually another.
While I agree with this in theory, my primary goals are to enable this
communication in a way that's not currently possible and for most cases to
do so in a way that's superior to what little does exist. I would *much*
rather compromise on things like reliability than simplicity, especially in
the bus itself.

I think getting something available soon is far more important than getting
perfection available some day.
Post by Ian Blenke
SMTP is often not bi-directional. I have systems distributed all over
the planet, often behind customer firewalls. This means those systems
can send email out via some relay, but return messages _to_ those
systems will never make it.
If I cant ACK messages received at the other end, how do I know I can
safely delete the messages from the bus sending queue after they are
sent via SMTP? If I haven't received a message within a timeout window,
how do I tell the other side to re-send a message from the sending queue?
Yeah, that's a great point that I hadn't thought about, and people will
generally be pretty darn hesitant to open up firewalls for general SMTP,
regardless of what you're doing with it.
Post by Ian Blenke
In cases like this, you can work things out with a "touch" directory
mirroring the queue files representing each message, and some contrived
method of rsync over ssh _back_ to the system in question to mark
messages as having been delivered.
Mmm, contrived.

While it'd definitely be nice to support this kind of thing, especially in
the sysadmin world where we can always depend on ssh and generally not much
else, I would generally prefer to keep that a degenerate case, especially
since I want to build this as a hub and spoke system, instead of point to
point, mostly because of the publish/subscribe stuff you mention below.
Post by Ian Blenke
SMTP, if email delivery is bidirectional, can be made into a two-way
message bus. If unidirectional, you end up with a unreliable (no ACK, no
resends) "fire and forget" message delivery method, very UDP like.
On the other hand, I can ssh into any server in the field from trusted
management IPs. Were the message bus to open up ssh sessions (nailed up
with ssh4 connection sharing) to connect to a remote "message bus agent"
and stream across objects that way, I could get reliable sequential
delivery. Scaling would then be a matter of setting up a series of
trusted interim "message routers" with ssh sockets open to every server
on the bus.
This is why a message bus must be transport agnostic if it is to gain
full support by the largest audience.
Yeah, I agree on this. Hmm.
Post by Ian Blenke
Don't get me wrong, SMTP is a wonderful message transport, if you can
get it to work.
- Abstraction -
The other problem we are solving with this is the actual message data
abstraction.
Each message should have a "message type" as well as a "type version"
(as the message types will usually gain fields over time as new
requirements are found). Ideally, you would permit user-defined fields
as well, should they want to add their own "X-*" style fields to the
messages without making that change formal.
This immediately makes me think of LDAP schemas, where there is a set of
base objects, simple inheritance, and easily customized schemas.

That might not be a bad model -- you have a simple schema definition
(preferably easier to read than LDAP's is), and if all of your objects are
just key/value pairs, it shouldn't be too hard. I kinda like this ad-hoc
form better than Microformats, even, because their model is (somewhat
necessarily) a bit too ivory-tower for me.

Hmmm.
Post by Ian Blenke
Each message type will be a data abstraction of some system event or
another, be it a syslog message, apache log snippet, system stats, or
netflow data.
Exactly.
Post by Ian Blenke
The message needs to know how to be marshalled at each end. You could
make it raw binary, XML, a microformat, YAML, or some other simplistic
data representation (which I think is what Luke is looking to do).
I guess I would say that it is not yet an abstraction until it has been
marshalled, or rather that marshalling it converts it to the abstract form,
but yet.
Post by Ian Blenke
Each message should have a destination queue address (something beyond
the underlying message transport endpoints), as well as a sequence
number for reliable and sequential delivery.
I hadn't really thought of sequence before people on the thread mentioned
it, but I suppose it's necessary. It does make me think of all of the
sequence prediction attacks in TCP, though.

As to destination addresses, though, I think for most cases you will
definitely not want those. Why should all of my nodes know or care who
exactly is monitoring them? They should just be able say, "Hey, I've got a
web service that needs to be monitored", and the system should be able to
route that message appropriately.

Sure, it's somewhat reasonable to support an optional destination address,
but for all of the message types we've discussed so far, I would _much_
rather make that routing decision centrally than on every single node.
Post by Ian Blenke
Some messages are more important than others, so you probably want a
priority level and some terms of service for the underlying transport.
I probably want those... eventually.

We should probably start drawing up a spec of what we need/want/would be
cool, because some of this stuff obviously needs to be there in the
beginning but some of it can be tacked on later, and again, I'd rather get
it working and useful than wait for it to be perfect.
Post by Ian Blenke
- Publish/Subscribe -
Another common problem is "pubsub" (publish/subscribe), where you want
many systems to subscribe to specific message feeds for distribution of
data in a large cluster. With SMTP it's easy enough to add some aliases
or setup a list server.
This is something that I think could be a killer application for this
system. As mentioned above, I say skip the whole destination address in the
first place and ship a simple routing node that supports plain routing based
on subscriptions, for those destinations that support it, plus convert/route
for those systems that don't.

For example (ignoring all the messy problems like capacity planning and
service reliability for now), take a network that has Netcool, Puppet, and
custom apps for centralized logging and trending. You build your central
server, configure each of your nodes so that it sends all of their messages
to that central node, and then you configure the central node to convert and
route alerts to Netcool, route metrics to the custom trending app (which,
because it's custom, was easily modified to accept messages of this type),
and convert logs and just store them directly in the central log database.

I can already imagine the config file for that central node, and it's not
that complicated. It should be easy enough to develop it so that you could
run one big one at the center of your network and have everyone talk
directly to it, or have one simple one on each node and use CLI tools to
talk to it, or any combination in between.

Yes, I willfully ignored reliability and capacity planning, but as long as
the router didn't need any state, you could easily stick a bunch of these
behind a load balancer, and/or make sure that part of the spec requires the
ability to specify multiple routing nodes, just like DNS supports multiple
resolvers.
Post by Ian Blenke
This could go on and on...
Yes, quite. :)
--
Basic research is what I am doing when I don't know what I am doing.
--Wernher von Braun
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Doug Hughes
2005-11-30 18:10:39 UTC
Permalink
Post by Luke Kanies
Hi all,
I've been working on Puppet[1] for a while now, and while it's coming along
nicely and I'm starting to collect clients who will use it, I'm also
thinking about how Puppet will interact with the rest of the network.
One of the things that's always driven me nuts about sysadmin tools is that
they don't talk to each other. I've thought a lot about this in Puppet's
development, and initially it seemed like I'd end up writing a lot of hooks
into the different tools, so that, for instance, Puppet would know how to
talk to Nagios and RT and a few others.
In thinking about it, though, that sounds like a pain, because I would never
be able to support all of the tools out there, and no one else could reuse
the connecting work I did.
So, I've been thinking about how I can decouple things so that I get the
end result of being able to integrate with these tools but in a way that's
reusable and also allows others to implement one side of a hook (e.g.,
support for a specific application) in a way that I can use with Puppet.
I started thinking about the kind of data I want to exchange, and most if it
is pretty darn simple: log messages, metrics, alerts, trouble tickets,
things like that. Basically, if two sysadmin tools _should_ be talking, it
seems like it is almost always pretty easy to come up with a simple data
concept for that talking. For instance, Puppet will be producing metrics,
notifying Nagios that service X on machine Y should be monitored, and will
be opening trouble tickets when it encounters a failure.
So, can we come up with a standard way of talking about those little data
formats, such that all of our sysadmin apps can support some subset of these
objects and accept them over the wire?
Initially, it looked like Microformats[2] was the answer, but it turns out
that those really only exist for web browser documents, and their existence
is driven by the fact that HTTP header tags are limited. Or something. The
point is, they aren't really useful for non-HTML data exchange.
So, what else is there? Well, we in the Unix world are already constantly
using a simple messaging format, SMTP, so I am thinking maybe we could
produce formats akin to microformats but built to ride on SMTP instead of
HTTP. It's generally pretty damn easy to configure Unix systems to hand
messages off to an external program -- I know Postfix and Sendmail support
this out of the box, and I'm sure QMail does, also. So the question is,
could we create something like a little microformats router, in that the
SMTP server accepts the objects, sends them to our router, and then our
router passes them off to Nagios, or RT, or whatever.
It might be that these apps should thus learn to speak SMTP natively, but,
well, that doesn't really seem necessary. SMTP is well-known, supports
everything I can think we would need (authentication, certificates, etc.),
and is already very fast and very stable. Best of all, there are multiple
fast and stable servers out there, and they all know how to hand data off to
an external program, which is all we need here.
First, would something like this be useful?
Second, would you use it, and potentially even contribute to it?
Third, how should we go about creating it? Should the formats be special
SMTP messages, just SMTP attachments, something entirely different, or what?
I'll contact a few of the developers for key related projects, like RT and
hopefully Nagios, but if anyone else can think of developers who should be
informed, please ping them with this discussion. I'm also going to ping the
microformats people and see if they're interested in trying to do SMTP based
formats, or supporting both SMTP and HTTP, so that I can reuse their
structure.
Really, I'm serious here -- I'm going to do something here, and I'm willing
to stand up and lead the effort, but I'd like some feedback.
Well, if I think of it in terms of how other tools do it then I have
to answer a modules API. Most things handle this in terms of dynamically
loadable .so files, but that might be overkill here, or not fit inside
the idiom you're working in. Still, having an API to define modules
that people could write to arbitrarily extend to other things via
message passing or whatever would be useful. Then you could provide
some examples like RT and Nagios and people could run with it.
maybe all modules must have a <modulename>_init method that defines
callbacks or whatever.

(e.g. Apache, php, tcl, perl, gaim, etc)

There's a lot of examples done 'right' out there.
Luke Kanies
2005-11-30 18:31:16 UTC
Permalink
Post by Luke Kanies
So, what else is there? Well, we in the Unix world are already constantly
using a simple messaging format, SMTP, so I am thinking maybe we could
produce formats akin to microformats but built to ride on SMTP instead of
HTTP. It's generally pretty damn easy to configure Unix systems to hand
messages off to an external program -- I know Postfix and Sendmail support
this out of the box, and I'm sure QMail does, also. So the question is,
Yes, qmail supports this trivially, as does exim.
Yep, figured; thanks for the confirmation.
RT certainly has bidirectional mail interfaces. Nagios does
outbound notifications, but I don't think it has any inbound
parsing. The same is true for mon and monit.
In order for anyone to use RT's interface, though, you need to specifically
write to it. If I decide that I want to replace RT with Remedy or whatever,
I now have to change all of my ticket generation tools, which is what I'd
like to avoid.
Oh yes. When I generate a new box, I would like to be able to
have my monitoring system automatically start monitoring it --
we've been working towards that by having our installer also
generate mon stanzas.
Yeah, this is exactly the kind of thing I'm trying to provide.
Post by Luke Kanies
Second, would you use it, and potentially even contribute to it?
Of course.
Heh, so casually said. Don't think I won't hold you to that. :)
Post by Luke Kanies
Third, how should we go about creating it? Should the formats be special
SMTP messages, just SMTP attachments, something entirely different, or what?
The easiest method is to use the existing X-Header mechanism to
define variable/data pairs, and use the message body for passing
explanatory text, human-readable warnings and such.
Alternatively, define an XML spec and put everything into the
message body.
------
Subject: New webserver-ext-corp
X-MR-IP: 10.1.22.3
X-MR-DNS-name: web33.ext.management.com
X-MR-please-monitor: ping, apache-80, apache-443, ssh-22, snmp-disk-SLASH, snmp-process-crond, snmp-process-localtest, ntp-sync, smtp-25-block
I don't think this will work well, because at the least it rules out the
ability to send more than one object in a given message. To compare, here
is how the microformat version of vCard looks:

<div class="vcard">
<a class="url fn" href="http://tantek.com/">
Tantek Çelik
</a>
<div class="org">Technorati</div>
</div>

Obviously, you can take as many of these as you want and tack them together.
[...]
Parsing is easy, but inflexible; it's easy to accidentally pass
unreadable gublick along. There's no authentication. If there's
a well-known spec and you accidentally let it be known that you
have a conformant message-router, you may be allowing random
strangers to muck with your infrastructure remotely.
Yeah, your examples have convinced me that the formats should not be in SMTP
header form. Hmmm. I'll contact the microformats guys (they've got a
mailing list) and see what they think. Maybe we can use their same system
and just mark the objects as attachments to SMTP messages. This certainly
allows us to keep the format separate from the transport.
--
The Washington Bullets are changing their name. The owners no longer
want their team's name to be associated with crime. So from now on the
team will be known as The Bullets.
-- Paul Harvey, quoting Argus Hamilton
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Brendan Strejcek
2005-11-30 19:16:53 UTC
Permalink
Post by Luke Kanies
In order for anyone to use RT's interface, though, you need to
specifically write to it. If I decide that I want to replace RT with
Remedy or whatever, I now have to change all of my ticket generation
tools, which is what I'd like to avoid.
What you are proposing sounds like creating a new protocol (even if
running on top of SMTP, it is still new), and modifying all possible
senders and recievers to be able to speak this protocol. That's a lot
of work, and not all project maintainers will be amenable. Further, it
adds N units of complexity, and with M participants, that is N x M extra
units of complexity.

Instead, put something between them (cf proxy pattern). Model the
data bits you care about (log messages, trouble tickets, etc) and
create input and output modules for the proxy. Tools like puppet will
be completely insulated from the vagaries of tools like Nagios or
RT. The downside of this is that you are, in some sense, continually
reverse-engineering the interfaces.

SMTP may be time-tested, but in my experience it is one of the harder
services/protocols to really manage correctly. On the other hand, the
reliability of SMTP queuing would be nice for a system like this. Back
to the first hand, getting the PKI (or whatever) necessary to actually
secure a system like this would not be easy, either to conceptualize or
for an end-user sysadmin to get right.

I think the really interesting bit of your proposal is the recognition
that the data you are interested in is simple. Do you have a model?

Also, have you looked into CIM?

http://www.dmtf.org/standards/cim/

Sounds similar to what you are talking about, though I don't claim to
really understand it, and I would beware of committee-induced protocol
bloat.

Best,
Brendan

--
Senior System Administrator
The University of Chicago
Department of Computer Science

http://www.cs.uchicago.edu/people/brendan

http://people.cs.uchicago.edu/~brendan
Luke Kanies
2005-11-30 19:28:23 UTC
Permalink
Post by Brendan Strejcek
What you are proposing sounds like creating a new protocol (even if
running on top of SMTP, it is still new), and modifying all possible
senders and recievers to be able to speak this protocol. That's a lot
of work, and not all project maintainers will be amenable. Further, it
adds N units of complexity, and with M participants, that is N x M extra
units of complexity.
Well, I'm hoping to just define a new payload for an existing protocol, but
there will always need to be those extra bits like signing and such that get
added, so I'm sure it will end up looking a lot like a new protocol.
Post by Brendan Strejcek
Instead, put something between them (cf proxy pattern). Model the
data bits you care about (log messages, trouble tickets, etc) and
create input and output modules for the proxy. Tools like puppet will
be completely insulated from the vagaries of tools like Nagios or
RT. The downside of this is that you are, in some sense, continually
reverse-engineering the interfaces.
That's exactly what I'm talking about, except that the proxy is basically
the bus system itself. Tools like Puppet that natively support the formats
will just write their objects directly to the bus, while tools like RT and
Netcool that don't (yet) natively support the formats will need a
translating proxy in front.

This is one of the reasons I think SMTP might work well, though -- most
tools already speak it, and those that don't almost all speak popen() which
can be a sufficient proxy for speaking smtp.

I can't think of any other protocol that is as pervasively deployed and used
as SMTP is.
Post by Brendan Strejcek
SMTP may be time-tested, but in my experience it is one of the harder
services/protocols to really manage correctly. On the other hand, the
reliability of SMTP queuing would be nice for a system like this. Back
to the first hand, getting the PKI (or whatever) necessary to actually
secure a system like this would not be easy, either to conceptualize or
for an end-user sysadmin to get right.
Yeah, that's true, but we couldn't possibly do security worse than it's
being done in management tools right now (because it's not really done at
all), and these are just simple strings, so signing those strings with a
keypair should be sufficient for nearly all purposes. Then it's a question
of just getting the gateways and endpoints configured correctly; the bus
shouldn't need to know or care whether its payload is signed or not.
Post by Brendan Strejcek
I think the really interesting bit of your proposal is the recognition
that the data you are interested in is simple. Do you have a model?
Not yet.
Post by Brendan Strejcek
Also, have you looked into CIM?
http://www.dmtf.org/standards/cim/
Sounds similar to what you are talking about, though I don't claim to
really understand it, and I would beware of committee-induced protocol
bloat.
Ah, CIM, "the solution that's been looking for a problem since 1999"(tm).

Yeah, I'm aware of it, but I'm not aware of, um, anyone really taking
advantage of it, and committee-induced bloat is definitely a problem here.

One of the things I like best about the term 'microformats' is the 'micro'
part.

I'm a big believer in starting with simple ad-hoc standards and running up
from there (like RSS did), rather than starting with nastily huge standards
and trying to scale them down.
--
A citizen of America will cross the ocean to fight for democracy, but
won't cross the street to vote in a national election.
--Bill Vaughan
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Alex Honor
2005-11-30 19:04:56 UTC
Permalink
I am supportive of any effort that will provide a standard way to
allow communication between admin tools.

It is important to separate transport (eg, SMTP, HTTP, etc.) from
content (eg, HTML, mail attachments). Of these two concerns,
establishing the requirements of message content is more important.
For example, there are roughly two ways content can be handled:

1) The content (ie, it's structure, encoding, etc) is only
interpreted by the sender and receiver with some communication header
providing for addressing. The advantage to this strategy is the only
agreement necessary is on the communication requirements. The big
disadvantage is that this could lead to a proliferation of formats.
This disadvantage can be offset by agreeing on basic message encoding.

2) The content is based on a common format that supports a general
set of information in common between all tools. The advantage to this
strategy is there is just one format to support. The disadvantage is
inflexibility.

It would be interesting to create a standard gateway that can be
placed in front of any tool to manage all interactions.
Post by Luke Kanies
Hi all,
I've been working on Puppet[1] for a while now, and while it's
coming along
nicely and I'm starting to collect clients who will use it, I'm also
thinking about how Puppet will interact with the rest of the network.
One of the things that's always driven me nuts about sysadmin tools is that
they don't talk to each other. I've thought a lot about this in Puppet's
development, and initially it seemed like I'd end up writing a lot of hooks
into the different tools, so that, for instance, Puppet would know how to
talk to Nagios and RT and a few others.
In thinking about it, though, that sounds like a pain, because I would never
be able to support all of the tools out there, and no one else
could reuse
the connecting work I did.
So, I've been thinking about how I can decouple things so that I get the
end result of being able to integrate with these tools but in a way that's
reusable and also allows others to implement one side of a hook (e.g.,
support for a specific application) in a way that I can use with Puppet.
I started thinking about the kind of data I want to exchange, and most if it
is pretty darn simple: log messages, metrics, alerts, trouble tickets,
things like that. Basically, if two sysadmin tools _should_ be talking, it
seems like it is almost always pretty easy to come up with a simple data
concept for that talking. For instance, Puppet will be producing metrics,
notifying Nagios that service X on machine Y should be monitored, and will
be opening trouble tickets when it encounters a failure.
So, can we come up with a standard way of talking about those
little data
formats, such that all of our sysadmin apps can support some subset of these
objects and accept them over the wire?
Initially, it looked like Microformats[2] was the answer, but it turns out
that those really only exist for web browser documents, and their existence
is driven by the fact that HTTP header tags are limited. Or
something. The
point is, they aren't really useful for non-HTML data exchange.
So, what else is there? Well, we in the Unix world are already constantly
using a simple messaging format, SMTP, so I am thinking maybe we could
produce formats akin to microformats but built to ride on SMTP
instead of
HTTP. It's generally pretty damn easy to configure Unix systems to hand
messages off to an external program -- I know Postfix and Sendmail support
this out of the box, and I'm sure QMail does, also. So the
question is,
could we create something like a little microformats router, in that the
SMTP server accepts the objects, sends them to our router, and then our
router passes them off to Nagios, or RT, or whatever.
It might be that these apps should thus learn to speak SMTP
natively, but,
well, that doesn't really seem necessary. SMTP is well-known,
supports
everything I can think we would need (authentication, certificates, etc.),
and is already very fast and very stable. Best of all, there are multiple
fast and stable servers out there, and they all know how to hand data off to
an external program, which is all we need here.
First, would something like this be useful?
Second, would you use it, and potentially even contribute to it?
Third, how should we go about creating it? Should the formats be special
SMTP messages, just SMTP attachments, something entirely different, or what?
I'll contact a few of the developers for key related projects, like RT and
hopefully Nagios, but if anyone else can think of developers who should be
informed, please ping them with this discussion. I'm also going to ping the
microformats people and see if they're interested in trying to do SMTP based
formats, or supporting both SMTP and HTTP, so that I can reuse their
structure.
Really, I'm serious here -- I'm going to do something here, and I'm willing
to stand up and lead the effort, but I'd like some feedback.
Thanks,
Luke
1 - http://reductivelabs.com/projects/puppet
2 - http://www.microformats.org
--
I object to doing things that computers can do.
--Olin Shivers
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-11-30 21:16:39 UTC
Permalink
Post by James Neal
At least with sendmail, it'll stop accepting connections when the load
crosses a certain boundry. That can definitely happen if you're doing
a lot of mail processing with scripts.
That's a good point; I think you've mostly convinced that it's probably not
the right protocol. Ian's point about the unidirectional way that most
people use SMTP in DMZs and such is also key.
Post by James Neal
(This is just from experience with a pretty heavy "data dumping over SMTP
system" that was in use at a previous company. It was sending gigs of
data, and doing a lot of processing though.)
Ah, good to know.
Post by James Neal
I'm assuming these messages will be passing through one's normal mail
infrastructure. You could instead have these go through a subdomain
that has a different mail processing infrastructure. smtpmb.example.com,
or somesuch.
Well, they're kind of passing through the normal infrastucture, but they're
entirely generated from internal sources, so you could certainly get away
with doing things like skipping spam checking (although it would be somewhat
hilarious to have some group's systems produce such useless information that
they be considered spam).
Post by James Neal
It's not very far along, and it's more focused on event management, and
less on RPC, which seems more like where you're going.
On the other hand, with YAML/microformats, you _could_ have one system
for both events and RPC...
Exactly. I'm actually not thinking of this for RPC and much more for a kind
of fire-and-forget event management. "Here's an event/log/metric, please
send it to the right place."
Post by James Neal
Jabber is also store-and-forward once it hits a jabber server, and
I _believe_ it's ordered delivery, but I'm not positive on that.
Interesting. I've not used Jabber, but it sounds like a bit of a candidate,
and it would certainly be easy to set it up out of band, which would be far
more difficult for SMTP.
--
I think that all good, right thinking people in this country are sick
and tired of being told that all good, right thinking people in this
country are fed up with being told that all good, right thinking people
in this country are fed up with being sick and tired. I'm certainly
not, and I'm sick and tired of being told that I am.
-- Monty Python
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Kris Buytaert
2005-11-30 21:46:44 UTC
Permalink
Post by Luke Kanies
So, what else is there? Well, we in the Unix world are already constantly
using a simple messaging format, SMTP, so I am thinking maybe we could
produce formats akin to microformats but built to ride on SMTP instead of
HTTP. It's generally pretty damn easy to configure Unix systems to hand
messages off to an external program -- I know Postfix and Sendmail support
this out of the box, and I'm sure QMail does, also. So the question is,
could we create something like a little microformats router, in that the
SMTP server accepts the objects, sends them to our router, and then our
router passes them off to Nagios, or RT, or whatever.
It might be that these apps should thus learn to speak SMTP natively, but,
well, that doesn't really seem necessary. SMTP is well-known, supports
everything I can think we would need (authentication, certificates, etc.),
and is already very fast and very stable. Best of all, there are multiple
fast and stable servers out there, and they all know how to hand data off to
an external program, which is all we need here.
Lots of people seem to look at Jabber as a message transport protocol
for similar things these days.

Just my 2eurocents ;)
--
Kris Buytaert <mlkb-1zPVpAQ/vZCzQB+***@public.gmane.org>
David Lutterkort
2005-11-30 22:28:47 UTC
Permalink
Post by Kris Buytaert
Post by Luke Kanies
It might be that these apps should thus learn to speak SMTP natively, but,
well, that doesn't really seem necessary. SMTP is well-known, supports
everything I can think we would need (authentication, certificates, etc.),
and is already very fast and very stable. Best of all, there are multiple
fast and stable servers out there, and they all know how to hand data off to
an external program, which is all we need here.
Lots of people seem to look at Jabber as a message transport protocol
for similar things these days.
I think it is very important to keep the issues of transport and
protocol separated here. In my mind the hard part of the proposal is
coming up with the protocol/API, and making sure it can is usable by as
many tools as possible without turning into a bloated monster.

So, for puppet/monitoring integration, it might be better to focus on
integrating with a couple of monitoring tools than to try and accomodate
everything anybody can think of.

David
Luke Kanies
2005-11-30 22:45:03 UTC
Permalink
Post by David Lutterkort
I think it is very important to keep the issues of transport and
protocol separated here. In my mind the hard part of the proposal is
coming up with the protocol/API, and making sure it can is usable by as
many tools as possible without turning into a bloated monster.
I say we start with the format, then talk about the protocol, and if
necessary talk about the transport.

If tools only use the format and not the protocol, that's fine, because at
least those tools will be modeling their data in the same way.
Post by David Lutterkort
So, for puppet/monitoring integration, it might be better to focus on
integrating with a couple of monitoring tools than to try and accomodate
everything anybody can think of.
Yeah, that's my plan; I certainly don't plan on implementing anything beyond
what my customers are specifically asking for, but I definitely want to make
it easy for people to hack up their own support for their own tools. I
guess my main point is that I don't want puppet-only integration anywhere.
--
Somebody has to do something, and it's just incredibly pathetic that it
has to be us. --Jerry Garcia
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-12-02 05:30:07 UTC
Permalink
[your sig]
Post by Luke Kanies
Somebody has to do something, and it's just incredibly pathetic that it
has to be us. --Jerry Garcia
How apropos :).
Yeah; I actually use that for the byline of one of the slides of my Puppet
presentation.
--
Reality is that which, when you stop believing in it, doesn't go
away. -- Philip K. Dick, "How to Build a Universe"
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-12-01 04:26:16 UTC
Permalink
For those interested, here is the thread on the microformats list that
resulted from my query:

http://microformats.org/discuss/mail/microformats-discuss/2005-November/002156.html

I haven't yet replied to the responses I got later in the day, but I figured
I'd post the link earlier rather than later.
--
If computers get too powerful, we can organize them into a committee -- that
will do them in. -- Bradley's Bromide
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-12-01 23:25:24 UTC
Permalink
Okay, I think we have a name, and I've written up a quick summary of what
we've come up with so far:

https://reductivelabs.com/projects/runnels

I don't want this to be considered "my" project, but it was a convenient
place to stick the content I have already, so it'll do for now.

Comments are muchly appreciated, and recommendations on how to go on from
here would be great.
--
The hypothalamus is one of the most important parts of the brain,
involved in many kinds of motivation, among other functions. The
hypothalamus controls the "Four F's": 1. fighting; 2. fleeing;
3. feeding; and 4. mating.
-- Psychology professor in neuropsychology intro course
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
David Lutterkort
2005-12-02 00:06:07 UTC
Permalink
Post by Luke Kanies
Okay, I think we have a name, and I've written up a quick summary of what
https://reductivelabs.com/projects/runnels
Thanks for putting together the page (and coming up with a name ;)

Two quick remarks:

* I don't understand the need for 'static routes'. Is their main
point that the consumer doesn't need to subscribe to a channel ?
* I would model the issues of message routing/brokering closely on
an existing message bus like JMS (or Jabber or maybe even DBus).
I think it's important that the underlying messaging
implementation can be easily replaced by something that
addresses a wide variety of environments and load scenarios,
i.e., something that makes it possible to scale the message bus
to a large number of clients.
* The message format should be more flexible than just key/value
pairs. Think of a scenario where you send messages with
performance metrics, and discover that batching the messages at
the producer would significantly reduce the load on the bus and
the consumer. Changing the message format for such a scenario is
trivial if the format is structured (XML) but very hard with
flat key/value pairs.

David
Luke Kanies
2005-12-02 00:36:26 UTC
Permalink
Post by David Lutterkort
Thanks for putting together the page (and coming up with a name ;)
No problem. I *need* this tool. :)
Post by David Lutterkort
* I don't understand the need for 'static routes'. Is their main
point that the consumer doesn't need to subscribe to a channel ?
I think mostly that static routes will be used for people who don't know how
to configure their systems correctly, but I think in general it will be a
useful thing. I could be convinced otherwise, though.
Post by David Lutterkort
* I would model the issues of message routing/brokering closely on
an existing message bus like JMS (or Jabber or maybe even DBus).
I think it's important that the underlying messaging
implementation can be easily replaced by something that
addresses a wide variety of environments and load scenarios,
i.e., something that makes it possible to scale the message bus
to a large number of clients.
Yeah, we will be doing everything we can to copy other people. At the
least, we are doing what we can to cleanly separate the message format, the
protocol, and the transport. Jabber is currently the top contender for the
transport, and the protocol hasn't been discussed much yet.
Post by David Lutterkort
* The message format should be more flexible than just key/value
pairs. Think of a scenario where you send messages with
performance metrics, and discover that batching the messages at
the producer would significantly reduce the load on the bus and
the consumer. Changing the message format for such a scenario is
trivial if the format is structured (XML) but very hard with
flat key/value pairs.
I think that the protocol should be able to handle multiplexing messages
without much effort.

I think key/value pairs will be fine for the messages but the protocol will
definitely have a bit more complexity.
--
Real freedom lies in wildness, not in civilization. -- Charles Lindbergh
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Alex Honor
2005-12-02 00:46:08 UTC
Permalink
This messaging layer is shaping up to be quite an infrastructure. At
this level of complexity I would have to suggest considering a real ESB.
Post by Luke Kanies
Post by David Lutterkort
Thanks for putting together the page (and coming up with a name ;)
No problem. I *need* this tool. :)
Post by David Lutterkort
* I don't understand the need for 'static routes'. Is their main
point that the consumer doesn't need to subscribe to a channel ?
I think mostly that static routes will be used for people who don't know how
to configure their systems correctly, but I think in general it will be a
useful thing. I could be convinced otherwise, though.
Post by David Lutterkort
* I would model the issues of message routing/brokering
closely on
an existing message bus like JMS (or Jabber or maybe even DBus).
I think it's important that the underlying messaging
implementation can be easily replaced by something that
addresses a wide variety of environments and load scenarios,
i.e., something that makes it possible to scale the
message bus
to a large number of clients.
Yeah, we will be doing everything we can to copy other people. At the
least, we are doing what we can to cleanly separate the message format, the
protocol, and the transport. Jabber is currently the top contender for the
transport, and the protocol hasn't been discussed much yet.
Post by David Lutterkort
* The message format should be more flexible than just key/
value
pairs. Think of a scenario where you send messages with
performance metrics, and discover that batching the
messages at
the producer would significantly reduce the load on the bus and
the consumer. Changing the message format for such a
scenario is
trivial if the format is structured (XML) but very hard with
flat key/value pairs.
I think that the protocol should be able to handle multiplexing messages
without much effort.
I think key/value pairs will be fine for the messages but the
protocol will
definitely have a bit more complexity.
--
Real freedom lies in wildness, not in civilization. -- Charles Lindbergh
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-12-02 05:07:22 UTC
Permalink
Post by Alex Honor
This messaging layer is shaping up to be quite an infrastructure. At
this level of complexity I would have to suggest considering a real ESB.
The problem is that ESB's are, um, really complicated. I want something
simple. It might not seem like it, but what we've got so far is actually
pretty simple, and is pretty usable (or, at least, will be usable when
there's an implementation).
--
DOS is the _only_ operating system -- and I'm using that term
loosely -- which [exhibits this behavior].
--Aeleen Frisch, "Essential System Administration"
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
John Rouillard
2005-12-02 15:06:01 UTC
Permalink
Post by Luke Kanies
Okay, I think we have a name, and I've written up a quick summary of what
I assume your static routes are a mechanism for restricting data from a
producer to a particular consumer regardless of the wishes of other
consumers. Whether this resides in restrictions at the routing level
or at the producer level is an open question. I can see the producer
labeling data with "access allowed by luke_consumer" in which case
it breaks your premise:

"Because producer nodes should not have to know or care who is
interested in the data they are producing, runnel messages do not
specify a destination."

In a distributed system, I may have a nagios server being administered by
another group (say for backup coverage on major servers). They should
not be able to ever see my security data on the bus regardless of what
their consumer registers for. They should only be able to get the
nagios alerts and metric data for my machines that they are
monitoring. They shouldn't be able to get data for machines I monitor
via nagios, but that they don't monitor.

Since it seems that the bus is multi-cast by nature, you have to
segregate the consumers/producers in some manner and provide for
filters at the producer level or support user configurable filters at
the router level.

I have seen schemes based on different buses (e.g. think channels in
IRC) networks. So one of my nagios producers (with filters restricting
what is sent) could subscribe to "net-nag-backup" bus to send data to
my backup nagios monitoring sites. Data sent on those buses would be
restricted to exactly what the subset of consumers should see. Then
the authentication for each bus is responsible for the authorization
to see the data on the bus.

Since the producer will send only appropriate data onto the network.
security data won't be seen on net-nag-backup. Their nagios consumer
would subscribe to "theirnet" and "net-nag-backup" to see my data.

If you put the logic in the router to filter/route only correct data,
the consumers may also have to subscribe to different routers since I
will need config access to the router to impose my security
restrictions. In general this means running my own router I would
guess unless the router has support for multiple buses and the static
routes take place at the bus level.
--
-- rouilj

John Rouillard
System Administrator
Renesys Corporation
603-643-9300 x 111
Luke Kanies
2005-12-19 23:43:03 UTC
Permalink
Hi John,

Sorry I didn't reply to this earlier; I missed it somehow.
Post by John Rouillard
I assume your static routes are a mechanism for restricting data from a
producer to a particular consumer regardless of the wishes of other
consumers. Whether this resides in restrictions at the routing level
or at the producer level is an open question. I can see the producer
labeling data with "access allowed by luke_consumer" in which case
The static routes, I figure, will mostly be used by organizations that will
only have one or maybe two consumers of data, so they don't need the hassle
of subscriptions. Frankly, I'm not sure they'll ever be used, but it seems
to make sense to support their existence, anyway.
Post by John Rouillard
"Because producer nodes should not have to know or care who is
interested in the data they are producing, runnel messages do not
specify a destination."
In a distributed system, I may have a nagios server being administered by
another group (say for backup coverage on major servers). They should
not be able to ever see my security data on the bus regardless of what
their consumer registers for. They should only be able to get the
nagios alerts and metric data for my machines that they are
monitoring. They shouldn't be able to get data for machines I monitor
via nagios, but that they don't monitor.
Since it seems that the bus is multi-cast by nature, you have to
segregate the consumers/producers in some manner and provide for
filters at the producer level or support user configurable filters at
the router level.
I agree that this will be needed eventually. I expect that it will be
supported through router-level filters or something similar, where you are
only willing to route certain data to specific consumers or something. I'm
not sure how all of that will work, and I'm not going to bother trying to
solve those problems until I actually encounter them.
Post by John Rouillard
I have seen schemes based on different buses (e.g. think channels in
IRC) networks. So one of my nagios producers (with filters restricting
what is sent) could subscribe to "net-nag-backup" bus to send data to
my backup nagios monitoring sites. Data sent on those buses would be
restricted to exactly what the subset of consumers should see. Then
the authentication for each bus is responsible for the authorization
to see the data on the bus.
I really don't want to encourage this kind of point-to-point bus. The whole
idea behind Runnels is that you're casting data into the wild blue yonder,
making it available to all who want it. Yes, of course you want to be able
to restrict things at the router so that not just anyone can get the data,
but the producers should just be able to say, "Hey router, here's some
data", without knowing anything else.

I adamantly don't want any extra configuration on the producers beyond
knowing where the router is and knowing the message schema.
Post by John Rouillard
Since the producer will send only appropriate data onto the network.
security data won't be seen on net-nag-backup. Their nagios consumer
would subscribe to "theirnet" and "net-nag-backup" to see my data.
If you put the logic in the router to filter/route only correct data,
the consumers may also have to subscribe to different routers since I
will need config access to the router to impose my security
restrictions. In general this means running my own router I would
guess unless the router has support for multiple buses and the static
routes take place at the bus level.
Yeah, I think that breeds too much complexity, and I'll be avoiding it if at
all possible. I expect I'll have something like a control bus and a data
bus, but I don't want other busses unless I just can't avoid it.
--
Risk! Risk anything! Care no more for the opinions of others, for those voices.
Do the hardest thing on earth for you. Act for yourself. Face the truth.
-- Katherine Mansfield
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies
2005-12-04 19:35:29 UTC
Permalink
I got a bounce notice about this email, and I don't remember seeing to, so
I'm forwarding it on.
--
It's a small world, but I wouldn't want to paint it.
-- Stephen Wright
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


---------- Forwarded message ----------
Date: Sat, 3 Dec 2005 19:18:38 -0800 (PST)
From: owner-config-mgmt-aWjYu2gNfz0dnm+***@public.gmane.org
To: config-mgmt-approval-aWjYu2gNfz0dnm+***@public.gmane.org
Subject: BOUNCE config-mgmt-aWjYu2gNfz0dnm+***@public.gmane.org: Admin request of type
/\bsubscribe\b/i at line 9

Subject: Re: [config-mgmt] Re: Sysadmin tools, microformats, and data
exchange
Post by John Rouillard
Since the producer will send only appropriate data onto the network.
security data won't be seen on net-nag-backup. Their nagios consumer
would subscribe to "theirnet" and "net-nag-backup" to see my data.
=20
If you put the logic in the router to filter/route only correct data,
the consumers may also have to subscribe to different routers since I
will need config access to the router to impose my security
restrictions. In general this means running my own router I would
guess unless the router has support for multiple buses and the static
routes take place at the bus level.
I think segregating the bus into several 'channels' makes a lot of
sense. If each channel requires its own authentication (think a
password-protected IRC channel), there's no need to change the router's
configuration to allow a variety of producers/consumers onto the
channel.=20

OTOH, to get more finegrained security, you might want to give each
participant in the bus its own identity, and then control permissions
for each participant on each channel individually (service S can produce
in channel C, service T can only consume in channel D); that would
indeed require either changes to the router's configuration, or that
there is a separate security service that the router accesses.

One related issue to this is that I think the actual client machines on
this bus should be considered particularly untrustworthy, and the system
should, at a very low level, prevent machine X from ever sending a
message to reboot machine Y, i.e., the actual clients on the bus should
probably go through a separate concentrator that checks the messages
they want to send to the services on the bus.

David


--=-EYPmgI3n2ugc421czvCa
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBDkl/94BLQetDjzDARAqi8AJ4s0eGlT8ODfxguwTGWXPATnCR38wCeM/6q
SXTDx/1qAwNPaMpyiXsybNI=
=bQgw
-----END PGP SIGNATURE-----

--=-EYPmgI3n2ugc421czvCa--
DJ Gregor
2005-12-05 22:13:25 UTC
Permalink
Post by Luke Kanies
I started thinking about the kind of data I want to exchange, and
most if it is pretty darn simple: log messages, metrics, alerts,
trouble tickets, things like that. Basically, if two sysadmin
tools _should_ be talking, it seems like it is almost always pretty
easy to come up with a simple data concept for that talking. For
instance, Puppet will be producing metrics, notifying Nagios that
service X on machine Y should be monitored, and will be opening
trouble tickets when it encounters a failure.
So, can we come up with a standard way of talking about those
little data formats, such that all of our sysadmin apps can support
some subset of these objects and accept them over the wire?
Good Day,

I would like to point out that we, as system administrators, are not
the only ones who are looking to have a product-independent standard
for tying together systems to share the data mentioned above. As of
late, I have learned (a relatively tiny amount) about where the
telecommunications industry is moving with managing and operating
their services. I think that there are existing models and APIs that
would be good to look at to see how well they could fit this need in
the sysadmin community for tools like Puppet.

The TeleManagement Forum[1] (TM Forum) has a technical program called
NGOSS[2] (Next Generation Operation Support System). One of the
outputs of NGOSS is a set of specifications for data models called SID
[3], the Shared Information/Data Model. The SID covers all of the
types of data that you mentioned above, excluding log messages (which
might be better filtered and then tossed around as alerts, leaving
unfiltered log transfer to the protocols we all know and love).

Sun and others have created the OSS through Java Initiative[4] (OSS/
J) to create a set of Java-based APIs[5] to tie different operation
support systems together, and to do so based on the SID data models.
Before you get scared off by the Java word, let me say that the
standards include XML schemas for interfacing through a message
service--the standard specifies that a Java message service (JMS) is
used, but we could obviously use a different message service. I'll
also note that a white paper has been published on using web services
for the over-the-wire format[6], allowing other languages to readily
use the OSS/J APIs.

I have not implemented anything using any of the OSS/J APIs, and I
have only browsed over a few of the SID and OSS/J documents. I have
spent a reasonable amount of time looking at the Performance
Management APIs in OSS/J (part of the OSS Quality of Service API) as
part of work I'm doing in OpenNMS, and based on what I've seen so
far, those APIs seem reasonable: not too much fluff, and not missing
anything obvious. FWIW, there already is a project that is extending
OpenNMS to add OSS/J interfaces, OpenOSS[7][8]. I'm not sure if they
have complete interfaces for any of the APIs, but I see code for
performance management, fault management, and trouble ticketing.

I think that NGOSS SID and OSS/J are pretty mature, at least in their
data models, and they cover a lot of the areas that we need as
sysadmins. I can't claim to be any more than a neophyte when it
comes to these standards, but so far they look very promising to me.


- djg

References:

[1]
http://www.tmforum.org/
[2]
http://www.tmforum.org/browse.asp?catID=1911
[3]
http://www.tmforum.org/browse.asp?catID=1684
[4]
http://www.ossj.org/
[5]
http://www.ossj.org/downloads/api.shtml
[6]
http://www.ossj.org/learning/docs/wp_ossj_wsip.pdf
[7]
http://www.openossinitiative.org/
[8]
http://sourceforge.net/projects/openoss/
Luke Kanies
2005-12-18 01:43:44 UTC
Permalink
Post by DJ Gregor
Good Day,
I would like to point out that we, as system administrators, are not
the only ones who are looking to have a product-independent standard
for tying together systems to share the data mentioned above. As of
late, I have learned (a relatively tiny amount) about where the
telecommunications industry is moving with managing and operating
their services. I think that there are existing models and APIs that
would be good to look at to see how well they could fit this need in
the sysadmin community for tools like Puppet.
I will look at these as closely as I can.

Do you think that these are simple enough that it would take less time to
use them than it would be to create a far simpler standard? I don't have a
lot of faith in standardization processes these days; they always seem to
create something so complex that normal humans can't actually use them.
Post by DJ Gregor
The TeleManagement Forum[1] (TM Forum) has a technical program called
NGOSS[2] (Next Generation Operation Support System). One of the
outputs of NGOSS is a set of specifications for data models called SID
[3], the Shared Information/Data Model. The SID covers all of the
types of data that you mentioned above, excluding log messages (which
might be better filtered and then tossed around as alerts, leaving
unfiltered log transfer to the protocols we all know and love).
How do you get the log messages to the thing that does the filtering and
alerting? What log transfer protocol can be used for transferring multiline
logs?
Post by DJ Gregor
[...]
I think that NGOSS SID and OSS/J are pretty mature, at least in their
data models, and they cover a lot of the areas that we need as
sysadmins. I can't claim to be any more than a neophyte when it
comes to these standards, but so far they look very promising to me.
Thank you for these pointers; I'll look at them closely before I go much
further. Please forgive me for expecting that standards created by multiple
corporations for use with Java will be unusably complex, though. :/
--
You don't learn anything the second time you're kicked by a mule.
-- Anonymous Texan
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Loading...