Kamailio and SIP training: notes from the field

Being one of the leading companies involved in Kamailio and open-source SIP infrastructure implementation for VoIP service providers in North America, we run our Kamailio and SIP fundamentals training curriculum a fair bit. It’s a distinctly secondary line of business for us, but since 2011, we’ve done it somewhere around 15 times by now, mostly here in the USA and occasionally internationally. That’s enough repetitions and customer feedback cycles for us to draw some conclusions and generalise about what we ourselves have learned.

It’s usually a two or three-day affair. The first day consists of SIP fundamentals training, which we consider prerequisite to anything else (if you know a bit about what’s required to configure Kamailio effectively, you’ll agree). There are customers who want that portion only, and depending on how in-depth they want to go, that can last two days. Otherwise, the second day is usually focused on Kamailio, and, when there is a third day, it’s usually filled with hands-on “lab” activities and applied exploration of things the customer is specifically interested in.

We generally tweak the structure to emphasise what the customer most hopes to get out of it, and this varies a lot. Some customers are impatiently laser-focused on applied use of Kamailio to solve very immediate needs. Some customers are focused more on shoring up a general understanding of SIP among their support staff to improve troubleshooting outcomes. Some customers’ primary goal is to get people with a fairly “standard” IT background more conversant with the exotic vocabulary and history of IP telephony and real-time communications. Some customers are highly technical developer types and are looking to reach into more rarefied knowledge of some APIs, or really niche aspects of SIP standards and protocol formalities. We have experience catering to that entire spectrum.

All told, teaching SIP and Kamailio is not so different to teaching most other niche software systems, tools, or frameworks. Most lessons we’ve learned seem to apply to all technical training of a 2-3 day introductory format (as opposed to an intensive and more long-term course). I’ll share a few:

Training is not a conference talk

There’s a point of view out there that slide content should be minimal, to serve only as speaking prompts, akin to a speaker’s private note-cards, or to hold illustrations. Certainly, walls of text aren’t useful, and nobody likes speakers who just read their lengthy slides verbatim. Pedagogically useful slides are by nature somewhat abstemious.

However, we’re yet to have a customer who did not ask to keep our slide deck for their own reference. That alone means that the slides have to have some standalone informational value, and can’t be too minimal.

Some hipster slide deck with five slides of faux-“Zen” rhetorical questions,

or vacuous treacle like:

will be of zero value to anyone. In situations where there is a declared intention to use the slides as reference material, they have to strike a balance between walls of text on the other hand, and an utter paucity on the other. They’re a document of sorts.

More generally:

I have observed that many people, when asked to teach a training course or a seminar of some kind, go to drink from a common well of “public speaking skills” they may have deployed in other contexts, such as presentations to management, conference talks, etc. The skills for conference talks seem to be an especially common departure point, where the focus is on keeping the audience engaged.

While all public speaking is a performance that makes demands upon one’s artistry, and there is no question that the challenge of keeping the audience engaged falls into your purview as a trainer, training is not the same as giving a conference talk.

For one, two days is not thirty minutes. More importantly, one’s purpose in being there is specifically to convey non-trivial information as a specialist, and the audience carries a greater responsibility to absorb it. It’s not a sales pitch. You’re not marketing your specialty. The business objectives of a half-hour conference talk given to a general audience are entirely different. It’s worthwhile to ponder that when wrestling with the temptation to pilfer the “performance art” of one and channel it into the other.

Your audience are mostly there because their boss said they have to be, so you don’t have to get them “amped” about the subject. You’re not a clown, and the primary purpose of your visit is not entertainment.

I’m not saying you should aim to be boring—no, by all means add earnestness, humour, wit and charisma to your presentation if you can, and good trainers do. However, if you feel like you have to make it “dynamite” enough for a bunch of ADHD hamsters who will move on to a different room/booth/track in 20 seconds if you don’t keep them on the edge of their seats, stop yourself. You’re optimising for the wrong problem. This is training; it’s their time and their dime.

Have a clear idea of the objective

Having a clear idea of an objective and mindfully allowing it to guide you is not the same as merely stating an objective or marketing an objective. Lots of folk do the latter without a dime of sincere thought capital invested in the former.

You’ve seen it in the facile syllabi of sundry curricula before:

By the end of the VoIP Bushido Expert Seminar 3XL, the student will have mastered the skills of real-world SIP aikido and H.323 jujitsu. The VoIP Bushido Expert Seminar 3XL will empower the student for maximum success in a fast-paced, ever-changing Ameriglobal VoIP marketplace that demands advanced expertise.

Yeah, okay. If you’re running anything remotely describable as a “seminar”, there is exactly 0.0% chance that anyone will come out of it with a mastery of anything. Either you’re teaching something utterly trivial and obvious, or you are abusing the concept of “mastery” in a way that is deeply fraudulentYour marketing department might say everyone’s doing it and it’s not meant literally, but this is the service you are rendering unto the use of language and the meaning of words:



It gives me no pleasure to say that it’s especially apropos in this very cultural moment.



But assuming we’ve been relieved of the notion that anyone stands to “master” anything from a seminar, or for that matter a 2 to 3-day training course, the entirely legitimate question arises: what, exactly, do you mean to accomplish?

Most people in pedagogy will agree, I think, that the primary goal of any “introductory” endeavour should be to leave everyone in the room with a greater level of knowledge and skills than you found them. In the case of the Kamailio side of our training, I like to abuse the cliché “teaching a man to fish”. Going from zero to a “distinguished and commercially viable skill set” with a system like Kamailio takes years. My intent is that everyone leave our training sessions:

  • With a better grasp of the ontologies surrounding Kamailio, and more especially a sense of the Kamailio idioms for various general concepts in SIP proxy behaviour and SIP routing;
    • Transactions;
    • Dialogs;
    • Initial vs. sequential requests and “loose routing”;
    • Hop-by-hop messages (CANCEL, 100 Trying, negative ACK) vs. end-to-end messages;
  • A clear high-level sense of where Kamailio is typically used in building large-scale SIP service provider architectures (e.g. registrar, load balancer, redirect server to add routing intelligence, and the rest);
  • Some familiarity and comfort level with the names of Kamailio concepts and the ideas to which they refer, e.g.
    • Core functions;
    • Modules;
      • Essential modules needed for almost any useful configuration; modules which are “good as core” (e.g. TM);
      • Ancillary modules to provide specific functionality (e.g. JANSSON);
    • Pseudo-variables;
    • Transformations.
  • A clear sense of where to find documentation and how it’s laid out, and some intuition of where to look for certain kinds of things;
  • Some visual and reflexive familiarity with the appearance and anatomy of the Kamailio configuration file
    • Core configuration directives;
    • Module parameters;
    • Subroutines (in essence, SIP event callbacks):
      • Request routes;
      • Reply routes (onreply_route);
      • Failure routes (failure_route);
      • Branch routes;
    • Specialised event routes (callbacks/event handlers exposed by modules).
    • Concepts analogous to general-purpose programming languages and runtimes:
      • String transformations (kind of like string methods in OO languages);
      • Variables
        • Ephemeral/scratch-pad variables ($var(…));
        • Transaction-persistent variables ($avp(…)/$xavp(…));
        • Dialog-persistent variables ($dlg_var(…));

This is not “mastery” of anything, including these very concepts. But the goal of the training is to expose these ideas and vocabulary to the audience so that they recognise them and can use them in the future to develop their knowledge toward their goals.

The “leave them better off than you found them” bit will have different results for different people and groups in our SIP and Kamailio training. People with some development background may go from having a loose-fitting acquaintance with these things already to a more buttoned-down one, allowing them to be more focused and efficient in building further knowledge and experimenting, or at least asking more focused questions of us or on mailing lists, leading to better and more useful answers. For others, it will simply mean putting these words on a mental map where they did not exist before, so that references to them in the future “ring a bell”, an improvement over total bewilderment. There is a notable difference in the nature of the leaps we can expect from developer sorts versus operations types.

That’s a realistic assessment of what we can hope to achieve, though hardly a guarantee. That’s what I tell management when they want to learn more. Some nod approvingly and appreciate my candour, while others, accustomed to viewing programming as a fancy form of typing, bristle at the notion that we can’t get the staff “trained up” so that they can just, you know, code up the product real quick. Regardless, honesty and specificity are the best policies.

Group dynamics

Technical proficiency among our audiences follows a fairly typical Pareto distribution. Due to our prevailing flat-fee structure for most engagements, management will send send five to ten people. One or two of them will have had deeper Kamailio and SIP experience internally and extract a lot of specific information from the training, while the rest are there to soak up “exposure” so that the activities of the other one or two will not be a completely opaque mystery to them.

Just about every group will have That One Guy. (I don’t mean that disparagingly; he’s just That One Guy for lack of a better name coming to mind. And doesn’t have to be male.)

He’ll already have come into contact with 40-70% of your material in some fashion, and is often keen to demonstrate that with pep and vigour. He’ll ask a lot of questions and generate a lot of tangents. The psychological motive is rarely to ingratiate himself to the trainer, who, after all, will pack up and leave soon, but the motives will vary, from genuine intellectual curiosity and affability at one end, to a more ulterior plot to position himself as the “go-to guy” for this subject matter in front of his colleagues. The latter is more common in large organisations, where ownership of projects, and the budgets and clout that come with them, is a contentious topic in the sizzling (or slowly marinating) “office politics” inevitable in any group of nontrivial size.

As in any other consulting project, so it goes in training: every active, invested participant has explicit and covert objectives—well, “covert” implies something nefarious in a way I don’t intend, so perhaps “tacit objectives” is better. Either way, a few walks around the consulting block will lead to the insight that identifying all of these—as best as one can—with sensitivity and perceptiveness is a very important “soft skill” and a key part of the value proposition to management stakeholders. Naturally, it’s a tacit one. This holds true in training as well.

Anyway, trainers and teachers seem divided on whether That One Guy is friend or foe from the point of view of maintaining structure. There are some trainers who believe that such people “hijack” the agenda and do a disservice to the rest of the group as well as the trainer’s efforts to bring things to a common denominator that everyone can access. And it’s true enough that I’ve had some training sessions with small groups, in earlier iterations of doing this, that seemed to turn into a conversation between me and That One Guy. It’s important to remember that the goal is to leave everyone better off than you found them.

But not everybody in a group of more than about 3-4 is going to get something out of the training, and one must accept that. Some attendees may have the potential, but instead zone into their laptops, fighting fires and paying half-attention. It’s their time and their dime, and you don’t need to slow the bus down for their benefit. If they tumble out, they tumble out.

I personally think that That One Guy is an asset. All interactions, even the overwhelmingly lopsided dynamic of lecturing to a group, are two-way. It’s still a conversation with the audience, whose temperature and tempo one must gauge. As long as That One Guy’s role is properly managed, he provides much-needed anchoring and telemetry for how to proceed, helps to generate good energy and convection around the topic, and, often, provides a window into the tacit objectives in the group.

Frequent breaks and atmosphere

There are some managers who would say taking a 10-15 minute break every 1 to 1.5 hours is too frequent. And to them I say: people simply cannot be bombarded with detailed information for two or three hours, even if it’s quite riveting. They will zone out. I would say it’s best practice to insist on it over any objections of management.

A related idea:

Darkened rooms are great for reading slides, but even better for inducing sleep. Windowless fluorescently-lit rooms and depressing dungeons seem to have a similar effect. Bright, sunny conference rooms with picturesque views of trees and park benches serve to grimly remind everyone that they could be having fun outside, but alas, are hearing about Via branch parameter GUIDs from some propeller-head instead. The effects of this on the mood of people in an hour-long meeting are different than on the mood of people stuck with you for two full days of training.

Adjust the implied sympathy of your approach accordingly, but ideally, find a venue that represents a happy medium.