5 Experimental Ideas

The environment that I work in professionally involves a serious amount of computer science and engineering.  There are always measurements to be made and new solutions to experiment with, but it isn't a place where you can just let things stay as a long term science experiment.  This is a bit of a blessing, because it allows you to create a to-do list of things that you might want to experiment with and then prioritize it according to your own time.  I do actually have a cluster that is designated to be nothing but experiments, and I only touch it about once a month. 

Since I don't have too much free time to dig in and try some of the larger ideas, I started thinking about what would simplify the experiments.  But I'd like to get good measurements to validate the experiments, so I will start this off with an oddball. 

1. Precision Time Protocol in a container

Like much of the modern IT world today, I am all about containerization.  But how do we measure internal bridge speed without having to account for the hops to physical interface, switches, routers, and PTP/NTP servers?  When you are looking at tight timing requirements for software, sometimes a fraction of a microsecond determines viability.  I'd love to configure PTP in a container to validate some of the other experiments that containerize existing applications.

Aggregate 1
PTP Overview

An experiment I'd like to perform to test speed and capability is to replace some multicast applications with MQTT using a broker and json publication/subscription models.  This could be a serious amount of reworking on an existing tool due to the volume of multicast groups.  To help review an effort to see if it is worthwhile, I'd like to see:

2. Multicast to MQTT converter

This is a longshot. The goal is to have pub/sub messages writing to a json database for review which adds a hop to the network.  Multicast is great for large integrated systems, and works very well in regards to speed.  MQTT without a broker lacks the command and control functionality of multicast, but it works on garbage networks.  But, it can get messages across daisy chained serial cables pretty quickly.  Pulling the published multicast packets and pushing them through a MQTT broker should allow review and replay of multicast to review existing systems and evaluate retooling. 

Aggregate 2
I want this for every multicast messaging tool

Aggregate 3
And this study is why I want it

Another neat thing about a crazy lab environment is experiment replication.  Not just replaying the experiment, but rebuilding the entire thing on a different set of hardware.  I've gotten used to finding the shortest path possible to building out some of the underlying architecture.  Some of these things are meant to be replaced prior to going into production, but I'm going to fill you in on a dark secret about that.  Development and test can become production if time runs out on the development cycle.  Things like standing up a DNS server can fall between the cracks.  I use PFSense for my DNS needs in small labs, which results in a much lower risk if it slips into production.  That's fine, once the environment is built.  But what I really need is point and click DHCP interface to quickly build the environment.

3. Point and click DHCP configuration tool

Replicating an existing environment means I should also be able to replicate the DNS server.  Being able to point and click at a MAC to modify an existing IP lease to match DNS is the goal.  We really need to stop right here and realize that I am being incredibly lazy and reliant not just on a GUI, but on a feature that isn't really that hard to configure without a point and click option.  I'm just interested in getting an hour or two of my month back.  This actually isn't incredibly difficult to accomplish in PFSense, but the UI doesn't do it exactly how I want it to.  The optimal solution would link into IPMI, show you host MAC addresses, and allow a click to configure the DHCP lease for the application network.  I'll figure something out that is close enough in my spare time.

Aggregate 4
The reigning champion of quick DNS and DHCP setup

One of the tools that I rely on heavily at home is just not possible in a sandboxed environment.  A ready to go Certificate Authority.  An appliance that is ready to shoot out certificates based on a small amount of configuration and can reset it's root cert settings quickly would be very nice for things like management servers, registries, git repos, proxies, and application gateways.  It's usually a couple hours of fumbling around with outdated SOPs to build something that is not remotely secure.  A simple appliance that can be blown away in an instant without feeling bad would be nice.  Minding the dev becoming production, it can't have any bloat, especially anything that would create an instant security issue, like flash or java.

4. CA Appliance

Since you cannot rely on LetsEncrypt when you don't have internet access.  This would also be nice for the home user to establish SSL without reaching out to the internet.  Keep everything internal without the risk of having to do crazy things like opening port 80 on your router to get your LetsEncrypt certificate.  Of course, this is double edged.  If you want the security of a certificate, will you trust the appliance?  You still need to perform the legwork of adding the CA to all of the client systems.  But it would be convenient to just have something local that can be rolled out quickly.

Aggregate 5
Like this, but without things like PKI or Java Requirements

The final idea is something that would reduce troubleshooting hours and pinpoint the majority of issues in minutes.  I have been able to happily use OpenFlow as part of the VMware NSX product.  You can watch the network flow between virtual machines or within the software defined network with ease.  This allows you to start an application, watch it's traffic, and see where it stops.  What would be really nice is having some type of flow service on a local system.  Turn on the application, and from within the application execute your application or scripts.  Have a nice startup message about the current state, such as "iptables are on" or "firewalld is running", then display text based on firewall/iptables status.  Green for no firewall, blue for firewall. 

5. Application integration to protocol analyzer

Imagine logging into your box, executing the monitoring script, then starting an application.  You can already get good information on if the application is started from systemctl.  Being able to see if there is some network issue downstream is important to troubleshooting.  Things like Wireshark already exist, but let's go deeper.  Let's find a way to tag the application in Wireshark prior to execution.  Then, all of the returned capture data should be related to the application.  The current tools can work, I just want them to work better.  You shouldn't need to capture all network traffic to see where the application is failing.  It might be time for some new tools in this arena.

Aggregate 6
I guess this will have to do, for now

While most of these ideas are for lab environments, the benefits would be wide spread.  I will be putting effort into my own configurations to eliminate need for some of the tools, but some of these ideas were due to come around years ago.