The rapidly evolving hardware of SBC

I'm not really a hardware junky, I just play one in the planning meetings.  I was probably a little more excited than most people about the raspberry pi 3b+.  I had no idea what it meant, but there was a footnote on a few articles about power over ethernet, or PoE, capability.  This meant that I could add a pi and not have to consider power requirements to the location it served.  That's the most recent example of getting excited over a feature that I actually cannot use.

I've also experienced roughly 5 power outages in my house over the past month.  Not kidding.  I think I brought it up when I was bragging about buying my new computer hardware.  With what I have experienced at work, flash media dies quickly when you slam the power on it over and over.  I've had to recover time servers, single board systems, and some other random pieces of tech due to failed flash.  That's what makes the decision to get more flash based gadgets painful.  I need a good way to back them up, a load of spare flash media, and some use cases that don't require 5 9s of uptime.

So, with that in mind, I'd like to share the four devices that I will purchase based on my projects.  The first two are obvious, based on my blog.  The Pi and the Pi Zero W.  No need to dig around for numbers, the most recent are what I will use.  Standardized on micro SD, and for the most part they are also standardized on power.  Replacing a power block or drive requires little to no effort, and for things like security cameras, I have backups to clone from.

Aggregate 1:
Pi!

But, I am also prototyping a mechanical device.  That means I want a small form factor that responds to commands rather than a credit card system with a full OS that can replace a computer from the early 2000s.  Which is why I am looking into getting a few Arduino boards.  Commands and responses with cheap and easy Z spectrum add-ons.

Aggregate 2:
Arduino

And finally, I want a device that I can make a single use device for a specified purpose.  I mess around on my computer more often than not.  I recently forced an OS upgrade that stalled due to multiple previous upgrades, failed software experiments, and a massive hardware change.  I want to set up an AI system without worrying about what kind of strange stuff I am doing on my desktop.  And in a couple of months, I will be able to get what will basically be a chipset made just for that purpose.  It includes a CPU, GPU, and NPU (Neural-network Processing Unit).

Aggregate 3:
RockPro

So, who else has bought into features that they cannot use?  And who else is looking forward to tech that they have already planned a purpose for?

Modern computing, remote execution

I'm somewhat certain that the rise of secure remote execution was only a few years ago.  I've been playing around with some stuff at work that allows remote execution, but it is all locked into the same network, so it does not really answer my question.  When did we get the ability to securely execute remotely?  I'm sitting at my desk with a bag full of hobby electric motors and a few spare raspberry pi systems, knowing I can make these motors start spinning from anywhere in the world without exposing ports to the internet.  I'm probably going to use a combination of shell scripts, nodejs, and python to do it.  But this didn't just spring from the cloud.

I guess we will need to start from the beginning, the first computing systems that operated closer to a series of engineered gauges than a piece of hardware running software.  Let's try to follow some paths.  The very first path I want to look at is the Automatic Computing Engine (ACE).  This dives down to almost a century ago.  It was based off of classified work that Turing had performed, and may be the first view into how a program runs from memory.

Aggregate 1:
ACE

We will hit fast forward through some government contractor and academic nerd fights.  There was the Electronic Discrete Variable Automatic Computer (EDVAC) that used binary instead of decimal.  It helped lead to the Harvard Architecture that separated instruction and data in memory.  This is the first steps towards modern computers, but they lacked networking and the ability to run multiple applications simultaneously.  Probably the first example of using multiple computers for a single application was SAGE, using RADAR data to create a unified image.  This eventually led to ARPANET, which was the birth of the internet.

Aggregate 2:
EDVAC

Aggregate 3:
SAGE

This will eventually lead to distributed computing, which has some amazing characteristics.  Concurrency of components, lack of a global clock, and independent failure of components.  They distribute tasking, do not have high security, and have a low tolerance for failure.  Still, they were amazing when they were created.  This is what I had in high school and college on the internet.  Napster and online video games.  About 5 years prior, Object Request Broker (ORB) becomes a mainstream technology.  Issuing commands from one computer to another.  You can now issue commands from your desktop without having to login to the system that will execute the commands. Created without any form of trust, of course.  A long step away from our current cloud computing or domain methods.  These technologies will converge.

Aggregate 4:
Distributed Computing

Aggregate 5:
ORB

I need to pause for a second to apologize for the heavy reliance on Wikipedia.  The first part of this is history, the last part will be a little more cutting edge education.

Becoming more modern, we have Common Object Request Broker Architecture (CORBA).  Which allows ORB to work on Windows, Linux, Mac, Unix, and BSD.  You should be able to use any compliant language to issue commands to other systems. There is also The ACE ORB (TAO), but not the ACE mentioned in aggregate 1, a new ACE. We have gotten into the nested acronym portion of our evolution.  This ACE is the Adaptive Communication Environment, which is a framework that can tie together advanced features of operating systems.  TAO is a low level computing engine that can speak between systems to execute applications in real time.  When I say real time, in this instance it means "as if it were a local command".

Aggregate 6:
TAO

Aggregate 7:
ACE part 2

As security grew and became an integrated part of computing, some components of the model shifted.  Although the security was half-hearted, (security features were turned off by default in the OS) it was at least present.  Kerberos ruled environments with tokens and realms.  Massive SNMP configurations were used to trigger responses to events.  A serious mashup of security and automation.  I don't feel like linking to these technologies, and I have more to add to them.  There were LDAP and AD domains that could link into a realm.  SNMP was a shitty replacement for the old MQTT and DDS messaging that should have been configured in the first place.  But none of these things matter as much to consumers.  They wanted a way to use familiar tools to accomplish trivial tasks quickly.

Microsoft almost got us there with Simple Object Access Protocol (SOAP).  I still remember checking for SOAP port activity at an old job.  Messages that could control a networked application or device.  This is still heavily used today, and has been a great stepping stone to what I currently use for cloud based applications.  Representational State Transfer (REST).  While SOAP was built to be used on the internet, REST was built of the internet.  Commands on HTTP, with no need to open local firewall ports to execute fit the description of both SOAP and REST.  Simple messages that execute quickly, the ability to send code rather than a simple command, and floating on top of existing protocols rather than attempting to become a protocol are what make REST the winner. 

Aggregate 8:
REST

The nice thing about using HTTP to execute is that it is baked into every OS.  The uniform interface of REST makes it easier to tie in any other method of code execution that can be used.  Think about your smart home hubs, your lights may operate on a Zigbee spectrum using MQTT to send the message to each individual light bulb.  But the message that the hub is listening to is from REST.  For the time nerds, it's similar to Precision Time Protocol (PTP) compared to Network Time Protocol (NTP).  REST would be the NTP  that feeds the PTP that is MQTT.  While I have built many devices that use standard networks running on WiFi or Ethernet, converting them to an alternative spectrum like Zwave or Zigbee woule just be a matter of adding a hub to capture the REST command and sending it to the device with MQTT.

The future will obviously hold advances on the existing systems.  If we look at progress in a tick-tock fashion, I would guess that the remote execution has ticked.  The tock is machine learned behavior that executes without a command, being developed and fleshed out on the cutting edge right now.  The next tick will be an increase in speed and rapid reconfiguration based on integrated machine learning to further automate.  The next tock is basically what people will be gambling on the stock market about.  Could be micro-expression recognition to not only automate, but personalize based on observed response to an automation.  In ten years, your lights might turn down because your hub knows you are hung over. 

I adore comments of ridiculous speculation about the future.

Writers block cleared

I have a bad problem with writers block.  Not necessarily on my blog, I can yammer on about anything for at least 500-1000 words.  But when I'm writing a script or attempting to write an application, I can seize up pretty bad.  The typical problem is that I may know what I want the end result to be, but there are a few paths to get where I want to go.  I had been feeling especially locked up with how to play with one of my toys, an IoT hat for a Pi.  I just realized that I had been experiencing another creative block, not knowing what I might want to do now that I have something that I can use to start digging into creative uses of cloud technology.  So it sat there with me being able to look at what I can make the pretty lights do, checking temp and humidity, and considering what I might want to do with the facial recognition capabilities that are baked in.

My block is over.  I have figured out what kind of device I want to build, and now I am looking into what tools are at my disposal for the device.  I can't spoil the secret yet, but I think my goal is going to be building a proof of concept and then quickly selling out.  I will hopefully have a working model within 60 days, and then I can show off some videos of the device in action.  I guess if it stops being fun I will spill the idea and see if anyone can figure out how to build it.

Did you see this thing?

I've learned so much from people who start their conversation with "did you see this thing?".  The world is full of knowledge and knowledgeable people, but on occasion you can get a dud.  Someone trying to sell you on bad math or science.  There are massive gaps in all of our knowledge, and we need to be capable of self correction.  But, we also need to know how to share information and have information shared to us.

The key to being able to share really cool and new things is being able to understand them.  Since this is a tech blog, I figured I would share some of the better websites that have been shared with me.  I have had a nice older coworker that I refer to as my adopted Tiger dad that was helping me with some college math courses.  The first aggregate is pretty much the first thing he shared with me.  And yes, I would feel like I let him down if I got anything less than an A in my math courses.

Aggregate 1:
Best math site ever!

I was jumping back into math with Trigonometry.  A rougher than average high school led me to believe that it only had to do with triangles.  I honestly believed that it was going to be a ridiculous geometry class, until I started The Unit Circle.  That's a pretty special course on this website, because it purifies the math to a level where you could share knowledge with extra-terrestrials if you are capable of understanding the material.

Lets dig around a little deeper into my profession.  To get hands on experience with enterprise grade virtualization.  You want VMware knowledge.  VMware offers labs for free as Hands On Labs (HOL).  They do not offer a path to certification for free.  Let me repeat that so it is clear: VMWARE DOES NOT OFFER A PATH TO CERTIFICATION FOR FREE.  It costs a few thousand dollars, and it isn't necessarily worth it.  What you would be certified for is a technology that you could learn for free with their offerings.  Located at the next aggregate.

Aggregate 2:
My quick link to HOL

This was shared to me by the employees of VMware, probably a few days after it went live.  I still direct coworkers to it, making sure that they have access to the basics of how to work with virtualization.  I can't do anything about people not understanding the virtual stack.  I am still working on that with some basic white papers for my coworkers.

Everyone that has spent more than 10 minutes reading my blog knows that I have a burning desire to know as much as I can about Docker.  Infrastructure, development, commands, management, orchestration, and architecture.  I want to know all about it the same way I wanted to know all about virtualization.  Because I want to understand the common pieces of all containerization and jail technologies.  The more pieces and parts that show up in Docker, the more I wonder about where else they could have come from or be used.  Fortunately, Docker has been building out their knowledge base, and they now have a Hands-on-Lab.

Aggregate 3:
Docker HOL!

This is not going to be tuned to any particular vendor or OS.  So, my complaints about iptables on Photon OS are still just as valid.  Also, check out my blog about iptables on Photon OS if you need help understanding what could go wrong with iptables on a container host.

Finally, something I have not been able to dive into further than installing and playing a little, Machine Learning and Artificial Intelligence.  This is the cutting edge of computing right now.  In general, this is a big data technology.  You should probably have a whole lot of stuff to throw at it to make it a worthwhile investment of your time.  I have data I want to throw at it, I just don't have time to mess with it too much right now.  I also need to dig through the following aggregate.

Aggregate 4:
Making machines smarter

Some of my first blog posts were about setting up security cameras in your house.  Now, imagine being able to have those cameras identify who they see in their lenses.  Going a step further, imagine the security system knowing if that person should be there.  Getting a text message from a computer, with an image, asking if it is okay for someone to be in your house.  It may sound strange, but it is exactly what I am trying to build right now.  Stay tuned for a post in the near future about exploring facial recognition with Machine Learning and applying it to a system with AI.

The debate between whatever

This is going to be heavy on linking to tech opinion articles.  I have tried to keep my eye on tech trends for many years.  About 15 years ago I would pic up something like PC magazine and scan through it.  I still remember the first time I saw a WTF crazy article, it was written by John Dvorak.  To sum it up, he claimed that smartphones were bound to fail due to the high price and risk of losing them in a cab.  This was around the time of the Palm TREO, so he even got smart phones wrong again with his iPhone article years later.  His viewpoint is interesting, but he gets a whole bunch of things wrong about technology and trends.  For the most part, it is entertaining to read his articles.  The articles that I have seen popping up lately though, faulty logic and a disturbingly low understanding of tech and integration are on full display. 

Aggregate 1:
Dvoraks infamous iPhone article

Starting a few years ago, I noticed things like "the debate between virtualization and containers".  I had to take a step back and wonder what poorly constructed AI wrote this miserable article.  It was equal to a garbage post on a comment section.  The authors always seemed to be bringing up specific examples of how and why containers work better on platform X, which makes me wonder about their kickbacks.  Container hosts are generally going to be virtual machines.  Calling containers a VM killer is hyperbolic and does nothing to educate or enlighten the reader.  It is also ignoring all development trends with the biggest companies in both businesses.  I went about my business, creating VMs and not caring about containers until the tech was a little more mature.  It took about four years before I really started getting serious about using them in a professional environment.

Aggregate 2:
Containers vs Virtual Machines!!!

So, now that I am using them in a professional environment, the next big thing that we can quickly dispose of is "will the cloud kill your data center?".  These articles are always fun to read.  In some instances, yes, you will migrate to a nice secure cloud and your company probably will not save money when they do it.  But currently, I am not concerned about clouds destroying all data centers.  They are not as good at certain tasks and protocols that are requirements for a significant amount of companies, for now.  I'm not kidding, go sit down with AWS and ask how they handle multicast in their offerings.  There are also tools to migrate workloads to the cloud when necessary, once again making the technology a complimentary one.  So, while this is an actual threat to some data centers, it is not nearly the end of data centers around the world. 

Aggregate 3:
Cloud killed the on-prem star

My most recent example of ridiculous articles was based on a really cool offering from Amazon.  I have written about the use of AWS Lambda in my echo controlled remote article.  It is amazing technology and an excellent example of cloud services.  The sales pitch I got from AWS was that this is "serverless" technology.  A fancy way to say that they have isolated your code in an environment that does nothing but run scripts when necessary.  It has been referred to as "black box" technology because Amazon won't spill the beans on exactly how they do it.  It's a clever method to tie your code to a container running whatever language your script was written in.  This is where the term "microservice" becomes important. 

Your cloud based application, a container, probably has an engine that works on a specific long running task.  Serving a web page, monitoring a device, maybe a database engine.  But, for a specific event, there should be a script that runs.  In traditional computing, that script would run locally and you would have to worry about patching the application to run it.  Maybe that script offloads data to another application or service, which traditionally would reside on the same box or container.  With current cloud architecture, the correct flow for this would be Container A --> Lambda --> Container B.  You have broken a series of applications apart and linked them back together with a Lambda.  You can now scale horizontally to improve your application, which means you only need to turn on additional instances of whichever container is getting slammed.

This is obviously a complimentary technology.  I apologize for the long buildup on this, but I really wanted to frame this appropriately before I share the article.  Because, to me, this article is the equivalent of "RAM vs CPU, which should you put in your computer?".  To make matters worse, this is from a company that is trying to sell DevOps-as-a-Service.  It is their job to know about these things and explain them correctly. 

Aggregate 4:
containers-vs-serverless

No matter how much I want to write things like "John Dvorak is a crazy guy that shouldn't be taken seriously" after reading some of his articles, I am yet to find an instance in where he attempted to sell a service (other than as a commentator) based on his point of view.  I get it, people are trying to make money off of open source technology and vendor offerings.  They need to appear educated about what they are selling.  It doesn't help Caylent that they have massive errors on their "devops-as-a-service" sales page, and then present a testimonial from someone that is happy about how they set up his WordPress.  Not kidding, as of 06/15/18, they have the blurbs under "Built to Grow" and "Automated Disaster Recovery" mixed up.  This is a problem, one that involves credibility. 

Think about that whenever anyone tells you that some new technology is competing when it could actually be complimentary.

Why was this missing?

I've done a pretty good job of making amateur mistakes when I know better.  Modifying an xorg.conf file without making a backup is usually a favorite.  About 15 years ago I would rebuild the box because I didn't know any better.  I think I wrecked a file consistently for years trying to add additional mouse buttons to the configuration.  This time I did something ridiculously stupid.  My container hosts are mostly virtual machines.  I should be able to clone them in a few minutes.  Instead of doing that, I attempted to migrate all persistent storage to a NFS share.  As expected, it failed miserably.  Nothing important was lost, but it stung a little knowing that I had just wrecked something because I didn't even bother to look up the correct method of doing something.

I began my quest for information on the correct way to mount a share and use it for persistent storage.  Unless someone tells me otherwise, it looks like the correct method for a home lab is to install another package to allow NFS mounts in docker.  A package specifically for docker that is not included, and doesn't get the publicity it deserves.  I was quite surprised that it wasn't included on the host image. 

Aggregate 1:
Netshare plugin for docker

Of course the next step is figuring out how to use it.  Fortunately someone already wrote about that.

Aggregate 2:
NFS on a swarm

This is a pretty big deal to me.  There were some things missing in documentation that I figured out from searching around and tinkering, such as the port used for Swarm being blocked by iptables.  The absence of shared storage instructions may have been one of the reasons why my first attempt at creating a service and swarm was a failure.  I don't recall what all failed in that environment, but I do know that the ability to migrate containers with their own persistent data from one host to another should not be an after thought.  For something like the MotionEye container, the shared storage is considerably more important.  A simple NGINX config file is not hard to replace.  Security camera footage causing all containers to fail because the storage filled up is a problem.  That is exactly why I attempted to migrate everything from the local storage to a share.

It might just be the Photon hosts that I use.  So, coming soon, a blog about setting this up in Portainer and using it to create services to deploy to a swarm.

Add an APC (or other) UPS to Ubuntu 16.04 and Netdata

Netdata has become a method to push for actual statistical data on a system.  In my line of work, the actual statistical data of what a computer is doing while running an application is worth as much as the application.  That is the nature of computer science.  We have figured out creative methods of adding Netdata to running systems, including diskless nodes.  I have been running it at home for months, yet I didn't even look at the glaring issue of what was missing until I added it to my NAS.  I run FreeNAS at home, and you can look at their website to see my blurb about how much I like it.  It really does everything I want it to do.  It was ridiculously easy to add a battery backup and configure it in the UPS settings.  Then, I noticed that there was a service for Netdata included.  Turned it on, and now I can see the battery backup in a browser window when I look at Netdata.

The issue I ran into with my home lab box was not with Netdata, it was with the configuration and installed resources on my box.  Ubuntu 16.04 requires apcupsd to be installed.  I thought that it was automagic, since the system showed the battery in the status bar next to audio and language.  Nope, you need to install the package for battery backup.  So, I installed apcupsd and configured it. 

Aggregate 1:
Install apcupsd

Not sure if that will fix the problem alone.  I changed the UPSNAME to my UPS name.  I have a 1500 because power sucks at my house.  Modified the existing /etc/apcupsd/apcupsd.conf file as a copy rather than copy over the listed config.  The big secret to all of this is that I had to change the configuration of the status to allow it to be read.  I found out that I needed to change it from 0 to 1 in the config.

Aggregate 2:
Change the status

Next up is modifying the Netdata stuff.  Perform a locate to find out if you are changing all of the configs in the same place.  If not, copy and paste as necessary.  The key files you need to uncomment are listed in the Netdata configuration helper. 

Aggregate 3:
Netdata configuration helper

Then you just need to pay attention to the logs listed in /var/log/netdata just like grandpa had to.