When working with AWS, there are plenty of features that make your life easier, but it's rare to find a feature that feels like magic...

Before we dive into the magic, what's the problem we're trying to solve? When working with servers anywhere, usually there needs to be some sort of remote access to SSH or RDP and install/troubleshoot things as needed. Worst case, your server has a public IP with port 22 (SSH) open to the world, which means theoretically anyone, anywhere in the world could break in. As you get more advanced, you setup VPN tunnels or bastion hosts to secure your remote access. This can increase in complexity very quickly with multiple VPC's across multiple regions, multipied by each AWS account. And with the rise in "Zero Trust" networking, how do you securely provide access without blindly trusting network CIDR's?

Enter AWS Systems Manager Session Manager. While not a very magical name, it does provide a magical experience. Session Manager gives you remote access to your servers without opening any ports in security groups, building any VPN tunnels, or using bastion hosts. All access is based on IAM permissions, and sessions are proxied by AWS Systems Manager. You can either use the AWS Console for web access, or the AWS CLI for a native "terminal" experience.

What are the requirements?
The official documentation has the most detail, but the short answer is:

1. Verify SSM agent is running on your EC2 instances with an Instance Role that gives access to Systems Manager
2. If you're using a local terminal:
   a. Verify AWS CLI 1.16.12 or later is installed
   b. Install the Session Manager Plugin for AWS CLI

Once the requirements are met, you can view your managed instances by going to Systems Manager -> Managed Instances in the AWS Console. You can easily start a web-based session by clicking an Instance and then clicking Actions -> Start Session. The same thing can be accomplished from the EC2 console by clicking your instance and then Connect -> Connection Method: Session Manager. While this feels similar to other tools (like a VMware console), the real magic is using the terminal on your local computer.

You can start a remote "SSH" style session with your instances that have zero inbound access with this command:

$ aws ssm start-session --target instance_id

At this point, you have SSH access to your instance, and can go through all your normal installation or troubleshooting steps. The first time you do this feels like magic, as you realize all the time you no longer need to spend with VPN tunnels, bastion hosts, and opening up security groups for remote access.

But wait, there's more! Not only can you initiate remote access sessions for SSH or Powershell, but SSM can also do port-forwarding from your local computer to the remote server! Want to see what a particular web-server is showing on port 80? Just forward a random local port (ex. 9999) to your web-server port 80, and then navigate to http://localhost:9999 in your web-browser. How about Windows RDP access? Session manager only supports SSH or Powershell for remote sessions, but with the port-forwarding feature, we can also get remote RDP access:

$ aws ssm start-session --target instance_id --document-name AWS-StartPortForwardingSession --parameters portNumber="3389",localPortNumber="33389"

Since the "localPortNumber" is set to 33389 above, you would just RDP to 127.0.0.1:33389 to port-forward to your remote instance.

Ok, at this point AWS has provided zero trust remote access, but remembering the AWS CLI SSM commands (especially port-forwarding) aren't the most intuitive. Also, getting the EC2 instance id requires going back and forth between your terminal the AWS web console. I decided to write a Python script that abstracts some of complexity while making it easy to start remote sessions and port-fowarding.

I've called this script AWS Remote: https://github.com/dkuchenski/aws-remote

Some of the main features of AWS Remote:

• Listing all EC2 instance ids, important attributes, and SSM management status
• Start remote sessions via instance id or instance name
• Friendly port-forward syntax via instance id or instance name

So instead of remembering the AWS CLI syntax and finding instance id's, you can do this:

$ aws-remote session instance2-example.domain.com

or

$ aws-remote port-forward 8080 80 instance2-example.domain.com

Note: The instance "name" is referencing the Name tag of the instance

This script uses a combination of the Python Boto3 SDK and the AWS CLI to get information from AWS and start sessions based on your parameters. This is also the first time I've used Click to build the command line interface. Click is great for building easy to use CLI's, and therefore you can use --help at any point to see what your available options are. I've also mimicked the AWS CLI --profile and --region options for a familiar setup.

$ aws-remote --help
Usage: aws-remote [OPTIONS] COMMAND [ARGS]...

  AWS Remote is a simple, command line tool to view and interact with AWS
  instances via SSM. Requires the AWS CLI and Session Manager Plugin to be
  installed locally.

Options:
  --profile TEXT  Specify AWS profile
  --region TEXT   Specify AWS region
  --help          Show this message and exit.

Commands:
  list          List EC2 instances and SSM management status
  port-forward  Start SSM port forward to instance id/name
  session       Start SSM session with instance id/name

I'm still figuring out the best way to distribute scripts like this, but for now there are a couple installation steps that can be found here: https://github.com/dkuchenski/aws-remote/blob/master/README.md

Hope AWS Remote is a useful tool, and if you have any feedback, please let me know!

Amazon Blog about Session Manager:
https://aws.amazon.com/blogs/aws/new-session-manager/

References for using Click in a Python CLI:
https://dbader.org/blog/mastering-click-advanced-python-command-line-apps
https://stackoverflow.com/questions/23626972/how-do-i-pass-variables-to-other-methods-using-pythons-click-command-line-inte

At re:Invent 2018, AWS announced the Transit Gateway, finally giving us a native solution to provide scalable transit connectivity. After attending sessions and deploying Transit Gateway, I wanted to dive into the solution and see what is possible.

Tech Specs

In case you haven't read the official docs, the AWS Transit Gateway is a regional layer3 router connecting VPC, VPN, Direct Connect (soon) across multiple accounts, with support for multiple route tables (VRFs). The Transit Gateway (TGW) has much higher scalability limits than VPC peering.

How did Amazon put a big router in the cloud?
The Transit Gateway is part of the AWS Hyperplane architecture, an internal AWS service that provides terabits of capacity. This is tested and proven tech, already powering NLB, NAT Gateway, and EFS:

Configuration

To setup a TGW, log into your AWS account, and go to VPC. When you click "Create Transit Gateway", you'll be greeted with this screen:

There are a couple options that weren't immediately clear to me, so lets go through them all:

• Name tag: Up to you, I use $accountName-$region-tgw
• Description: ^
• Amazon side ASN: Depends on your routing policy, I use a unique private ASN per TGW for routing policy flexibility
• DNS support: Allows private DNS resolution across VPCs, I would keep this checked
• VPN ECMP support: A single VPN connection is limited to 1.25Gbps, so enable ECMP and use multiple VPN connections to aggregate VPN throughput. AWS has tested up to 50Gbps successfully with ECMP, but theoretically there is "no limit".
• Default route table association: Automatically associate attachments with the default route table. For example, when you attach a VPC to the TGW, use the default route table. This depends on how you use route tables, but I would keep this checked.
• Default route table propagation: Automatically propogate routes from attachments into the default route table. This will allow anything connected to the default route table to route to each other (assuming the VPC route table is also configured). Depending on how you use the TGW, you may not want this. If you do not want spoke-to-spoke connectivity, do not check this.
• Auto accept shared attachments: Automatically accept cross-account attachments to your VPC. I would leave this unchecked.

CAUTION: You cannot change the above values later. You would need to create a new TGW.

Once your TGW is created, you need to "attach" it to a VPC or VPN connection. For VPN, this should look familiar, as you use BGP or static routing to connect to a customer gateway. For VPC, things start to deviate from previous transit solutions. First, you must select VPC subnets in every AZ that you plan to use. This provisions an ENI in those subnets that the TGW then uses to route traffic to that AZ. You can only select one subnet per AZ. This does not affect routing to other subnets in the AZ, its just what the TGW uses to route to that AZ. Then, you manually add routes in the route table that point to the TGW target. The VPC route table limits have not changed (50 route default, 100 route max), so it is best practice to use a summary route to the TGW. I would just route the RFC1918 address ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to the TGW, and then any VPC peering you may do will still work with a more specific route.

Once the TGW is attached to your VPCs, you then need to associate the attachment with a route table, and propopage routes into the route table. If you left all the default options when creating the TGW, both of these actions are done automatically. This provides flat, any-to-any VPC routing out of the box.

If you are looking for isolated hub-and-spoke routing (prevent spokes from talking), then we need to change our config:

1. Create a new TGW route table (VRF) for your hub location, which could be another VPC or VPN
2. Associate the hub attachment with the TGW hub route table
3. Propagate the spoke routes into the TGW hub route table
4. Propagate the hub routes into the TGW default route table, used by the spokes
5. If the default route table has all the spoke routes present, delete those. You can always add routes back if needed. This prevents the spokes from routing to each other, while still allowing access to the hub.

Finally, since TGW is a regional service, how do you connect multiple regions? Right now, this would be done with a transit VPC, or an on-premise router. But if you can wait until early 2019, AWS should have a native solution!

To summarize, the two architectures discussed here were:

1. Flat, any-to-any connectivity. Achieved with the default TGW configuration. All associations and propagations go into the default route table.
2. Isolated, hub-and-spoke connectivity. Requires another route table for the hub, and changing route propagation so the spokes only see the hubs route. The hub route table should have propagated routes from all the spokes.

AWS Transit Gateway is a great addition to the AWS networking stack, and really provides a extremely scalable, secure solution for routing in the cloud.

Official Docs

Introducing AWS Transit Gateway
AWS Transit Gateway Solution
AWS Transit Gateway Docs

Update: AWS has posted all the slides from the NET402 session

Can the AWS Certified Advanced Networking Specialty exam be legitemately passed with only a few weeks of studying? Yes.

Is there a secret shortcut? No.

Although there is no shortcut, I will discuss my previous experience, study materials, and general impressions of the exam. If you have already been working with AWS networking, and wondering what it takes to be certified, this post should give you a pretty good idea. Regarding my timeline, I passed the AWS Certified Solutions Architect Associate exam on 10/18/18, and then passed the AWS Certified Advanced Networking Specialty exam twenty days later on 11/7/18.

1. Previous Experience

First, I've worked in "traditional" networking before starting my AWS experience, and have passed the Cisco CCNP and CCDP exams. This is a huge advantage if you are looking to take the AWS Certified Advanced Network Specialty exam (ANS), since the exam guide recommends the following "General IT Knowledge":

• Advanced networking architectures and interconnectivity options (e.g., IP VPN, MPLS/VPLS)
• Networking technologies within the OSI model, and how they affect implementation decisions
• Development of automation scripts and tools
  • Routing architectures (including static and dynamic)
  • Multi-region solutions for a global enterprise
  • Highly available connectivity solutions (e.g., DX, VPN)
• CIDR and sub-netting (IPv4 and IPv6)
• IPv6 transition challenges
• Generic solutions for network security features, including WAF, IDS, IPS, DDoS protection, and Economic Denial of Service/Sustainability (EDoS).

90% of those topics are covered by the CCNP, and if you have worked in an enterprise environment, you have probably seen or used similar network security features listed on the last line.

Second, I have been working with AWS networking for about 4 years. This started out small, troubleshooting a security group, or creating a vpc, and growing to designing multi-account, multi-region network services and connectivity. This was supplemented by the knowledge I recently gained studying for and taking the AWS SAA exam. Even though an associate exam is no longer required to take a specialty exam, I would highly recommend it, so you can learn the AWS way of doing things "right".

2. Study Materials

Even with the experience described above, I did not immediately go and take the exam. For me, passing the exam isn't just filling in the correct answer, its learning the AWS approach to solving problems, and all the available solutions. I used two main resources to study for the exam:

• Linux Academy Online Course
• The AWS Certified Advanced Networking Official Study Guide

After using Linux Academy for the SAA exam, I was excited to try their ANS cource. Unfortunately I wasn't as impressed, maybe because I was already familiar with the material, but there also weren't as many labs, and the quizes towards the end felt lazy with only a couple questions each. Overall, I did find the material useful and mostly listened to the instructor at 1.25x or 1.5x speed while taking notes on important points. I still recommend Linux Academy as a general training resource, since I'm a fan of their on-demand labs and wide range of content.

If you are looking to get the most bang for your buck, I would definitely purchase the official study guide. The guide is written by seven AWS Solutions Architects, and I found the content very thorough. At the end of every chapter are resources to review, exam essentials, exercises and review questions. The book also includes online practice exams and flashcards. I recommend taking the practice exams over and over again until you genuinely understand each question and the correct answer.

Last, AWS provides whitepapers and various documentation that can also be used as study material. Near the end of my studying I also found more content on AWS Answers. For example, these articles on network connectivity:

• Multiple Data Center HA Network Connectivity
• Single Data Center HA Network Connectivity
  
  

3. General Impressions

I thought this was a difficult test. It isn't enough to know one solution to a given scenario. AWS expects you to know every solution to a scenario, the benefits and limits of each solution, and be able determine the best solution for the scenario. You need to deep dive into each subject. There were still times that I didn't 100% know the answer that AWS was looking for, and I had to make an educated guess. I think "Domain 5.0: Design and implement for security and compliance" from the exam guide was my biggest challenge. Unfortunately I can't go into more detail, as I don't want to risk violating the exam agreement. I am proud to have passed the AWS ANS exam, and I think it is worth pursuing if you work with AWS networking frequently.

Hopefully this post was helpful, if you'd like to discuss AWS networking, come find me at re:Invent 2018! I can be reached via Twitter.

PS. I also discovered a couple other blog posts with some amazing study content after I had already passed the exam:

https://blog.ashiny.cloud/2018/07/29/aws-certified-advanced-networking-specialty/

https://www.itdiversified.com/aws-certified-advanced-networking-prep-direct-connect/
https://www.itdiversified.com/aws-certified-advanced-networking-prep-vpc/
https://www.itdiversified.com/aws-certified-advanced-networking-prep-vpn/
https://www.itdiversified.com/aws-certified-advanced-networking-prep-route-53/

https://www.whizlabs.com/blog/aws-certified-advanced-networking-specialty-certification-preparation/

Ansible 2.5 just came out, and with it comes new network modules (and the network_cli and netconf connectivity methods). The new modules allow us to manage certain IOS configurations without always depending on the "ios_config" module, while the network_cli connectivity method means we don't always need to configure a "provider" for every network playbook. These new modules and connectivity methods allow networking configuration management to look and feel similar to our server management playbooks, which is great for cross-functional teams and reusability.

Ansible 2.5 Release Info:
New Networking Features in Ansible 2.5
List of All Network Modules in Ansible 2.5
Network Best Practices for Ansible 2.5

For a quick introduction, I'm going to take my previously discussed VLAN provisioning playbook and convert it to Ansible 2.5 syntax.

First, remember to upgrade to Ansible 2.5. If you have already have ansible installed, then just upgrade with:

apt-get update && apt-get install ansible

Our first major change is going to use "group_vars" to define connection settings for our network devices. By default, Ansible will read any variable in group_vars/all.yml, group_vars/<group_name> and host_vars/<host_name> when a playbook is executed. Since we are defining how to connect to all our Cisco IOS devices, I am going to put our connection variables in groups_vars/ios-switches.yml

ansible_connection: network_cli
ansible_network_os: ios
ansible_user: ciscoadmin
ansible_ssh_pass: cisco123

To use the network_cli connection, you must specify the "ansible_network_os". Also, please do not store your password in plain-text and instead use Ansible Vault, but to keep this example simple, we are using the password "cisco123".

Since our "ansible_connection" is defined in group_vars, we no longer need the "connection: local" parameter in our playbook.

Next, instead of using the "ios_config" module to create and name VLAN's, let's use the new ios_vlan module.

---
- hosts: ios-switches
  vars:
    vlan_id: 998
    vlan_name: Ansible_VLAN
    
tasks:
    - name: Provision VLAN
      ios_vlan:
        vlan_id: "{{ vlan_id }}"
        name: "{{ vlan_name }}"
        state: present

    - name: Show VLAN
      ios_command:
        commands: show vlan brief
      register: show_vlan

    - debug: var=show_vlan.stdout_lines

As you can see the ios_vlan module simplifies our VLAN provisioning process, since we can define the VLAN ID and the VLAN NAME in a single task. The "state" of the VLAN is not required to create a VLAN, but I included it to show how the new modules use declaritive statements to make creating/removing configuration easier. If we want to remove a VLAN, just change the "state" to absent. I also left in our "show vlan" task from last time, so we can verify our configuration change without logging into the switch manually.

Now to run the playbook:

root@ubuntu:/etc/ansible# ansible-playbook network_new-vlan.yml

PLAY [ios-switches] *************************************************************************************************************************************

TASK [Provision VLAN] ***********************************************************************************************************************************
changed: [10.170.200.2]

TASK [Show VLAN] ****************************************************************************************************************************************
ok: [10.170.200.2]

TASK [debug] ********************************************************************************************************************************************
ok: [10.170.200.2] => {
    "show_vlan.stdout_lines": [
        [
            "VLAN Name                             Status    Ports",
            "---- -------------------------------- --------- -------------------------------",
            "1    default                          active    Gi0/10, Gi0/11, Gi0/12",
            "10   FW_VLAN                          active    Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "20   Data_VLAN                        active    Gi0/1, Gi0/2",
            "30   Voice_VLAN                       active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "998  Ansible_VLAN                     active    ",
            "1002 fddi-default                     act/unsup ",
            "1003 token-ring-default               act/unsup ",
            "1004 fddinet-default                  act/unsup ",
            "1005 trnet-default                    act/unsup"
        ]
    ]
}

PLAY RECAP **********************************************************************************************************************************************
10.170.200.2                : ok=4    changed=1    unreachable=0    failed=0

Success!! While this is great for individual VLAN changes, the real power comes when you define all your environment VLAN's in group_var or host_var files, and looping through the "ios_vlan" task for every VLAN required. I'll save that for a future post!

You can find this playbook (including ansible.cfg and directory structure) on my github: https://github.com/dkuchenski/thenetworkstack-ansible/tree/master/vlan-provisioning-2.5

A couple years ago I posted how to Optimize Ruckus Configuration, and since then I've been continually tweaking so that my dense, multi-story WiFi environment performs well with Windows, Mac OSX, and mobile devices. Here are my current optimizations:

Note: This is based on Ruckus ZoneDirector 10.0.1.0 build 44

System Wide:

• Channel Optimization: Optimize for Performance. This enables all the 5GHz channels, which is especially useful in a dense environment.
• Self-Healing: Automatically adjust 2.4Ghz and 5Ghz channels using Background scanning (I found ChannelFly too busy)
• Background Scanning: Run a background scan every 3600 seconds. The default is too short, causing unnecessary channel changes.

AP Specific:

• Radio B/G/N(2.4G): Only allows channels 1, 6, 11
• Radio A/N/AC(5G) Indoor: Depending on the environment, I may allow all, or disable certain DFS channels or Apple TV peer-to-peer channel (149).
• Channelization 2.4GHz: 20
• Channelization 5.0GHz: 40 (80 just uses too many channels in a dense environment)
• I also disable the 2.4GHz radio on approximately every other AP, since AP placement is built for 5GHz coverage. This helps with 2.4GHz channel contention.
• Turn down TX power for 2.4GHz radios, and potentially 5GHz radios depending on placement.

WLAN Specific:

• If wireless clients do not need to communicate directly with each other (less common with everything "cloud"), enable Wireless client isolation.
• We want Band Balancing, so uncheck "Do not perform Band Balancing on this WLAN service"
• Enable OFDM Only (Disable legacy 802.11b only clients)
• BSS Min Rate: 24.00mbps (Encourages roaming to closer AP)
• Enable 802.11k

With these changes, I have little complaints from users, and wireless "just works". Of course, Apple clients (particulary OSX) are the most difficult, but most of their problems can be solved by turning their WiFi off/on, or sometimes resetting PRAM.

If you have worked with AWS networking, you know there is a laundry list of items that need to be initally configured so the environment is ready for use:

• VPC
• Internet Gateway
• VPN Gateway
• Public Subnets
• Private Subnets
• Public Route Tables
• Private Route Tables
• NAT Gateways
• and more depending on your environment

Since AWS is a software defined cloud datacenter, you will get more and more requests for a Prod environment, then Dev environment, then QA environment, then DevOps environment, etc...but wait, this also needs to be done in every region! Instead of spending your time pointing and clicking your way through the AWS GUI, Amazon has provided an infrastructure automation solution called CloudFormation. CloudFormation uses the AWS API's to automate the provisioning of AWS resources, and its very simple to use.

1. Set up your YAML or JSON file that describes your infrastructure
2. Use the AWS-CLI or GUI to launch (or delete) your infrastructure

Luckily AWS has great sample docs, and I have added VPN Gateway parts, extra subnets, and my naming scheme to the code below (also available on my github). YAML is easy to read, so feel free to take and change what you need for your environment:

Description: 

    This template deploys a VPC, with three public and private subnets spread 
    across three Availability Zones. It deploys an Internet Gateway, with a default 
    route in the public subnets. It deploys three NAT Gateways (one in each AZ), 
    and default routes for them in the private subnets. It deploys a VPN gateway,
    attaches the gateway to the VPC and enables route propagation in all route tables.

Parameters:
    EnvironmentName:
        Description: An environment name that will be prefixed to resource names
        Type: String
        Default: thenetworkstack
    EnvironmentLocation:
        Description: Name of the AWS region that is being used (ex. us-west-2)
        Type: String
        Default: us-west-2
    VpcCIDR:
        Description: Please enter the IP range (CIDR notation) for this VPC
        Type: String
        Default: 10.64.0.0/19
    PublicSubnet1CIDR:
        Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
        Type: String
        Default: 10.64.0.0/22
    PublicSubnet2CIDR:
        Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone
        Type: String
        Default: 10.64.4.0/22
    PublicSubnet3CIDR:
        Description: Please enter the IP range (CIDR notation) for the public subnet in the third Availability Zone
        Type: String
        Default: 10.64.8.0/22
    PrivateSubnet1CIDR:
        Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
        Type: String
        Default: 10.64.16.0/22
    PrivateSubnet2CIDR:
        Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
        Type: String
        Default: 10.64.20.0/22
    PrivateSubnet3CIDR:
        Description: Please enter the IP range (CIDR notation) for the private subnet in the third Availability Zone
        Type: String
        Default: 10.64.24.0/22
Resources:
    VPC: 
        Type: AWS::EC2::VPC
        Properties:
            CidrBlock: !Ref VpcCIDR
            EnableDnsSupport: True
            EnableDnsHostnames: True
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-vpc
    InternetGateway:
        Type: AWS::EC2::InternetGateway
        Properties:
            Tags:
                - Key: Name
                  Value: !Sub ${EnvironmentName}-igw
    VPNGateway: 
        Type: AWS::EC2::VPNGateway
        Properties: 
            Type: ipsec.1
            Tags: 
                - Key: Name
                  Value: !Sub ${EnvironmentName}-vpg
    InternetGatewayAttachment:
        Type: AWS::EC2::VPCGatewayAttachment
        Properties:
            InternetGatewayId: !Ref InternetGateway
            VpcId: !Ref VPC
    VPNGatewayAttachment:
        Type: AWS::EC2::VPCGatewayAttachment
        Properties:
            VpnGatewayId: !Ref VPNGateway
            VpcId: !Ref VPC
    PublicSubnet1: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 0, !GetAZs '' ]
            CidrBlock: !Ref PublicSubnet1CIDR
            MapPublicIpOnLaunch: true
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-public-${EnvironmentLocation}a
    PublicSubnet2: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 1, !GetAZs '' ]
            CidrBlock: !Ref PublicSubnet2CIDR
            MapPublicIpOnLaunch: true
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-public-${EnvironmentLocation}b
    PublicSubnet3: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 2, !GetAZs '' ]
            CidrBlock: !Ref PublicSubnet3CIDR
            MapPublicIpOnLaunch: true
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-public-${EnvironmentLocation}c
    PrivateSubnet1: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 0, !GetAZs '' ]
            CidrBlock: !Ref PrivateSubnet1CIDR
            MapPublicIpOnLaunch: false
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}a
    PrivateSubnet2: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 1, !GetAZs '' ]
            CidrBlock: !Ref PrivateSubnet2CIDR
            MapPublicIpOnLaunch: false
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}b
    PrivateSubnet3: 
        Type: AWS::EC2::Subnet
        Properties:
            VpcId: !Ref VPC
            AvailabilityZone: !Select [ 2, !GetAZs '' ]
            CidrBlock: !Ref PrivateSubnet3CIDR
            MapPublicIpOnLaunch: false
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}c
    NatGateway1EIP:
        Type: AWS::EC2::EIP
        DependsOn: InternetGatewayAttachment
        Properties: 
            Domain: vpc
    NatGateway2EIP:
        Type: AWS::EC2::EIP
        DependsOn: InternetGatewayAttachment
        Properties:
            Domain: vpc
    NatGateway3EIP:
        Type: AWS::EC2::EIP
        DependsOn: InternetGatewayAttachment
        Properties:
            Domain: vpc
    NatGateway1: 
        Type: AWS::EC2::NatGateway
        Properties: 
            AllocationId: !GetAtt NatGateway1EIP.AllocationId
            SubnetId: !Ref PublicSubnet1
            Tags:
                - Key: Name
                  Value: !Sub ${EnvironmentName}-${EnvironmentLocation}a
    NatGateway2: 
        Type: AWS::EC2::NatGateway
        Properties:
            AllocationId: !GetAtt NatGateway2EIP.AllocationId
            SubnetId: !Ref PublicSubnet2
            Tags:
                - Key: Name
                  Value: !Sub ${EnvironmentName}-${EnvironmentLocation}b
    NatGateway3: 
        Type: AWS::EC2::NatGateway
        Properties:
            AllocationId: !GetAtt NatGateway3EIP.AllocationId
            SubnetId: !Ref PublicSubnet3
            Tags:
                - Key: Name
                  Value: !Sub ${EnvironmentName}-${EnvironmentLocation}c
    PublicRouteTable:
        Type: AWS::EC2::RouteTable
        Properties: 
            VpcId: !Ref VPC
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-public
    DefaultPublicRoute: 
        Type: AWS::EC2::Route
        DependsOn: InternetGatewayAttachment
        Properties: 
            RouteTableId: !Ref PublicRouteTable
            DestinationCidrBlock: 0.0.0.0/0
            GatewayId: !Ref InternetGateway
    PublicSubnet1RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PublicRouteTable
            SubnetId: !Ref PublicSubnet1
    PublicSubnet2RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PublicRouteTable
            SubnetId: !Ref PublicSubnet2
    PublicSubnet3RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PublicRouteTable
            SubnetId: !Ref PublicSubnet3
    PrivateRouteTable1:
        Type: AWS::EC2::RouteTable
        Properties: 
            VpcId: !Ref VPC
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}a
    DefaultPrivateRoute1:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref PrivateRouteTable1
            DestinationCidrBlock: 0.0.0.0/0
            NatGatewayId: !Ref NatGateway1
    PrivateSubnet1RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PrivateRouteTable1
            SubnetId: !Ref PrivateSubnet1
    PrivateRouteTable2:
        Type: AWS::EC2::RouteTable
        Properties: 
            VpcId: !Ref VPC
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}b
    DefaultPrivateRoute2:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref PrivateRouteTable2
            DestinationCidrBlock: 0.0.0.0/0
            NatGatewayId: !Ref NatGateway2
    PrivateSubnet2RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PrivateRouteTable2
            SubnetId: !Ref PrivateSubnet2
    PrivateRouteTable3:
        Type: AWS::EC2::RouteTable
        Properties: 
            VpcId: !Ref VPC
            Tags: 
                - Key: Name 
                  Value: !Sub ${EnvironmentName}-private-${EnvironmentLocation}c
    DefaultPrivateRoute3:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref PrivateRouteTable3
            DestinationCidrBlock: 0.0.0.0/0
            NatGatewayId: !Ref NatGateway3
    PrivateSubnet3RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            RouteTableId: !Ref PrivateRouteTable3
            SubnetId: !Ref PrivateSubnet3
    VPNGatewayRouteProp: 
        Type: "AWS::EC2::VPNGatewayRoutePropagation"
        Properties:
            RouteTableIds: [!Ref PublicRouteTable, !Ref PrivateRouteTable1, !Ref PrivateRouteTable2, !Ref PrivateRouteTable3]
            VpnGatewayId: !Ref VPNGateway
        DependsOn: VPNGatewayAttachment
Outputs: 
    VPC: 
        Description: A reference to the created VPC
        Value: !Ref VPC
    PublicSubnets:
        Description: A list of the public subnets
        Value: !Join [ ",", [ !Ref PublicSubnet1, !Ref PublicSubnet2, !Ref PublicSubnet3 ]]
    PrivateSubnets:
        Description: A list of the private subnets
        Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2, !Ref PrivateSubnet3 ]]
    PublicSubnet1:
        Description: A reference to the public subnet in the 1st Availability Zone
        Value: !Ref PublicSubnet1
    PublicSubnet2: 
        Description: A reference to the public subnet in the 2nd Availability Zone
        Value: !Ref PublicSubnet2
    PublicSubnet3: 
        Description: A reference to the public subnet in the 3rd Availability Zone
        Value: !Ref PublicSubnet3
    PrivateSubnet1:
        Description: A reference to the private subnet in the 1st Availability Zone
        Value: !Ref PrivateSubnet1
    PrivateSubnet2: 
        Description: A reference to the private subnet in the 2nd Availability Zone
        Value: !Ref PrivateSubnet2
    PrivateSubnet3: 
        Description: A reference to the private subnet in the 3rd Availability Zone
        Value: !Ref PrivateSubnet3
    VPNGateway:
        Description: A reference to the VPNGateway created in the VPC
        Value: !Ref VPNGateway

Once your YAML file is setup the way you want, you can log into your AWS account and then:

1. Click Services -> CloudFormation -> Create new stack
2. Click "Upload a template to Amazon S3" and choose your YAML file
3. Give the stack a name, ex: prod-us-west-vpc
4. Change any default parameters if needed, then click "Next" and "Create"

Your VPC will be created in the region the GUI is set to, and you can easily delete everything you created by deleting the CloudFormation stack.

When executing an ansible playbook, you may get the following error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: paramiko.BadHostKeyException: Host key for server switch-name does not match!

This probably happened after you changed the name of your network device, and generated new RSA keys. Or, if you are using DNS device names in ansible, resolve the name of your network device from your ansible server and verify the DNS resolution is correct. To fix the issue, edit your "known_hosts" file and remove the key associated with the network device throwing the error:

1. Open your known_hosts file

vi ~/.ssh/known_hosts

2. Search for the line with the network device RSA key

/network_device_name

3. Delete the RSA key for that network device ("dd" deletes the entire line)

dd

4. Save the known_hosts file

#Press the escape key first
:wq

If you don't want ansible to perform this check, edit your ansible.cfg file with:

host_key_checking = False

On day two of Networking Field Day, Pluribus Networks gave us a rundown on what is possible with their Netvisor OS, whitebox hardware, and a distributed architecture. I was impressed with the flexibility of the solution, but like any design choice, there are some limitations to be aware of.

As a relative newcomer to the Pluribus world, I wanted to know what Pluribus was made of. The answer?

• Whitebox (or Pluribus branded) switch hardware
• Layer 3 connectivity between switches
• Pluribus Netvisor Operating System

Hardware

Most switch hardware is now pretty generic, so Pluribus gives you the choice to run Dell, D-Link, Edge-Core or Pluribus hardware, which means the magic is really in the software. I'm not going to spend time going over hardware options, because with those choices, you should be able to easily find an option that fits your price/performance needs - that's the beauty of whitebox.

Layer 3

Next on the list, why L3 connectivity? What about my precious L2 network? Well, the Netvisor OS uses VXLAN for L2 connectivity, which gives you some very interesting design flexibility. For example, you can run Cisco or Arista in your core/spine, and just use Pluribus at the edge/leaf nodes (where L2 actually matters). Your core provides L3 connectivity, and your Pluribus edge will route all our L2 traffic as needed over L3 via VXLAN. Why would you do this? How about lowering upgrade costs by leaving your core as-is. Or run two different fabrics side by side. Maybe span your L2 edge across datacenters? Well...let's not get too crazy...

Pluribus will advertise Netvisor OS as a DCI solution, but like I mentioned before, every design has a trade-off, and generally L2 DCI has a steep price - Pluribus is no different. It is important to ask questions like "What happens to both sides of the fabric if the DCI goes down?". The good part? The Netvisor fabric will continue to run as-is in split brain mode. The bad? You will be locked from making any configuration changes on any side until the DCI is restored. Personally, I would run two separate fabrics that are not dependent on the DCI.

Netvisor Operating System

Now, the most interesting part of the Pluribus solution - Netvisor OS. There are so many different features and components, but I'm only going to focus on two areas that grabbed my attention during the presentation:

1. Management Capabilities
2. Network Virtualization

Management Capabilities

Immediately I noticed that there are no controllers running the Netvisor OS fabric. Each switch is running the Netvisor OS application, with full-mesh connectivity to all other switches in the fabric. You can log into the CLI or Web or API from any switch and manage the entire fabric! Besides fabric wide configuration, you can also revert changes with fabric wide rollback. While controller-less has it's benefits, there are some scaling limitations to be aware of that other "controller-required" solutions do not encounter. Right now, a Pluribus network is limited to 40 switches in a single fabric. Considering two top-of-rack switches should easily be able to support 1000 VM's in any modern design (in half a rack), this shouldn't be a concern for many, but it's good to be aware.

Pluribus has also built out integrations with other management platforms, like VMware and Nutanix. The VMware integration works closely with all the major VMware products - ESXi, VSAN, and NSX. The Netvisor OS switch fabric will automatically build networks and VXLAN tunnels to support new ESXi hosts coming online and on-demand vMotions to ensure full VM mobility across the fabric. VSAN cluster multicast configurations are completely automated on the network side. NSX is particularly interesting, as Netvisor OS doesn't just provide L3 connectivity for NSX networks, it also provides visibility into the traffic inside the NSX VXLAN tunnels.

On top of management flexibility and third-party integrations, Pluribus has also built out a analytics/telemetry platform that allows deep visibility into all your network traffic. In the future, I'd like to see Pluribus provide a tight integration with Kubernetes for container networking.

Network Virtualization

100% Network/Tenant Virtualization. Not everyone needs this feature, but if you do, this is incredibly slick. Pluribus has virtualized the entire network stack for us, just like server hypervisors have done for the systems guys. When you create a new "tenant" in Netvisor OS, the tenant will have their own management portal, full set of VLANs, routing protocols, and policies - all running on the same underlying hardware. Tenant "A" can log into the CLI and make changes completely independent of Tenant "B". This concept may not be completely new if you are used to working with VDCs on Cisco Nexus 7ks, but if you are used to working on Nexus 9ks (which is what Cisco is pushing for all new datacenter deployments - especially if you are using VXLAN), then this feature will be new to you.

Even if you're not a multi-tenant datacenter provider, virtualizing your network infrastructure can be used for a number of scenarios:

• Separate Dev, QA and Prod environments
• Separate External, DMZ and Internal networks
• PCI network segments
• Guest and Contractor networks
• Automation testing

This is just another way for the network to provide value by reducing network costs for separate environments, and decrease risk by limiting changes to each unique environment.

For new datacenter builds, or even more mild datacenter refreshes, Pluribus has the capability to fit in with a solid set of features and integrations. Giving network operators the choice to bring our own hardware means we're not limited by the vendor's hardware pricing or design. And while Pluribus is far from the only company providing a network operating system on whitebox hardware, their network management and virtualization stack are differentiating features that deserve a look.

Disclosure: I attended this presentation as part of Networking Field Day, a Tech Field Day event. All expenses related to the event were sponsored, in addition to any free swag given out by vendors. All opinions expressed on The Network Stack are my own and not those of Tech Field Day, vendors or my employer.

A couple years ago, my coworker wrote a great post on finding disk space issues in Linux and how to resolve them. Helped me out this weekend when a critical server was having issues, so wanted to post it here:

http://www.randomlyexpressed.com/diagnosing-linux-disk-usage/

For future reference (after reading the post above)

Show filesystem usage:

~$ df -h

Show all files/directories consuming space:

~$ du -kx / | sort -nr | more

Show directories in current location:

~$ du -hc | sort -h
~$ du -hc --max=1 | sort -h #This limits your directory listing to one level

List files in directory:

~$ ls -larth

List files in directory sorted by size, ascending:

~$ ls -lSrh

List and then delete any .log file older than 7 days:

~$ find . -type f -name "*.log" -mtime +7 -print -exec rm -fr {} \;

Another easy way to reclaim space by removing unused dependencies:

~$ apt-get autoremove

Note: There is a newer guide for VLAN provisioning with Ansible 2.5

Ansible is such a powerfull tool that it can be easy to get lost in all the possibilities. Running your "network infrastructure as code" with full configs auto-generated and checked into git is the dream, but we can start simple with automating time-consuming tasks. This post will focus on getting ansible up and running with a playbook to configure new vlans across your switches.

If you have not installed Ansible yet, please see my guide for installing Ansible on Ubuntu 16.04

First, we need to build our Ansible hosts file which is an inventory list of our network devices. We can categorize and group devices within the hosts file, which makes configuring the right devices easier down the road.

root@ubuntu:~# cd /etc/ansible/
root@ubuntu:/etc/ansible# vi hosts
# This is the default ansible 'hosts' file.
#
[networkdevices:children]
switches
routers

[switches]
switch01.thenetworkstack.com
switch02 ansible_host=10.170.200.1
10.170.200.2

[routers]

I created a parent group called networkdevices, and then made two "child" groups called switches and routers. For the VLAN playbook we are going to create, we only really care about the switches group, but I wanted to show how you can easily create a grouping structure. For defining your actual network devices, there are three main options shown above in the switches group:

• The fully-qualified domain name of the device (assuming you have DNS working)
• A local device name, and define the device ip address with the "ansible_host" variable
• The ip address of the device

After you configure your Ansible hosts file with a network device or two, we can start building a playbook. A playbook is Ansible's way of defining a set of tasks. Within a playbook, we will use network modules to run different types of commands (for example, show vs config) on our devices. Ansible comes with support for most major network devices, listed here: http://docs.ansible.com/ansible/latest/list_of_network_modules.html

Let's get started with our playbook. First, give your playbook a name, and make sure it ends in ".yml". To start your YAML file, the first line should be "---", and remember spaces matter. If you get syntax errors, check your spacing on each line:

root@ubuntu:/etc/ansible# vi network_new-vlan.yml
---
- hosts: switches
  connection: local

  tasks:
    - name: Show VLAN
      ios_command:
        commands: show vlan brief
      register: show_vlan

    - debug: var=show_vlan.stdout_lines

This simple playbook just runs a command (show vlan brief), registers the output of that command to a variable (show_vlan) and then displays the formatted value of the variable (stdout_lines).

You may notice we don't have any credentials stored in this playbook - we will provide them when we run the playbook (I'm using cisco as my username):

root@ubuntu:/etc/ansible# ansible-playbook network_new-vlan.yml -u cisco -k
SSH password:

PLAY [switches] *******************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************
ok: [10.170.200.2]

TASK [Show VLAN] *****************************************************************************************************************************************************************
ok: [10.170.200.2]

TASK [debug] **********************************************************************************************************************************************************************
ok: [10.170.200.2] => {
    "show_vlan.stdout_lines": [
        [
            "VLAN Name                             Status    Ports",
            "---- -------------------------------- --------- -------------------------------",
            "1    default                          active    Gi0/10, Gi0/11, Gi0/12",
            "10   FW_VLAN                          active    Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "20   Data_VLAN                        active    Gi0/1, Gi0/2",
            "30   Voice_VLAN                       active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "1002 fddi-default                     act/unsup ",
            "1003 token-ring-default               act/unsup ",
            "1004 fddinet-default                  act/unsup ",
            "1005 trnet-default                    act/unsup"
        ]
    ]
}

PLAY RECAP ************************************************************************************************************************************************************************
10.170.200.2                : ok=3    changed=0    unreachable=0    failed=0

Pretty simple! I currently only have a single device in my Ansible hosts file, but with multiple devices you can issue show commands across every switch in your network. You can also modify the playbook above with any "show" command.

We used the "ios_command" network module above, but to actually make changes, we will need to use the "ios_config" network module.

When creating playbooks for common tasks (like creating a vlan), I try to use variables as much as possible so the playbook is easy to reuse. You can see below that I define the vlan_id and vlan_name variable, which makes them very easy to change when needed - you don't need to edit anything else in the playbook.

---
- hosts: switches
  connection: local
  vars:
    vlan_id: 998
    vlan_name: Ansible_VLAN

  tasks:
    - name: Configure VLAN ID
      ios_config:
        lines:
          - vlan {{ vlan_id }}

    - name: Configure VLAN Name
      ios_config:
        lines:
          - name {{ vlan_name }}
        parents: vlan {{ vlan_id }}

    - name: Show VLAN
      ios_command:
        commands: show vlan brief
      register: show_vlan

    - debug: var=show_vlan.stdout_lines

As you can see, we added the VLAN configuration before our "Show VLAN" task. There are two parts to configuring a VLAN - the VLAN ID and the VLAN name. The first task configures the VLAN ID on the switch, then the second task tells Ansible to configure the VLAN name under the VLAN ID we just created. Since the VLAN name is part of a heirarchical configuration in Cisco IOS, we can use the "parents" feature in Ansible to let it know that the name is placed under the VLAN ID configuration.

After the VLAN configuration is completed, our "Show VLAN" task will display the VLAN database with our newly created VLAN.

root@ubuntu:/etc/ansible# ansible-playbook network_new-vlan.yml -u admin -k
SSH password:

PLAY [switches] *******************************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************
ok: [10.170.200.2]

TASK [Configure VLAN ID] **********************************************************************************************************************************************************
changed: [10.170.200.2]

TASK [Configure VLAN Name] ********************************************************************************************************************************************************
changed: [10.170.200.2]

TASK [Show VLAN] ******************************************************************************************************************************************************************
ok: [10.170.200.2]

TASK [debug] **********************************************************************************************************************************************************************
ok: [10.170.200.2] => {
    "show_vlan.stdout_lines": [
        [
            "VLAN Name                             Status    Ports",
            "---- -------------------------------- --------- -------------------------------",
            "1    default                          active    Gi0/10, Gi0/11, Gi0/12",
            "10   FW_VLAN                          active    Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "20   Data_VLAN                        active    Gi0/1, Gi0/2",
            "30   Voice_VLAN                       active    Gi0/1, Gi0/2, Gi0/3, Gi0/4, Gi0/5, Gi0/6, Gi0/7",
            "998  Ansible_VLAN                     active    ",
            "1002 fddi-default                     act/unsup ",
            "1003 token-ring-default               act/unsup ",
            "1004 fddinet-default                  act/unsup ",
            "1005 trnet-default                    act/unsup"
        ]
    ]
}

PLAY RECAP ************************************************************************************************************************************************************************
10.170.200.2                : ok=5    changed=2    unreachable=0    failed=0

root@ubuntu:/etc/ansible#

Success!!

Now when a new VLAN needs to be provisioned, just change the vlan_id and vlan_name vars, and you can configure a VLAN across your entire network in less than a minute.

You can also find the hosts file and playbook on my github: https://github.com/dkuchenski/thenetworkstack-ansible/tree/master/vlan-provisioning

This is my first post in a series I'm calling Cisco Config Basics. These posts will serve as a reference for anyone new to Cisco, or those of you just looking to compare your current configs. After the full config, I will explain each config item line-by-line.

In a campus network, the User Port has a lot riding on it - vlan config, security controls, DHCP protection, PoE settings...not to mention new IPv6 policy that is needed even if you're not running IPv6 in your network! As promised I will give the full config first, then explain everything:

User Port Config (Catalyst 2960X - IOS 15.2.2E6)

interface GigabitEthernet1/0/1
 switchport access vlan 10
 switchport mode access
 switchport voice vlan 20
 switchport port-security maximum 6
 switchport port-security violation  restrict
 switchport port-security aging time 2
 switchport port-security
 load-interval 30
 no snmp trap link-status
 no logging event link-status
 power inline police action log
 no power efficient-ethernet
 storm-control broadcast level pps 1k 500
 storm-control action shutdown
 storm-control action trap
 spanning-tree portfast
 ip dhcp snooping limit rate 100
 ipv6 nd raguard attach-policy RA-BLOCK
 ipv6 dhcp guard attach-policy DHCP-CLIENT

Whew, that's a decent amount of config! Let's take a look:

switchport access vlan 10
Sets the 'untagged' vlan for the port. Most end-user computers/devices are not vlan aware, so this is the vlan the connected device will join.

switchport mode access
Forces the port into access mode instead of 'dynamic desirable' - it will not negotiate to be a trunk port.

switchport voice vlan 20
Set the 'tagged' vlan for VoIP devices. VoIP phones need to know to tag their traffic to this vlan, usually done through CDP, LLDP, or DHCP options.

switchport port-security maximum 6
Allows a maximum of six devices (mac-addresses) connected to the port. At a minimum, two mac-addresses need to be allowed for their computer and phone. 6 devices allows for a little wiggle room, while still protecting the switch from any mac-address flooding attempts.

switchport port-security violation restrict
The 'restrict' parameter will drop traffic from any mac-addresses past the maximum (6) limit. You can also use the 'shutdown' option to shut down the port instead.

switchport port-security aging time 2
If a device is connected to a port, but inactive, it's mac-address will be flushed after 2 minutes. Devices that physically disconnect will have their mac-address flushed immediately.

switchport port-security
The port-security commands above do not take effect until port-security is enabled on the port with this command. By default, port-security will automatically shutdown the port when a new MAC address is detected, which is why we tuned the config above with 'violation restrict' and 'aging time 2'. You can see the port-security policy on an interface with the command 'show port-security interface g1/0/1'.

load-interval 30
By default, Cisco calculates the load (utilization) of an interface over a 5 minute average. This makes it harder to see bursty traffic, so I prefer to lower the interval to the minimum - 30 seconds.

no snmp trap link-status
No need to flood your snmp trap server with interface up/down events for user ports. This information isn't generally helpful since users will be connecting and disconnecting their computers all the time, better to get rid of the noise.

no logging event link-status
Same as the command above, but for your switch logs and syslog server.

power inline police action log
Instead of disabling a port for exceeding power allocation, a syslog message is generated, notifying you of a misbehaving device.

no power efficient-ethernet
Disables Energy Efficient Ethernet (EEE - 802.3az), which is supposed to reduce power consumption during idle usage, but I've seen weird behavior from a number of devices that is solved by disabled EEE.

storm-control broadcast level pps 1k 500
Storm-control is used to detect a flood of broadcast, unknown unicast, or unknown multicast traffic and shutdown the port. The first number represents the 'rising threshold' after which traffic is blocked. The second number is the 'falling threshold' where traffic will no longer be blocked if it falls back below this value. There are multiple ways to configure this feature (% utilization, bps, pps), but this example uses pps to enforce a 1000 packet-per-second threshold for broadcast traffic, which I have found to be a good general number to start. This number will vary and can be tuned depending on your environment.

storm-control action shutdown
When a storm is detected, the port is shutdown. The port can recover with the 'errdisable recovery' global policy shown below or manually typing 'no shutdown' in the interface configuration.

storm-control action trap
Also sends an SNMP trap when the storm is detected, in addition to the shutdown above.

spanning-tree portfast
Spanning-tree portfast places the port in the forwarding state immediately, instead of going through all the initial spanning-tree states. Helps the connected computer obtain an IP address and get on the network faster. Only use this if connecting to a non-switch device.

ip dhcp snooping limit rate 100
Limits the number of DHCP requests to 100 requests per second. This prevents a misconfigured or malicious device from DOS'ing your DHCP server.

ipv6 nd raguard attach-policy RA-BLOCK
Blocks rogue IPv6 router advertisement (RA) messages. Even if you do not use IPv6 in your network, a rogue device could start advertising itself as a IPv6 router and funnel traffic through itself, since most modern operating systems will prefer IPv6 over IPv4.

ipv6 dhcp guard attach-policy DHCP-CLIENT
This blocks IPv6 DHCP server replies and advertisements from unauthorized (end-user) ports.

Global Switch Config

In addition to the user port config, there are global switch settings that are needed as well.

spanning-tree mode rapid-pvst
spanning-tree logging
spanning-tree portfast bpduguard default

errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause storm-control

ipv6 nd raguard policy RA-ALLOW
 device-role router
ipv6 nd raguard policy RA-BLOCK
 device-role host #default role
ipv6 snooping logging packet drop
ipv6 snooping logging resolution-veto
ipv6 dhcp guard policy DHCP-CLIENT
 device-role client #default role
ipv6 dhcp guard policy DHCP-TRUST
 trusted-port

ip dhcp snooping vlan 1-4094
no ip dhcp snooping information option
ip dhcp snooping

interface Port-channel1
 ip dhcp snooping trust

spanning-tree mode rapid-pvst
Enables rapid per-vlan spanning-tree, which I would recommend as the default spanning-tree setting for a Cisco network.

spanning-tree logging
Spanning-tree events are logged in the device logs and syslog.

spanning-tree portfast bpduguard default
Enables bpduguard on every port that has 'spanning-tree portfast' configured. With this, the switch will shutdown any port that recieves bpdus, which protects your network from rogue switches being plugged in and potentially becoming spanning-tree root.

errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause link-flap
errdisable recovery cause psecure-violation
errdisable recovery cause dhcp-rate-limit
errdisable recovery cause storm-control
Sets auto-recovery for the policies above (udld, bpduguard, link-flap, psecure-violation, dhcp-rate-limit, storm-control). Default recovery is 300 seconds. If the issue is still present after auto-recovery, the port will shut and wait another 300 seconds.

ipv6 nd raguard policy RA-ALLOW
device-role router
This defines a raguard policy you can attach to a port. This policy would be attached to a port connected to your IPv6 router, as the 'device-role router' allows RA's on the port.

ipv6 nd raguard policy RA-BLOCK
device-role host (default)
This raguard policy is meant to be attached to a user-facing port, and blocks any RA's received on the port.

ipv6 snooping logging packet drop
When IPv6 DHCP messages are dropped, a syslog message is generated.

ipv6 snooping logging resolution-veto
Additional logging in case the switch 'vetos' an IPv6 neighbor solicitation.

ipv6 dhcp guard policy DHCP-CLIENT
device-role client (default)
This defines a dhcp guard policy that you can attach to a port. With the role 'client' the port will block ingress IPv6 DHCP server reply and advertisement messages.

ipv6 dhcp guard policy DHCP-TRUST
trusted-port
The 'trusted-port' setting in this DHCP guard policy allows IPv6 DHCP server reply and advertisement messages. In the configuration example, we are applying the 'ipv6 dhcp guard policy DHCP-CLIENT' on each indivitual port, so we don't need to apply a trusted-port policy to the uplink interface. If you applied the 'DHCP-CLIENT' policy to the entire VLAN, then you would need to apply the trusted-port policy on an uplink, but unfortunately Cisco does currently support use of trusted-port policies on etherchannel interfaces.

ip dhcp snooping vlan 1-4094
You must specify which VLANs IPv4 DHCP snooping is enabled on, and so I enable it for all VLANs. Makes it easy when you add a new VLAN, you don't have to also modify your DHCP snooping config.

no ip dhcp snooping information option
Disables the switch from adding option 82 into the DHCP request before forwarding to DHCP server, which I've never seen needed (and can potentially break DHCP if you don't disable it).

ip dhcp snooping
Enables IPv4 DHCP snooping on the switch. DHCP snooping uses the concept of 'trust' to only allow DHCP server reply messages from specified ports. This protects you from someone plugging in a DHCP server and handing out IP addresses to your clients.

interface Port-channel1
ip dhcp snooping trust
This is an example of an IPv4 'trusted' DHCP port. I'm using a port-channel interface, but it can be any type of port, typically an interface that is facing your DHCP server.

Final Thoughts

After all that, is there anything still left for user facing ports? Of course, QoS! I won't be going over QoS in this post, but it's an important thing to research and implement correctly, especially if you have VoIP/Video in your environment.

The config above will vary based on your environment, but it's a solid start to configuring and securing your user edge. Please let me know if you have any suggestions or improvements in the comments below!

Formal verification. Two words unfamiliar to me before Networking Field Day. To provide a brief summary, formal verification uses mathematical proofs to verify a system is working as designed – the same process used to create hacker-proof code. Veriflow has taken this method and applied it to networking to verify the intended design or operation of the network.

But what does this actually mean to us as network operators? Instead of monitoring isolated metrics like an interfaces’ utilization, we can monitor the intent of the entire network as a whole, including security policies. As a practical example, you can write policy that says:

Veriflow will monitor the operational state of your network – mac address tables, route tables, access control lists, firewall rules, and alert you if your policy is violated. As more and more network devices are added to your network, this becomes an increasingly powerful tool.

Let’s get into the nitty gritty, how does this work? Veriflow’s solution is made up of two components – the “collector” and the “verification engine”. The collector is a VM that runs in your environment, using SSH credentials to log into your network devices and gather operational state using various 'show' commands.

Once the collector has aggregated the data, it forwards all network info to the verification engine, which can be another local VM or SaaS environment run by Veriflow. The nice thing about the SaaS option is that the collector VM never shares your SSH credentials with the verification system. Once the data is in the verification system, it is modeled to predict network-wide behavior and evaluated against any policy rules you have created. Each time the collector retrieves data (user configurable intervals), this is considered a “network snapshot”, and is kept as long as you have the space to store it. A typical network snapshot is 1-2 MB per device. By keeping these snapshots, you have a documented history of your network, and you can go back to any point in time to see what has changed. Besides security policies, this can help you verify network operation after an operating system upgrade, since Veriflow will inform you of any state change, for example if a routing table is incorrect because of a misapplied route-map.

For Networking Field Day, Veriflow also announced the 2.0 version of its software with two features I wanted to callout:

• First, CloudPredict enables the collector to use API credentials to connect to your AWS environment and gather routing table info, security group rules, and S3 bucket policies. Since “the cloud” can sometimes be a wild west run by developers, CloudPredict can allow you to monitor network and security changes even when others have write access to your accounts – like when you need to make sure your dev VPC is completely isolated from your PCI VPC.

• Second, Preflight will model your network changes to see potential impact before changes go live. At launch, Preflight only models ACL changes, which is a good start towards more intelligent network change control.

With these new features, and others I haven’t mentioned yet (like automatic network diagram generation), Veriflow was one of my favorite presentations at NFD16. I think Veriflow is showing us where the future of network monitoring and configuration verification is headed. Monitoring the network intent is much more elegant that writing individual alert rules for interface utilization, cpu and memory thresholds, and link up/down events. Testing network policy before applying to your live network – that’s a dream most of us can’t afford with completely redundant network hardware.

Veriflow is doing a lot right, and while listening to their presentation I couldn't help but think about how their platfrom could be expanded into additional areas. I'll be watching to see if any of the following will be rolled into future releases:

1. Expand Preflight checks to include more components, with API access so we can build Preflight config verification into network automation workflows.
2. Network utilization analysis. If a link is becoming saturated, use historical data of other existing links to know where traffic should be moved to. If a network device fails, will traffic flow overload the remaining links?
3. Gather netflow data from the network to compare operational state to actual network flows - also use with Preflight checks to show what traffic flows would be affected.

Formal verification and monitoring the "intent" of the network are exciting advances, and I believe will have a real impact for the network operator.

Veriflow 2.0 Press Release

Disclosure: I attended this presentation as part of Networking Field Day, a Tech Field Day event. All expenses related to the event were sponsored, in addition to any free swag given out by vendors. All opinions expressed on The Network Stack are my own and not those of Tech Field Day, vendors or my employer.

Duo has thorough documentation for adding MFA to your SSH sessions, but there are a couple additional steps needed to also integrate with Active Directory. This post will go through the installation for both Duo and Active Directory for Ubuntu 16.04. For other Linux distros, the Duo documentation linked above has you covered.

System components:

• Ubuntu 16.04 server
• Duo MFA account
• Duo Unix package
• Centify Express for Active Directory integration
  
  

Duo Installation:

Install OpenSSL development headers and libraries:

apt-get install libssl-dev libpam-dev -y

Add Duo repository and install duo-unix package:

echo 'deb http://pkg.duosecurity.com/Ubuntu xenial main' | tee /etc/apt/sources.list.d/duosecurity.list
curl -s https://duo.com/APT-GPG-KEY-DUO | apt-key add -
apt-get update
apt-get install duo-unix

Edit /etc/duo/pam_duo.conf with your integration key, secret key, and API host. You can find these values in the Duo Administration page for your "UNIX Application".

[duo]
; Duo integration key
ikey = XXXXXXXXXXXXXXXXXXXXXXX
; Duo secret key
skey = XXXXXXXXXXXXXXXXXXXXXXX
; Duo API host
host = XXXXXXXXXXXXXXXXXXXXXXX
; Send command for Duo Push authentication
;pushinfo = yes

Centrify Installation:

First, edit /etc/ssh/sshd_config so that ChallengeResponseAuthentication is set to "yes"

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes

Download, unzip, and install Centrify Express:

wget https://downloads.centrify.com/products/centrify-suite/2017-update-1/centrify-suite-2017.1-deb7-x86_64.tgz
tar -xvzf centrify-suite-2017.1-deb7-x86_64.tgz
./install-express.sh

Go through the installation script to connect to your Active Directory environment:

You can type Q at any prompt to quit the installation and exit
the script without making any changes to your environment.

How do you want to proceed? (E|S|X|C|Q) [X]: X
Do you want to continue to install in Express mode? (C|Y|Q|N) [Y]:Y

Do you want to run adcheck to verify your AD environment? (Q|Y|N) [Y]:Y

Please enter the Active Directory domain to check [company.com]: your-domain.com
Join an Active Directory domain? (Q|Y|N) [Y]:Y
    Enter the Active Directory domain to join [your-domain.com]:
    Enter the Active Directory authorized user [administrator]: your-admin
    Enter the password for the Active Directory user:
    Enter the computer name [server-name]:
    Enter the container DN [Computers]:
    Enter the name of the domain controller [auto detect]:
Reboot the computer after installation? (Q|Y|N) [Y]: Y

After the server restarts, you should be able to login with your Active Directory credentials via SSH, but you are not prompted for Duo MFA. To support this, we need to edit /etc/pam.d/common-auth as follows:

# lines inserted by Centrify Direct Control { CentrifyDC 5.4.1-455 }
#auth   sufficient      pam_centrifydc.so
#auth   requisite       pam_centrifydc.so deny
auth    requisite       pam_centrifydc.so

# here are the per-package modules (the "Primary" block)
#auth   [success=1 default=ignore]      pam_unix.so nullok_secure
auth    requisite       pam_unix.so nullok_secure
auth  [success=1 default=ignore] /lib64/security/pam_duo.so

In both sections, I commented out the existing config and added the necessary lines so that pam_duo is used after a successful pam_centrifydc authentication.

Next, we need to edit /etc/pam.d/sshd and comment out "@include common-auth" so we can add the Centrify and Duo PAM modules (but we don't need pam_unix):

# Standard Un*x authentication.
#@include common-auth
auth    requisite       pam_centrifydc.so
auth    [success=1 default=ignore]      /lib64/security/pam_duo.so
auth    requisite       pam_deny.so
auth    required        pam_permit.so

You should be able to SSH to the server with your Active Directory credentials, and then be prompted for Duo MFA (assuming you have a Duo account that matches your AD username)

Active Directory Group Restriction:

You may also want to restrict SSH access to users in a certain Active Directory group. This can be done with two lines in /etc/ssh/sshd_config at the end of the config, after "UsePAM yes":

UsePAM yes
#########
UseDNS no
AllowGroups root your-group-name

Be sure to add the 'root' group so that you can still manage your server with the root account if needed. Restart SSH and your group restriction will be in effect:

service ssh restart

A couple years ago, Ivan Pepelnjak wrote an interesting blog showing how all you need are two top-of-rack switches. Iwan Rahabok followed up on the systems side, with the perspective that 1000 VM per rack is the new minimum.

Now in mid 2017, I'd like to show you how we can fit 2M network routes, 1PB of storage, and over 1000 VM's in a half rack. Don't worry, I included redundancy as well.

Disclaimer: My examples use specific vendor equipment to show how this is possible, but I'm sure you can find different devices to acheive the same goal. I am not employed by any of these vendors, and only picked this equipment because it fits the design goal well.

Infrastructure:

(2) Arista 7280CR2K-30
(10) Netapp HCI 2U Servers - 13x Large Compute Nodes & 25x Large Storage Nodes

My network choice is fairly straight-forward. Dual-purpose switches with high-capacity ports that support 10/25/40/50/100 GbE, and also function as edge peering routers that handle 2M routes each.

For compute/storage, I chose to go hyper-converged, as I believe the era of big storage controllers is over. Distributed, fast storage is here to stay, and using something like Netapp HCI provides a controller-less architecture that allows you to add capacity and performance without worrying about your controller throughput. To be fair, I've been wary of hyper-converged products, but Netapp HCI retains the benefit of separate compute/storage upgrades like a traditional compute/storage stack.

Compute Sizing:

I currently run enterprise, dev, qa and prod workloads in 100% virtualized environments. I have VM's that range from 1 vCPU and 512 Mb RAM to 16 vCPU and 64 Gb RAM. I took my total provisioned vCPU/RAM and divided by total VM's to get my average use, which is 2 vCPU and 4Gb RAM per VM.

1000 VM = 2000 vCPU and 4 Tb RAM

I do not overcommit RAM, but I overcommit 5:1/vCPU:pCPU with good results. This may vary in your environment.

2000 vCPU ÷ 5 = 400 pCPU

Each Netapp HCI large compute node contains 36 pCPU.

400 pCPU ÷ 36 = 11.1 Netapp large compute nodes, lets round up to 12. N+1 means we need 13 large nodes. This will also provide almost 10 Tb of RAM, which is more than enough. Note: If you can push 10:1 overcommit on your compute, you can run 2000 VM's per half rack!

Storage Sizing:

Netapp claims 44 Tb of storage capacity with 10x space efficiency (dedup/compression/compaction) per large storage node. This may seem like a stretch, but it really depends on your workloads. My datastores see anywhere from 1.5x to 38x efficiency. My average is 23x, so 10x is actually a very reasonable number for me.

1 Pb ÷ 44 Tb = 23.2 Large storage nodes, round up to 24. N+1 means we need 25 large nodes.

Compute and Storage Totals:

13 Compute + 25 Storage = 38 Total Nodes. Each 2U Netapp HCI server holds 4 nodes, so divide 38 by 4 = 9.5 2U servers. 10 servers will give us 2 free node slots for future use, and we use a total of 20U.

Network Routing/Switching:

Each Netapp HCI node has 2x or 4x 10/25Gb ports (depending if its a compute or storage node). For our configuration, we will just use 2x 25Gb ports per node, for a total of 76 25Gb ports for the servers. We also need 2x ports for internet peering, and another 2x ports for WAN connectivity for a total of 80 ports. Each Arista 7280CR2K-30 gives us 120 10/25Gb ports, so we have plenty of capacity. The 7280CR2K-30 also handles 2M routes, so we can take full internet BGP feeds with no problem. Just separate your internal/external route tables appropriately.

Security:

To keep everything to a half rack, we can use VMware NSX with distributed Palo Alto firewalls. If you prefer a physical firewall, the Arista switches have plenty of room, you will just go over our half rack goal :)

And there you have it! 2M network routes, 1PB of storage, and over 1000 VM's in a half rack.

Using this page to keep track of all the useful powershell "mini-scripts" I've used:

Copy users from one security group to another security group

Add-ADGroupMember -Identity destination-group-name -Members (Get-ADGroupMember -Identity source-group-name -Recursive)

Add enabled users from an OU to a security group

Get-ADUser -SearchBase 'OU=Your-OU,DC=corp,DC=company,DC=com' -Filter {Enabled -eq $true} | ForEach-Object {Add-ADGroupMember -Identity 'your-group-name' -Members $_ }

Add users from a CSV file to a security group (username column is called "name")

import-csv c:\users.csv | Foreach-Object {add-adgroupmember -Identity your-group-name -Members $_.name}

Update user objects with organization attributes from a CSV file

import-csv c:\users.csv | Foreach-Object {Set-ADUser -Identity $_.name -Department $_.department -Title $_.title -Manager $_.manager}

Create security-groups from a CSV file (group-name column is called "name")

import-csv c:\groups.csv | Foreach-Object {new-adgroup -name $_.name -GroupCategory security -GroupScope global -Path "ou=Your-OU,dc=corp,dc=company,dc=com"}

Export list of all enabled users in an OU

Get-ADUser -SearchBase 'OU=Your-OU,DC=corp,DC=company,DC=com' -Filter {Enabled -eq $true} -Property * | Select -Property Name,SamAccountName,UserPrincipalName,mail,Title,Manager,Department | export-csv 'c:\users.csv' -notype

Export list of all enabled users in an security group

Get-ADGroupMember -Identity your-group-name | ? ObjectClass -eq "User" | Get-ADUser | ? Enabled | Select Name | out-file C:\group_membership.txt