Skip to content

Operation Guide: Network

AWS Vendor Connectivity Solution (Confluent)

Note on public clusters

Any public facing cluster will use the standard config on Confluent and the details described here are all concerning private clusters and resources otherwise needed by the platform to function.

Changes

This solution is qualified and a change control process must be followed for changes. Git tags are used to track code releases. All code releases must be tagged with a new version. Please contact someone from the Confluent team prior to making any changes in the main branch.

The setup that is operated and maintained by the KaaS team: Incidents and Changes can be assigned to the assignment group KaaS-L3-GL-IT/ HCL-KaaS-L3-GL-IT in ServiceNow. If raising an incident from the IT service portal, the Kafka as a Service (KaaS) Business application should be selected.

MAP and KaaS team are overall responsible for the operation and maintenance of the Network setup provided by GITO by creating and defining CHG requests in ServiceNow. The Network team delivers the configuration and operation support on need basis in the current scenario. A standard IT support process shall be followed to create a case with Global Service Desk

For Major or Severe IT incident, please create a case with Major Incident Team (Situation Management Team)

For more information on Network operations and support please refer (Network Operations & Support - Overview page.

Overview

The vendor connectivity solution is composed of a set of standard AWS resources which are configured to work in a specific way. The resources are deployed in dedicated vendor specific AWS accounts within the Novo Nordisk AWS organization.

The solution provides two key functions:

  1. Enable connectivity between a third party and Novo Nordisk via a transit gateway. (Connectivity into the CORP network traverses a firewall)
  2. Enable hosts on third party connected VPCs to perform DNS lookups against internal and external domains.

The resources which enable this functionality are deployed / configured both manually and through AWS CloudFormation templates.

Architecture

see connectivity network

CloudFormation

The CloudFormation stacks deployed from this repo are:

  1. vendorConnectivityBase (This template creates the base resources for the vendor connectivity solution)
  2. vendorConnectivityDNS (This template creates the DNS resources for the vendor connectivity solution)
  3. vendorConnectivityConfluentRouting (This template contains the Transit Gateway routing tables for Confluent EU connectivity)

Account Preparation

  1. Create an AWS account called AWS-NET-Confluent-TST in the test organization. (TST Environment Not Controlled)
  2. Create an AWS account called AWS-NET-Confluent-PRD in the production organization.
  3. Set up the OpenID Connect config with this github repo and the new accounts from step 1 and 2.
  4. Execute New-AWSDNS.ps1 from the AWS deployment server. This deploys a PrivateHostedZone in the account. (Outside of this repo)
  5. From the resource access manager in the AWS-NN-Core-DNS-PRD account add the AWS account # from step 2 to the principal list for the dns-prd-r53-euc1-rs resource share. This allows sharing of the core DNS resolver rules.
  6. If present delete any internet gateways that may exist in the regions you are using (e.g. eu-central-1, us-east-1) in the accounts from step 1 and 2.

Deployment / Configuration

To setup Confluent production connectivity a series of steps must be completed.

The AWS Direct Connect connections and related resources must first be setup, but those steps are not currently documented here.

Complete the following steps for each region you are deploying to. Variables are used in the step to represent differences based on region:

  1. Deploy the base cloudformation template by executing the VendorConnectivityConfluentBaseeuc1 action in GitHub.
  2. Delete the default dhcp option set from the utility VPC.
  3. Associate the DirectConnect Gateway with the TransitGateway.
  4. From the AWS console use Resource Access Manager to share the transit gateway with all Confluent principals.
    • 050879227952
    • 563559260525
    • 259082970751
    • 390327825978
    • 457492987184
    • 417601102659
    • 479111403931
    • 121194963621
  5. For vendors without the need for DNS resolution, associate the utility subnets with the Default Network ACLs instead of the Tier 1 ACL. Configure the default ACL to block DNS queries from entering the utilityVPC.
  6. Modify the DNS config file (/config/Confluent/dns-euc1-prd.json) and update the relevant variables based on the resources deployed in step 1.
  7. Deploy the DNS cloudformation template by executing the VendorConnectivityConfluentDNSeuc1 action in GitHub.
  8. From the Confluent console request the relevant transit gateway attachment.
  9. From the AWS console accept the transit gateway attachment.
  10. Steps 8 and 9 should be repeated for each attachment required
  11. Update the routing cloudformation (/cfn/vendorConnectivityConfluentRoutingeuc1.yml) to contain the correct information.
  12. Update the routing configuration (/config/Confluent/routing-euc1-prd.json) to contain the correct information.
  13. Deploy the routing tables. template by executing the VendorConnectivityConfluentRoutingeuc1 action in github.
  14. Steps 11,12,13 will need to be executed repeatedly if all attachments aren't added initally.

DNS Resolver

The vendorConnectivityDNS.yml deploys DNS inbound endpoints to be used by the vendors for DNS resolution. A DNS resolver firewall is also deployed to control which domains can be queried. The DNS resolver firewall uses a domain allow list to determine which domains the resolver will do lookups against. The domain allow list is deployed initially from a txt file located in an s3 bucket Confluent-euc1-Domain-Allow-List.txt). Changes after the initial deployment are done directly from the console. Updates from the console are required because updating from cloudformation would require a re-deployment of the template which would cause an interruption and the firewall to be dropped. The change rate is assumed to be extremely low.

Transit Gateway Peering

  • If two vendors need to communicate directly then their transit gateways needs to be peered manually.
  • The proper routing tables must be created for the peered transit gateways.

Router / Firewall Config

The router/firewall currently only supports a 1500 MTU. The networking team must set the mss (maximum segment size) on the vendor sub interface otherwise servers configured with a large MTU will not be able to communicate with the vendor. The following line must be added to the sub interface: ip tcp adjust-mss 1460

Direct Connect Config

Allowed Prefixes for the Direct Connect to Transit Gateway Attachment EUC1

10.190.82.0/24
10.191.0.0/16
10.192.0.0/16
10.193.0.0/16
10.194.0.0/16

Subnetting

Direct Connect

  • EU-Direct Connect /30 10.190.80.0/24
  • US-Direct Connect /30 10.195.80.0/24

Confluent

  • EU Utility VPC 10.190.82.0/24
  • EU DEV 10.191.0.0/16
  • EU TEST 10.192.0.0/16
  • EU QA 10.193.0.0/16
  • EU PROD 10.194.0.0/16

  • US Utility VPC 10.195.82.0/24 (NOT IN USE)
  • US DEV 10.196.0.0/16 (NOT IN USE)
  • US TEST 10.197.0.0/16 (NOT IN USE)
  • US QA 10.198.0.0/16 (NOT IN USE)
  • US PROD 10.199.0.0/16 (NOT IN USE)

Maintenance

Updating the Domain Allow List on the DNS Resolver Firewall

The list of domains for which lookups can be performed against is restricted. The domain allow list was initially configured using a txt file located in an s3 bucket (Confluent-euc1-Domain-Allow-List.txt). Subsequent changes to the domain allow list must be made directly from the AWS console. The source file in GitHub must also be updated when changes are made in the console. This is in case the connectivity solution needs to be redeployed. The Confluent team must open a CR prior to commiting changes to main or modifying production.

The following steps should be followed to update the Domain Allow List. somedomain.com must be changed to the actual domain you are adding.

  1. If you don't already have a development branch for the vendor-connectivity-confluent GitHub repo create one from the main branch.
  2. In the development branch update /config/Confluent/Confluent-euc1-Domain-Allow-List.txt by adding *.somedomain.com to the file.
  3. Commit this change to the development branch.
  4. Create a PR and merge this change to the main branch. (ensure the PR has a reviewer from the Confluent team)
  5. Add a new version tag to this commit in the main branch. The code below must be changed to match the appropriate version and domain values. This example uses the git command line and assumes you are in the proper repo. - git checkout main - git tag -a v2.1.0 -m "added *.somedomain.com to domain allow lists" - git push origin v2.1.0
  6. From the AWS console of account AWS-NET-Confluent-PRD, manually added *.somedomain.com to the AWS-NET-Confluent-PRD-Utility-VPC-Resolv-Fw-Domain-Allow-List Domain list in the eu-central-1 region.
  7. Verify the change fixed the issue by performing a DNS lookup of (somehost.somedomain.com) against the route53 resolver. The following example uses the dig utility. - dig @10.190.82.2 somehost.somedomain.com

Shutting Down Direct Connect Connections (Virtual Interfaces)

Direct Connect Connections should NOT be shut down unless being done in coordination with the network team. There are very few situations where shutting down the interfaces would be desirable. There are two Direct Connect Connections in eu-central-1 and each one has a virtual interface. From the AWS console you can select one of the virtual interfaces and then perform the Action "Bring down BGP". Stopping BGP will cause all traffic to run over the redundant link.

From the AWS console, the path to the interfaces in eu-central-1 is shown below: - AWS Direct Connect -> Virtual interfaces -> DXVIF-FGCOIWIT - AWS Direct Connect -> Virtual interfaces -> DXVIF-FGP0MRBS

Incident Contacts

  1. AWS related incidents can be assigned to the ServiceNow queue (Cloud-AWS-L3-GL-IT)
  2. Please follow the standard IT support process and create a case with Global Service Desk via email, IT portal or a phone call.