Appearance

Powered by Platform One

Party Bus Frequently Asked Questions ​

Repos ​

1. How much access does the product team get to the repo? ​

Your organization will make the determination as to how access is given for a repo/project. Each team is designated a PM/Onboarding supervisor who can onboard and set role levels as needed

2. Is there any documentation for importing an existing project? ​

We do not have P1-specific instructions. The generic GitLab instructions should be sufficient: https://docs.gitlab.com/ee/user/project/settings/import_export.html

3. How can product teams set up repo mirroring? ​

We do not allow repo mirroring from P1-external sources. We have security controls like signed commits on commits so that only approved users can add code. The only scenario where mirroring is allowed is from repo1 to Party Bus IL2, in which case all mirroring still has to be configured by a Gitlab Admin, which is Party Bus personnel only.

4. How to invite members to a GitLab group? ​

Users must already been registered users in GitLab. E-mail notifications are disabled. The Onboarding supervisor was designated as an Owner. They will be able to manage membership of the group. Onboarding ticket for Gitlab for reference

5. Are there any app size limits that repos can host? ​

We do not have a hard limit, but we do monitor pipelines for excessive resource usage and may shut down pipelines/repos if so. If a repo has many files/lines of code to process, some jobs, e.g., Fortity, may struggle to complete successfully within job timeout periods. We recommend breaking large repos into multiple microservices. Above baseline resource requirements may require additional funding to support. Repos support GitLab Large File Storage (LFS) and large files can be committed.

Pipelines ​

HowTo - GitLab - Pipelines

1. What logs are available to review? ​

You can review the Gitlab CI/CD pipeline log output as well as Argo CD logs (if access has been granted).

2. Where does code reside when deploying to IL6? ​

If you deployed your staging environment to IL2/IL4/IL5, your code base will remain there while the IL6/JWICS deployment is passed over to the proper POC's at DAF Cloudworks

3. What e2e frameworks are supported? ​

We primarily use Cypress, but can invoke integration tests for pytest as well. Please open a ticket for any other e2e framework support

4. Can we run parallel test in Cypress? ​

Yes, invoke cypress-parallel

5. Can we have a dev deployment environment, i.e. another environment before staging? ​

For most use cases we only provide two deployment environments: staging and production (which is only available after the CtF has been obtained). However, recently we have introduced the concept of Auxiliary staging and production deployments, which can be set up as variations of your existing deployments. Note that there are important considerations when it comes to pricing and resources for auxiliary deployments

6. We have a repo using multiple languages. Is there any support for that? ​

No, our pipelines are technology/language-specific; the repos need to be broken up

7. Is these any chance to support "new" languages? ​

While everything is in the realm of technical possibility, we must take into consideration resources to develop support for new languages. Additionally, Our scanning tools do not support languages such as Fortran, COBOL, etc, so we would be unable to check for any vulnerabilities. If a new language/package manager is desired, you can put in a feature request , but if there is low demand for a specific technology, we do not have the resources to design, implement, and maintain the pipelines needed to support it.

8. Can we have repos that are dependent on other custom libraries? ​

Yes, we can set up package pipelines which will result in an artifact that can then be imported into another repo/project. You must be able to import the code for said libraries into another gitlab project and utilize the Party Bus package pipelines to build them. You will not be able to import compiled packages unless given cyber approval to do so. Considerations will need to be made for funding another pipeline

9. What is the typical turnaround to create a pipeline? ​

It varies widely from product team to product team. Other teams also need to be involved, e.g., CNAP, DNS, Onboarding. Once an MDO engineer grabs your pipeline out of the queue, it can take anywhere from 1 day to 2 weeks, depending on how compliant it is with our standards. Updates to the repo to address SCA findings or set up unit testing are on your product team, but can be worked on your own timeline, but will block progress until they are resolved.

12. My anchore/twistlock scans seem to be failing because of RedHat/OS-level vulnerabilities ​

This is common due to base image issues and whitelisting. Vulnerabilities that are "inherited" from the Ironbank base images are automatically whitelisted, however this only applies for tags that have been built more recently. Please ensure you are using the most recent tag for your image, which can be found at https://ironbank.dso.mil. If you are still having issues, re-open a ticket. Additionally, please refer to How to: Docker Renovate on how to utilize Renovate in order to keep your base image up-to-date

13. Can someone on the Party Bus/MDO team assist with cypress/e2e testing? ​

We cannnot help you design your e2e tests or troubleshoot any cypress/e2e code, but we do ensure that your e2e test is functional from a platform perspective. i.e. we make sure you can reach your staging deployment from the pipeline and it returns a non 4xx/5xx http code. See more information on end to end testing

14. What are some common whitelists that are granted? ​

  • Vulnerabilities for parsing and executing untrusted data
  • Transient dependencies
  • Misattributions to other dependencies
    • Dependency Check uses fuzzy string matching and will sometimes associates it with similarly-named dependencies

15. What recourse does the product team have for vulnerabilities that cannot be resolved? ​

Any whitelist requests require justification. If we deem the justification insufficient or the risk too large, we may decline to whitelist the vulnerability.

16. Is it possible to downgrade the severity of SonarQube findings? ​

This is typically not offered. Depending on the number of requests, it may be granted.

17. What type of tests are required for the pipelines? ​

We require unit tests, end-to-end tests, and penetration tests. The product team is responsible for developing their own unit and e2e tests. Penetration tests are performed within the pipeline and by the CAT team

18. My pipeline job is running out of resources. Can I get more memory and/or CPU? ​

All pipeline jobs are given 10Gi of memory and 4 dedicated vCPU cores maximum. We do not have the ability to increase memory on a per job basis. If this is not sufficient, you will have to work with customer success and PB OPS to allocated a dedicated node to run high resources jobs. Please email the BAMS at AFLCMC.HNCX.BAM@us.af.mil to start this process

19. My unit tests are taking an abnormal time to finish, especially when compared to when I run them locally ​

We have seen, especially in the case of jest/npm, that the underlying code attempts to utilize every CPU possible when running. This is mostly fine, however our Gitlab CI pipelines run on kubernetes jobs and the CPU resource counts are abstracted away so jest ends up reading the underlying EC2 kubernetes node resources, which are significantly higher, thereby causing long response times. We have found that adding "–maxWorkers=4" to your unit test scripts results in some reduced test time

20. Why is my namespace-health-check job failing? ​

There are many answers to this question, but the best approach is to think of this job as more of an "e2e/pen test readiness" stage. The deploy for the microservice you are pushing out may have actually succeeded but there could be other microservices failing in your environment. In most cases you will need to check ArgoCD to see what this issue is. See question 7 below

21. Do pipelines support multiple branches? ​

Only the default branch (initially master) is allowed to deploy or generate a package. Any other "dev" branches will run the pipeline up to (but not including) the deploy-staging/publish-package stage.

22. Can product teams use tags to manage releases? ​

Product Teams cannot tag releases directly. The pipeline release-xxx stage automates the process and manages the release tags in correlation with the CtF version. This feature is enabled after receiving a Certificate-to-field (CtF).

23. How can we exclude certain files from deployment? ​

We scan all code files and persist them as pipeline artifacts. Any of those files can selectively be deployed to the Docker image.

24. We use a DAF Cloudworks version of GitLab on NIPR. Can we use that? ​

No, product teams are required to use our instance of GitLab.

25. If a product team is onboarded without pipeline support (GitLab seats only), are they able to utilize the pipeline-templates? ​

The pipeline templates repo is publicly available so they can use it as a reference However, We do not allow product teams to run pipelines if they have not purchased the support to do so

26. My Twistlock scans are coming back with findings that start with PRISMA-. What does this mean? ​

PRISMA-* findings are those that Palo Alto detects but do not yet (or ever) have a CVE ID. However, They can still be valid findings. Please address these as you would any other finding or create whitelisting ticket if you feel they are false positives. View more information on Prisma Cloud

Containers/Deployments ​

1. Is Docker Compose supported? ​

No, we use the Kubernetes container orchestration framework.

2. Do product teams need to set up pipelines for common container services like kafka, redis,etc? ​

No, pipelines do not need to be setup. Containers using Iron Bank images can be added easily if no special configuration or additional packages/code is required

3. Are Helm charts supported? ​

We use Kustomize to manage Kubernetes manifests and while ArgoCD may support helm in some capacity, Party Bus does not support this means of deployment

4. What is ArgoCD? ​

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes, which will allow you to monitor your application and its kubernetes configuration. Permitted users will be able to view the standard output for deployments and restart pods. For access to ArgoCD, please have your onboarding supervisor open a ticket

5. Are generated logs accessible through ArgoCD? ​

No, ArgoCD will display stdout/stderr from your deployments but output files cannot be accessed directly through it. Any output files can be persisted into S3 (if purchased).

6. Is there a way to get extended logging for ArgoCD? There appear to be issues getting the previous logs. ​

ArgoCD will display the output from the current running containers (and the previous if a crash occurred at the container level). If the pod resets, the logs are gone; this is a limitation of Kubernetes. This is a common request, but it is not something we have been able to implement yet.

7. Does P1 offer W3C log (or other) data aggregation/monitoring? ​

There is currently no service available as we are trying to implement a multi-tenant solution with the proper boundaries between customers.

8. My deployment is failing/I can't reach my application, I am getting a "connection reset"/503 errors. What is wrong? ​

For any deployment issues or web page problems, ArgoCD is going to be your best friend. Here are some common error codes and what they mean

Code/ResponseIssueResponse
503Microservice is crashing due to various issuesCheck argo for any problems in your logs. If logs are not indicating issues, check the status tab for your pod in argo and look for the status OOMKilled. This means that the pod does not have enough memory. You can provision more, up to 2G per pod.If all else fails, please open a ticket
301RedirectCheck your application for redirects. Additionally, if you are trying to reach the external address application from a non-browser based session, you will get redirected to keycloak, which will dead-end your session.
RBAC: Access DeniedVirtual Service missing or authconfig missingPlease open a ticket

9. Does Party Bus Kubernetes support node affinity for containers that need persistent storage, e.g., mongodb containers? ​

Yes, product teams are allowed to make updates to the app-manifest (which contains their Kubernetes configuration) as long as it does not violate any Party Bus requirements. See How-To Manifests

10 Is there anyway to dynamically spin up resources as workers in the cluster? ​

This would be configured in the app-manifest. Product teams are allowed to make updates to deploy pods and have them autoscale.

11. Are pods/clusters restarted periodically? ​

Yes, Party Bus ingests Big Bang, which will sometimes necessitate cycling pods to deploy upgrades, e.g., to istio. Upgrades are enforced to stay in parity with our security posture.

12. Is there any way to deploy components utilizing unsupported technologies? Can ATO still be achieved? ​

Since we cannot scan repos utilizing unsupported technologies list, we cannot set up pipelines for them. The alternative is to have a vendor, or someone, sponsor that image into Iron Bank.

13. Does Platform One support AWS QuickSight? ​

Apache Superset is in Iron Bank, but AWS QuickSight is not

14. What are the resource limitations? ​

There is no hard cutoff. We allow product teams to configure resource allocations for their containers but it should be within reason. In the multi-tenant cluster (t3.medium to t3.2xl), that's usually about 100 Gigs of storage, .5 Gigs of memory, and .25 CPU (peak). For single-tenant solutions, limits are determined by AWS and what you pay for.

15. Is it possible to deploy multiple apps on a container in Kubernetes? ​

Each app is it own cluster. Each repo/project should be a microservice that results in a single deployment container. Traffic between pods within an app/cluster is open.

16. What is the difference between the Iron Bank Catalog and Harbor? ​

The Iron Bank Catalog and Harbor are different views of the same Docker registry. Harbor is the open-source implementation whereas IB Catalog is the P1-built solution (which we recommend over Harbor).

17. What is the difference between K8s nodes and pods? ​

A node is a worker machine (can be physical or virtual) in Kubernetes that serves as a computing resource A pod is the smallest deployable unit in Kubernetes and represents one or more containers that are tightly coupled and share certain resources.

18. How are IPs for the apps set?* ​

Statically. We use Amazon Elastic IPs which forward to Elastic load balancers in our account which then forward to Istio gateways which listen on NodePorts to then route traffic to pods.

19. Is it possible for an app at a higher IL to send data to an app in a lower IL? ​

It is technically feasible but not approved by Cyber at this time

20. How to request new Iron Bank images? ​

/platform-one/academy/grogus/products-services-teams/iron-bank/#how-to-request-new-containers

21. Could a team develop at IL4 staging and deploy their application to IL2 prod? ​

Staging Deployments must be done at the same IL where the code resides and Production deployments can be the same IL or higher

22. Can I or should I manage my own https/TLS within my application? ​

Party Bus utilizes Istio for all things TLS. This means that any https you require for serving out your application will be done utilizing Istio ingress gateways and virtual services. Additionally, Istio does mTLS encapsulation for traffic so connections within the cluster or microservice-to-microservice connections, although they may be configured as cleartext/http, will be protected by mTLS. Therefore there is generally no need to generate or manage certificates, unless there are very specific cases where this is necessary.

Databases ​

1. What tools are provided for managing a teams database ​

Currently, Party bus customers have the option of managing Postgres through the use of PGAdmin. See our pgadmin documentation for more information.

2. Can product teams initialize their own DB's and create tables, users, etc? ​

Use tools like flyway that contain migration scripts, creation scripts, etc. Databases are initialized at the deploy stage with your own schema scripts.

3. How is database migration handled? ​

Product teams should handle database migration through their application or app-manifest, but a help desk ticket can be opened if an administrator is needed to dump or seed a database.

4. What is the maintenance schedule for databases? ​

There are automated daily backups for the last 7 days. Additional manual snapshots are taken at critical points.

5. Can I connect to my RDS from my pipeline? ​

We do not allow the connections from gitlab to the RDS due to the separation of boundaries between the clusters/environments. If you need to do any database operations during unit testing or e2e testing you can utilize Gitlab Services (Ticket needed). Otherwise you can programmatically add database procedures in your deployed application that can be accessed via api calls or otherwise. We will also note that in some cases doing database connections during a unit test can be considered an anti-pattern and also require extra effort from the Mission DevOps team to configure and help troubleshoot them. Therefore they are highly discouraged.

Checkstyle ​

1. Are check styles required for lint? ​

Yes

2. Can check styles for lint be customized? ​

Linting is customizable by product teams. Usually linting tools will have a config file associated that it automatically picks up ie pylint looks for a .pytlintrc file. This is tool dependent.

Miscellaneous ​

1. Best way for product teams to do notifications? ​

P1PS is a new offering from Party Bus that allows mission applications to send emails via Amazon SES. Please reach out to the Bullhorn team](https://chat.il2.dso.mil/platform-one/channels/p1ps-help-channel) for more information here or open a [ticket with the P1PS team. Note that this notification service is not available for collaboration apps like gitlab.

2. What AWS services are available to product teams? ​

We provide integration with RDS (PostgreSQL and MySQL) and S3. Other services may be supportable, but are not currently drop-in ready; a feature request will need to be submitted by the product team.

3. Where can further questions be asked? ​

The Value Stream - Party Bus - Support Mattermost channel is a good starting place for assistance. Simple, general questions can be asked there and there are links to the various support desks and documentation. We'll also be able to direct you to the right people for Party Bus requests from there.

4. Does P1 provide an artifact/package repository like Nexus? ​

We use GitLab's package registry to host artifacts and packages created by pipelines. GitLab's container registry is used to host containers created by pipelines.

5. Can I use my own keycloak server or can I use direct integration into login.dso.mil ​

Unfortunately we don't allow or recommend the use of a per-team keycloak instantiation. Party bus uses the Platform One keycloak for all operations. This means that you will need to use the JWT/Authorization header to authorize users and do any sort of RBAC. For more information on authservice, which is used to broker connections with Keycloak in Party Bus, and how it is used within Platform One, please reference the Big Bang docs

6. Do Platform One and ODIN share a keycloak instance? ​

No, each platform utilizes its own keycloak instance.

7. What is defined in a CtF letter? ​

A CtF letter authorizes a single application and its specific services/deployments for one (or more) production environments. Not all services deployed to staging need to be deployed to production. The CtF letter will specify what impact levels, the effective time period, and the major software version release in which it is valid.

8. How can we set up a working session with MDO to resolve an issue? ​

A help desk ticket should be opened if you have an issue that needs MDO attention. The assigned engineer will use the Jira ticket to communicate progress and relevant information. If a meeting needs to be held, it will be arranged.

9. What necessitates a new CtF? ​

  • Adding a database
  • Adding a service/pipeline
  • Adding another external connection

10. What kind of access is provided for AWS S3? ​

We expose connection details for the S3 bucket via environment variables that are available in deployed applications. IAM Roles as Service accounts also known as IRSA are used to broker the connection from your pods to AWS. more information on S3 We grant permissions to ListBucket, GetObject, PutObject, DeleteObject.

11. How can we escalate an issue if a ticket is not being addressed? ​

Reach out on Value Stream - Party Bus - Support

12. What Kubernetes distribution is Party Bus Using? ​

Currently Party Bus is using RKE2 but has plans to Migrate to AWS Elastic Kubernetes Service (EKS)

13. What are the SLAs for support tickets? ​

View our Party Bus SLAs

14. What is the difference between IL2,IL4,IL5 and beyond? ​

  • https://media.defense.gov/2020/May/18/2002302035/-1/-1/1/NAVY_TELEWORK_CAPABILITIES_V14.PDF
    • DoD Impact Levels are determined by a combination of: the sensitivity of the information to be stored and/or processed in the cloud
    • the potential impact of an event that results in the loss of confidentiality, integrity or availability of that information
  • The Impact Levels:
    • IL2 accommodates DoD information that has been approved for public release (Low Confidentiality & Moderate Integrity)
    • IL4 accommodates DoD Controlled Unclassified Information (CUI) (e.g., FOUO)
    • IL5 accommodates DoD CUI & National Security Systems (NSS)
    • IL6 accommodates DoD Classified Information up to SECRET

Party Bus ​

1. How did the Platform One start? ​

The Platform One software factory was initiated by the United States Air Force (USAF) as part of its efforts to modernize its software development practices and enhance its digital capabilities. It began in 2018 as a pilot program within the Department of Defense (DoD) to streamline software development processes and enable faster delivery of software solutions to support military operations.

Platform One was established to address the challenges faced by the military in adopting modern software development practices, such as agile methodologies, DevOps principles, and cloud-native technologies. These practices aim to accelerate software delivery, improve collaboration among development teams, and increase the agility and responsiveness of the military's software systems.

The initiative was driven by a recognition within the military that traditional software development processes were not keeping pace with the rapidly evolving technological landscape. By establishing Platform One, the USAF sought to create a centralized platform and set of tools that could be leveraged by various branches of the military to develop, deploy, and manage software more efficiently and securely.

Platform One emphasizes the use of open-source software, automation, and continuous integration/continuous deployment (CI/CD) pipelines to enable rapid and reliable software delivery. It also provides guidance and support to development teams across the military in adopting these modern software development practices.

Overall, Platform One represents a significant shift in how the military approaches software development, with a focus on agility, collaboration, and innovation to meet the evolving needs of modern warfare.

2. How to configure outbound traffic to an external https endpoint? ​

Customers will need to submit an egress request here for staging](https://jira.il2.dso.mil/servicedesk/customer/portal/73/create/524) or [here for production . Please note in your requests if and when you are trying to reach a service within NIPR (Non-classified Internet Protocol (IP) Router Network) as this will take extra time to work through the DISA egress request process. If you are unsure, check with the endpoint owner or your program office

3. I have an external service that needs to connect to Party Bus. Is this possible? ​

Party Bus now has an NPE workflow that uses mTLS to broker a zero-trust connection from an external entity to within the PB boundary. This does not go through the normal means of authentication, i.e. Keycloak, but instead relies on special Istio configuration. Please use the Jira service desk to open a ticket to get an NPE connection set up. This will require the "client" to be able to broker an mTLS connection using a certificate and key generated and provided by Platform One.

Powered By Platform One