Self-Hosted Development Servers

2024-12-01

Tags:

I work most of the time with European and Swiss companies. Regularly, some of them are slightly paranoid and deny access to cloud services for their collaborators.

You are forbidden to use well-established software development platforms such as GitHub, GitLab, or Bitbucket.

These solutions are the best of breed in our industry. The price per developer and year is around 250 Euro. You get the whole set of functions and goodies such as single sign-on and auditing of all activities.

The same companies not willing to use cloud solutions are often not able to provide financial and technical resources to build a similar onsite offer ^[1].

The development teams have no access to modern source code management or build pipelines.

How can you efficiently work in such a retrograde environment?

We present pragmatic recipes to alleviate the damage and help your team to work productively in such environments. You still have to acknowledge the necessity of a dedicated and knowledge administrator. He will administer and update the deployed solutions and support the teams.

You need a virtual machine or a physical workstation with reasonable memory, processor, and persistent storage resources. You must have local administration rights in this environment. Ideally, you should have a reasonable domain name to access the machine or at least a fixed internal IP address. ^[2].

The key functions we shall provide are:

Source Code Management: The source code management shall be based on Git. The solution shall provide support for pull requests to support distributed teams and home office.
Continuous Integration, Delivery, and Deployment Pipelines: The application shall be built in a clean environment [1, 2].
Artifact Management: Generated artifacts shall be published in a repository. Semantic versioning shall be used for traceability. Solutions depending on the artifacts shall fetch them from the repository. The reproducibility of the deployed applications shall be guaranteed.
Software Documentation: Agile development approaches promote incremental and iterative working [3, 4]. The documentation solution shall support these concepts. The project documentation shall be updated after each completed story. You should be able to have access to multiple older released documentation sets with just one command. The documents shall be searchable.
Project Management Services: Collocated teams can work with analog boards [5, 6]. Geographically distributed teams need an electronic board as an information radiator for all project members.

If your organization has partial support for the above function, try to configure a hybrid solution.

You have, for example, an organization-wide Git server but no continuous integration pipelines. Use multiple remotes in your project to easily push to your local and to the organization servers.

Better would be to set up a mirror configuration. This is sadly seldom supported or allowed in corporate setups.

Source Code Management

Forgejo is a self-hosted lightweight software forge build on top of Git. It provides essential functions to work efficiently as a team with a reasonable footprint. ^[3].

Easy to install and low maintenance, it just does the job.

If you like any of the following features, Forgejo is literally meant for you:

Lightweight: Forgejo can easily be hosted on nearly every modern machine.
Project management: Besides Git hosting, Forgejo offers issue tracking, pull requests, wikis, kanban boards, and much more to coordinate with your team.
Publishing: Have something to share? Use releases to host your software for download, or use the package registry to publish it for docker, npm, and many other package managers.
Customizable: Want to change your look? Change some settings? There are many config switches to make Forgejo work exactly like you want.
Powerful: Organizations and team permissions, CI integration, Code Search, LDAP, OAuth and much more. If you have advanced needs, Forgejo has you probably covered.
Privacy: From an update checker to default settings. Forgejo is built to be privacy first for you and your crew.

Continuous Integration, Delivery, and Deployment

Modern software development approaches require access to a continuous build pipeline. Either you write your own approach using build script and docker images, or you use a service.

Forgejo provides access to an experimental build server. You can also use other open source solutions such as Woodpecker.

Open source or commercial products such as Jenkins, TeamCity, or Bamboo are naturally available for decades.

Do not underestimate the effort to have the build environment always working. You need a dedicated expert who is responsible for the smooth operation of the build environment.

Artifact Management

Artifact server is used to store the artifacts created during the delivery activities in the pipeline.

The Forgejo application provides support for a set of artifact managers. There is no need to install an additional server to service package requests.

Alternative open source and commercial repository managers are available. We have experience with Apache Archiva and Sonatype Nexus Repository. These products are too powerful and complex for smaller organizations.

The Java community uses Maven and Gradle Build Tool to handle artifact dependencies.

The Python community has the pip3 tool. The C and C++ community often use Conan and vcpckg.

Software Documentation

The current approach to writing technical documentation is to use static website generators. GitHub, GitLab, and Bitbucket all offer such as solution through their pages concept.

We made a positive experience [1] [2] [3] with:

The toolchain Hugo as an efficient static website generator. Fell free to use another tool.
The technical documentation theme Docsy as site theme is working well for us. The theme is used in multiple well-known open source projects. Some examples can be found under Docsy Site Examples.
All documents are written with Asciidoc. AsciiDoc is a plain text markup language for writing technical content. It is packed with semantic elements and equipped with features to modularize and reuse content. AsciiDoc content can be composed using a text editor, managed in a version control system, and published to multiple output formats.
Software architecture is documented using arc42 template. The authors provide Asciidoc version of the template. This approach promotes the use of C4 Model model and Architecture Design Records ADR. Most of the diagrams are created with plantUML.

Beware of the drawbacks of wiki approaches. Wiki-based solutions are effortless to deploy.

Wikis often have the following problems:

It is cumbersome or challenging to version the content of a wiki site. You want to deploy the documentation associated with a specific release of your application. You often need multiple wiki versions at the same time to support the delivered version, the version under testing and the version under active development.
Support for diagramming in the wiki pages is often limited or only available through commercial plugins. You should never generate or manually copy diagrams. Asciidoctor support more than thirty diagram types out of the box. Source code highlighting for huge range of programming languages is also available in Asciidoc.
You must use a wiki supporting storing all contents in a git repository. You need the tracking and versioning features of a source code management solution.

The source code documentation approach shall be tailored to the used technology stack. Try to follow the good practices of your technology stack and the conventions of the organization you are working for.

Code is more often read than written.

— Guido van Rossum

Java code is always documented using Javadoc toolchain footonte:[Take time to study the standard Java library documentation. The Java API is recognized as one example with very high-quality.]. Code snippets introduced with JEP 413 greatly improve the quality of source code examples.

The Python community uses DocStrings. See PEP 257 for detailed information. Modern python developers also make use of type hitting for improved legibility.

The C and C++ developers mainly write API documentation with Doxygen.

Project Management Services

You need basic functions to work using agile approaches. If a team is working in the same location, use an analog board as a sprint board.

The issues can be tracked with a tool like Forgejo if you really need the additional functionality.

The solution also provides a minimal kanban board for geographically distributed teams.

Provide adequate tools for synchronous and asynchronous communication between team members. Quality video cameras and headsets are a must.

A simple test is to check if all video conference attendees always have their video feed active ^[4].

Channels and rooms in the chat application ^[5] shall be available to all internal and external team members.

Tips and Tricks

The bare minimum of services you need is a central git repository, a possibility to build your software in a clean environment, and a site to publish your documentation.

For each service you deploy, you must implement a backup strategy. The easiest approach is to regularly archive the project data and save it on a company network drive. Often the amount of data for an initiative is less than one GB. Bigger projects are below 10 GB. Remember your artifacts are plain text files, not huge office documents or binary files.

You shall advertise your approach in your organization. Better solutions shall be adopted to improve the efficiency of digital product development and optimize costs.

References

[1] N. Forsgren, J. Humble, and G. Kim, Accelerate: The Science of Lean Software and DevOps. IT Revolution Press [Online]. Available: https://www.amazon.com/dp/B07B9F83WM

[2] D. Farley, Continuous Delivery Pipelines. 2021 [Online]. Available: https://www.amazon.com/dp/B096YGZVZ9

[3] N. Ford, R. Parsons, and P. Kua, Building Evolutionary Architectures: Automated Software Governance, Second. O’Reilly Media, 2023 [Online]. Available: https://www.amazon.com/dp/B0BN4T1P27

[4] P. Clements, Documenting software architectures views and beyond. Addison-Wesley, 2011 [Online]. Available: https://www.amazon.com/dp/0321552687

[5] E. Brechner, Agile Project Management with Kanban. imusti [Online]. Available: https://www.amazon.com/dp/B00U017N4A

[6] J. A. Highsmith, Agile project management. Addison Wesley Professional, 2010 [Online]. Available: https://www.amazon.com/dp/B002HMJYAG

1. Do not understimate the effort to install GitHub or GitLab on premises. See for example GitLab Reference Architectures for the official GitLab reference architectures, expected infrastructure, and servers you need to maintain.

2. It is still possible to work with a plain IP address. You can try to add a record on the localhost file on each developer workstation. If it is not possible, your team will have to memorize the IP address.

3. You could also use Gitea.

4. Look how often the team members held unplanned video communications. It is an indicator of how well the team interacts in a remote synchronous working configuration. The intensity of chats is an indicator of how the members interact in an asynchronous team setup.

5. Slack is probably the best chat application I know of. The second best would be Discord. Microsoft Teams is a reasonable solution if all team members work in the same company. Otherwise, the solution is quite a pain because you need to manually log out and login each time you want access to a channel of another organization.