How we solved a Terraform state problems
We are a group of people clustered around two main sets:
-
“mentors”: a couple of tens of deeply technical guys (as for mid-2022) that are working within Universities or other research centers since the early days of the Internet. They are mostly relatively aged Italians, loving to spend time in front of a couple of monitors plenty of green-on-black terminals.
They tipically have the “power” to remotely send a single-command… and shut-down a whole campus-network (BTW: some of them, can even shut-down a whole Country-wide research-network!). Not only ‘cause they manage the core routers, switches and other networking devices, but also ‘cause most of them operates the virtualization platforms or the storage-area-networks.
-
“students”: another couple of tens of students coming from various ICT-focused universities, really passionate for technology and eager, –very eager– to learn about the wide/deep/complex technological stack that are powering the ICT infrastructure of their universities and providing them related Internet connection. Of course, they are supported by mentors, in such a discovery.
GARRLab activities can leverage the bleeding-edge infrastructures provided by GARR –who officially support GARRLab– to foster students (…and mentors) knowledge in a wide range of ICT technologies (networking, system and network infrastructure management, cloud-based CI/CD git-assisted –aka: GitOps–, system and network security, … and lots of other things).
In short, GARRLab mission is very simple: simplify our daily work (we, technicians) and put our hands on top-notch systems and equipments (we, students)
BTW: if you’re lucky, there could even be a page with further details….
Our students are passionate individuals that love spending most of their free time learning, experimenting and, in the end, doing_damages with every kind of DIGITAL technologies (actually, some of them also spend time on analog things, like input signals for the ADC converter of some embedded systems… but this is a detail :-) )
The main group originated from the University of Salerno, but quickly expanded to other universities (UniParthenope, UniMilano, UniMilanoBicocca UniPadova )
Probably the best way to get an idea about them, is to check their presentation at GARR-Workshop on Novembre 2021, where four of them got some time to expose their findings [ slide][video]
Our mentors are passionate technicians operating the main ICT infrastructure of their university / research center, on a daily basis
They are sort-of “master” of network-admin and sys-admin (both with white-hair or no-hair at all…) and with some background in cloud-technologies.
They strive to ensure proper operation of their infrastructure, and fight against various bad-actors (both external and internal) that constantely try to introduce problems…
Some of them manager 802.1x networks connecting several thousands internal hosts; Some others constantly analyze a stream of ~5.000 events per second, coming from her/his firewall; Some other else is constantly verifying the operation of its 3 x 10Gbps GARR uplink. Someone is fighting some reflection DDoS…. And so on.
They come from different institutions, such as (in alphabetic order): CNR, UniFerrara, UniGenova, UniMessina, UniMilano, UniSiena, UniTrieste, UniUrbino, UniVerona
Thanks to official GARR endorsement, GARRLab can leverage GARR-Cloud platform, an OpenStack deployment offering IaaS services, on top of which a K8S platform have been deployed to support an additional PaaS layer.
For our projects we can rely on resources spanning two geographically distributed openstack regions (Catania and Palermo). On top of them we have been given:
- 192 virtual CPUs
- 800 GB of RAM
- 8 TB of HA storage
…and, more important:
- full-API access to automate/orchestrate our exercises :-)
We put our hands on a lot of things. Sometimes we are successful, being able to obtain something useful! Other times… we simply fail – like when we started seriously playing with HTTPS connection routing - BTW: are you able to help us? – and discover something new to learn :-)
Among the various topics, we have choose the main three that currently characterize our activities.
We strongly embraced the GitOps framework and our self-hosted Gitlab instance is really acting as the single source of truth for all our projects.
We rely on GitLab not only for software development projects, but also for documentation projects (…yes: “documentation as code”) as well as other projects that fits in the middle (archiving Ansible playbooks/roles, tracking Terraform files, etc.)
We’re heavily using Gitlab pipelines to build containers to be deployed in production. Indeed, this very website has been built exactly with this approach.
At the moment, we’re cleaning the glass towards complex-containers-scenarios, and started testing the deployment toward the K8S infrastructure provided by GARR-Cloud. Up to now, we relyed on Portainer but we started finding its git integration/support a bit limiting.
We did… and are doing… lots of things! But lots of other things are still to be done!
Remember? Most of us spend their time operating large networks. As such, there are plenty of events (aka: message-logs) generated by all the equipments and all the applications running allaround the infrastructure.
We are really committed in analyzing those streams, and we’re open to perform such task using existing technologies. Indeed, we tested Wazuh, we deployed a standard ELK stack and we spent some time tuning kibana dashboard and logstash filtering.
But we are “hackers”… We really want to “hack” the perfect solution for our needs. So we also tryied to scratch down our own engine: we’re trying to route, collect and analyze tens of thousands of log events per second, via a geographically distributed network of NodeJS based pre-processors (each one enforcing proper levels of privacy/pseudononimization) supported by our self-hosted deployment of an elasticsearch cluster. And… it looks like it works!
Some of us comes from the early days of software development. We where there, when CGI was defined. Some of us wrote CGI-applications in C. Some others have strong background with PERL and PHP.
Someone else went twice along the route of Javascript, discovering that in ~20 year, a huge transformation have been registered by Ecmascript… And now, NodeJS, VueJS, React… are something powering some projects hosted on our Gitlab.
Lots of other things are flying around, like Nix/NixOS, Go and a huge list of related things (markdown, adoc, rst, yml, toml, etc.)
All of this are used to solve our own problems… like, for example, when one of us had a long list (~11k) of IP addresses (aka: IoCs), and decided to resolve them in terms of geo-location and Autonomous System names and numbers. Half a day after… a REST web-service implemented in NodeJS was ready to receive a simple POST, easily crafted with curl. Deployed as a container, of course…. Built with a GitLab pipeline… of course :-)
Join Us! We are an OPEN community!
We welcome everyone that is directly or indirectly involved in the NREN ecosystem.
We also welcome every student desiring to [ learn | test | research | practice | ... ] whatever aspect of the so-wide set of ICT technologies around us.
Are you willing to try automating the configuration of your hundreds of network switches?
Do you want to spin-up your VM or, even better, a whole infrastructure (VMs, Networks, Firewall rules, etc.) using a declarative approach, like the one offered by Terraform?
Are you a software developer eagerly waiting to consume REST APIs offered by plenty of tools that we have around?
Are you interested in testing how many events per second are you able to “filter”, searching for bad-behaviour, in near-real-time?
Are you searching for opportunities to automate your development pipeline, to test and release your application shaped around containers?
Are you waiting to practice your knowledge, to really check if your infrastructure is really “resilient”?
You got the point… If you answered yes to at least one of the above, of even if you have a different question waiting for an answer…. join us: we’re eagerly waiting for you, to discuss with you about yours (…and ours) problems.
A final note: We don’t have money! We have zero-budget! We don’t even have a business-plan and a roadmap. We have nothing! We are here, waiting for you, just to “share” our minds, providing only a part of our free time :-)
Telegram Group
The main group entry-point [...with a human AntiSpam filter]