[[calix]] it might actually turn out that self-hosted loomio is harder to use, as some people may log in to loomio for more than one community
F: current default is to not run other services
[[jamie]] it might be useful to pin these to [[user journeys]] for members of the community. Would love to join that.
[[flancian]] we can try to keep a map of project:person who is interested and notify when there’s activity.
[[flancian]][[twg]] meetings are on Mondays at 6pm UTC by default every other week, but they can be moved :)
F: organising circle, started thinking about this 2 meetings ago.
[[flancian]] AI: to talk to the organizing circle about the mapping of user journeys to ways to engage with the coop.
In general, given current TWG capacity, our position is: we shouldn’t run anything we don’t have to run. So loomio/nextcloud: better to get it hosted, like we do currently (either through the main instance of loomio or mayfirst in the case of nextcloud)
On ways forward for Mastodon
Character limits vs moving to hometown
[[calix]] bumping character limits would up the maintenance cost; but the cost of switching to hometown would be high upfront.
[[flancian]] +1, unclear what would be better for TWG members of the future if this generation disappears. Maybe moving to hometown and reducing the maintenance cost upgrade-to-upgrade would be preferable. But then again hometown would eventually ‘fade away’.
[[calix]] fair but it might be preferable to assume there will be some TWG continuity; which is likely to be the case given the number of members.
[[Evan Boehs]] there’s the issue of Hometown not being updated in the last few months.
On the character limit: looking at the codebase, if the only thing we really want is the possibility of increasing the character limit, that seems to be just one file to update.
[[Calix]] we believe it’s two files — but still not a lot of files. The issue is that making customizations to the instance would require us to build our own docker images, which would increase the work that the TWG needs to do (even for an e.g. security update, of which Mastodon has quite a few).
[[Calix]] on the path to move to Hometown: it’d mean holding in the 4.2 branch but we could still update (apply security fixes).
[[Evan Boehs]] the docker arguments makes a lot of sense. One question about that: could we just ‘hot patch’ the image? Another: glitch soc could be an alternative?
[[flancian]] will bring it up with the [[CWG]] as there was recent discussion about [[meatspace]] gatherings; this seems related. I could also see the [[FWG]] being interested in having a list of projects that can be funded for different purposes, although the focus so far was closer to supporting digital-first efforts.
what is the twgβs opinion on the feasibility of patching mastodon to increase the value of StatusLengthValidator::MAX_CHARS
[[flancian]] if we apply a patch, or modify the instance in any way, the next step would be to set up alpha.social.coop. A good starter task for a new member ;)
next action: give it a stab / share if we get something working, maybe a python/requests program?
the spam might have been trying to ‘make a point’ this time
it seemed quite targeted/singled out certain users
we could have a mastodon release announcement bot in matrix?
maybe there’s a release bot for github CI?
single-sign-on
there’s interest in SSO in the organizing circle
F: great place for us to get a feel of what other working groups need. requests may naturally flow through org. circle, instead of needing full-graph connectivity between working groups
F: next steps? auth.social.coop working, open. test logging in as admin (deets in pass)
F: invite link could work well for new users. currently we send invite link (CWG) by email which isn’t really secure. could replace signup screen with hardcoded "reach out to admins". low-tech solution. CWG also brought up that it would be nice if some part of the wiki could only be edited by people in a working group.
E: the organizing circle was asking about the calendar, and how to restrict editing to some people.
Our position: we prefer [[trust but verify]]. Meaning: we’ll check ‘is in social coop’ programmatically; but everything else is up to group trust/human protocols.
F: attended first meeting, missed second, hoping to hit next one. carried question of running more services. want to get a sense from community about what folks are interested in having.
shared https://anagora.org/twg+2024 with them, suggestion to poll the community for which services they would like to see take shape
Andrew: this should be bidirectional, a shared discussion
More generally the interaction between working groups must be n-directional
Andrew is interested in the interface with the ‘real world’ of laws and finance
What is the scope of this? What should we call it?
F: "goalsetting" sounds good. but what about new people? "call for contributors" / "call for feedback". "open day"
C: really like "open day" / "open house"
C: Shoot for March deadline for poll, mid-late April for meeting?
F: align priorities in next month. come up with tasks. community call: these are the tasks up for grabs. make it concrete for people to see how they could contribute.
F: even though migration to SSO seems like a bit of would be cool to write docs on it to benefit the federation.
other services/more integrations?
this seems like it could be a good cooperative effort with [[coop cloud]]?
F: would love to run new services on [[coop cloud]]. examples of recipes where you can optionally compose containers. ideally would love to base mastodon SSO on keycloak. what else? people have been asking for other services. could do a poll?
C: other platforms like [[cloudron]][[sandstorm]] have a little more integration between apps but not much, would also be a great opportunity for documentation / collaboration. gaps: mastodon-to-archive, mastodon-to-other-instance, matrix on top of SSO.
F: [[beeper]] interactions — messaging as the next social platform. pidgin, libpurple, moa.party (need to pay Elon Taxβ’ tho — although could be an option for smaller communities).
F: anti-fedipact position is based on wanting to help users get out. corporation which interops is offering ways to defeat walled gardens. give users an option.
related communication on the instance w/[[calix]], [[ntnsndr]], [[flancian]] et al
[[meta]] it is quite a bit better than loomio for this kind of conversation…
C: TWG could work on importing Twitter / Facebook archives into mastodon, on behalf of social.coop.
F: Like the bidirectionality where it encourages companies to add interoperability. Also usability of logging in is better than e.g. Twitter Takeout.
Let’s try to do both? :D
F: Moderation and federation actions should be federated within activitypub, multi-software. Federation within groups not just within instances. Threads is the next iteration of scaling. Maybe reasonable to model as sub-networks; reasonable people, fascists — saying we won’t federate with fascists. A lot of the work of fediverse is banning fascists. That needs to be automated; that’s a lot of work.
C: this seems like something worth exploring; it seems like it would require support by threads.net and/or other large instances, like for example mastodon.social. Also have hope that we can achieve this on a cultural and technical level to make it easier to host smaller servers than large monolithic ones overall (vs current reality where it’s kind of the opposite).
F: If fediverse ends up being decentralised; improving the toolchain so people can mostly be on small instances, that would be the best outcome. We should hedge and try and do both. As long as there’s a motivation under capitalism to centralise it will continue to happen. In 5-10 years there will be new platforms, formats, which could again see capitalist entities trying to centralise for profit. Fediverse could be more resilient. Idea: write an addendum to Fedipact which says "this is how Threads could federate". FRom a point of view of the company, harder to push back on specific technical demands.
C: they certainly find it easy to push on ethical arguments; trying to get them on a technical argument might make sense.
F: Hedging, we can see which of the toolchains / approaches we invest most in, happy to start with 50:50. Personally wany to try the corporation negotiation approach. Would take any chance to push FB in the right direction. Also full respect to people who say the safest thing is to not federate for now. Happy to help on both fronts.
C: +1 — we talked about this back in September. Would suggest:
Plan A: we do a one-off date poll for a [[social coop tech working group goalsetting]] session. People commit to attending that; then we discuss further engagements there.
Plan B: we work on a roadmap ourselves and we announce the roadmap at the same time as the poll.
A: would like to do SSO sooner vs later, then fewer people will need to go through migration steps
E: Concerns about doing things just because we can (unless we’re tightening up things we’re already doing), e.g. bringing new services online — does someone other than us want them?
A: there has been interest in SSO, as well as some interest in Hometown
A: Hometown has been stuck on v4.0.x so it’s of some concern
C: another concern is that there isn’t a docker setup, so we would have to build our own, or move away from Docker at social.coop and lean more on Ansible
C: do we want to add something to our roadmap about the Organizing Circle, to help us to communicate with other working groups better? e.g. how we rotate our representation there, what we should tell them, what we should ask
C: SSO does seem like a priority, for users to be able edit wiki they need to ask for a SSO account
E: Agree, it’s a little better than what we had before, but not good that it’s still gate-kept by the extra signin. Is the question whether to use SSO for social.coop mastodon?
A: Yes. wiki already uses it. Loomio we can’t unless we selfhost. So just mastodon. Saw a similar migration done on another instance, was a bit of work.
A: we had good communication channels with our users via email about the change, and letting them know the change to the login
C: I’ve only set up SSO for hometown, not sure how to hide the login button but it is important
C: You can pay Loomio more money to get SSO working. Agreed communication would be key, we would frustrate and potentially lose users if we didn’t communicate properly.
C: We did talk about migrating passwords, but we established that we couldn’t because mastodon and keycloak use a different hashing algorithm
F: backups and restores are high priority still
F: we could extract email addresses from Mastodon, email them to create a new username and password, and give them a month, and then have a cutover. This would allow us to handwave migration.
C: if there’s an ordering to the roadmap we could have SSO first; I think it would be confusing if you could login to mastodon with either approach (local vs sso).
F: agree
A: we also did not have both, it was switched over on a particular day
F: we can generate a CSV of usernames and generate accounts for them, and then email them links to complete the setup
We should probably use a test instance to verify SSO/Keycloak is working with Mastodon.
C: might be an opportunity to better integrate Ansible and coop.cloud tooling (abra)?
C: switching to hometown will need additionally planning since downgrading would be difficult/tricky ; we would need to slowdown updates to social.coop to let it catch up
F: we should leave in the roadmap in case people want to discuss, but not a priority
C: probably should put behind the wiki and SSO; we also should prioritize organizing circle involvement
F: still is a question about Bonfire, but we can postpone discussion
C: I wanted to ask about blip to access, on Sunday Oct 1. I didn’t get around to seeing if datadog would be helpful to check what happened, also it’s not clear if we are getting alerts about availability. I could ssh to it at the time.
A: I did look at it and couldn’t find any evidence; the Datadog credentials are in the password store. We don’t monitor a bunch of things, and maybe we should talk about that. Should we include Datadog/Grafana plans as part of the roadmap?
C: at coop.cloud we were using uptimerobot for a bit; we are doing self-monitoring, uptime-kuma at the moment; even signing up for a free service like pingdom.
F: we should get it (monitoring) into the roadmap and make a request
[[flancian]] attends the first few meetings as representative if nobody else volunteers (by default) :)
we rotate as people want/can, [[Calix]] volunteered whenever time frees up :)
Issue of attendance
Low attendance to these meetings as of late
Common after summer breaks, but we want to make sure we keep critical mass going forward
Idea: start writing a roadmap document with possible next steps for the group, good opportunity to converge on plans and get them reviewed by other working groups and the social.coop community
Wiki next steps
We need to set up and test backup/restores
[[Flancian]] tested backups for the wiki, restores come next :)
first finding: abra app backup auth.social.coop worked as expected I think — it left me with a (quite small) auth_social_coop_db tarfile in ~/.abra/backups. this is nice :)
second finding: abra app backup wiki.social.coop says ‘no backup configs discovered for wiki.social.coop’ — maybe the recipe doesn’t support this yet?
Fl: Yes! Intended to close it down, so we don’t have spam issues. Now is a good time.
Ed: Did we want self-registration off?
Fl: Another member was surprised it was on. "What we want" is the right topic. How to integrate with registration form? How could you allow people to "sign in with X"?
Ak: Better in the long-run to go the other way - logging in with Keycloak into mastodon.
[[akshay]] ideally we’d run the instance in mastodon.social.coop and do "domain magic" to have accounts still be @social.coop, but that ship has sailed
maybe let’s try once we run bonfire or something else?
Maybe they’re reusing the API key or they’re making a lot of requests for the web interface?
And Mastodon really should handle this better, but it doesn’t — maybe all the other instances are backed by multiple servers and load-balance so the user takes way longer to hit this condition
Q: Is there any way to have per-user request logs in Mastodon?
A: no, there is not — but we could change mastodon_log_level in roles/social.coop/defaults/main.yml to debug (it’s info)
Next action (Flancian):
report back to the user, tell them what we found
ask optionally for an IP address so we can skip nginx logs to check how many requests the server is actually handling for them
I still need to test the latest changes in wiki-alpha; I will try to do that before the meeting tonight, or else we could do it live (tm) :)
if those work we’d be ready to set up wiki.social.coop using the same stack/steps. the only remaining step at that point would be to either special case the registration form path within wiki or (my preferred option) move the form to e.g. join.social.coop out of wiki.social.coop, which IMHO would make more sense?
I still need to test the latest changes in wiki-alpha; I will try to do that before the meeting tonight, or else we could do it live (tm) :)
if those work we’d be ready to set up wiki.social.coop using the same stack/steps. the only remaining step at that point would be to either special case the registration form path within wiki or (my preferred option) move the form to e.g. join.social.coop out of wiki.social.coop, which IMHO would make more sense?
Check that certbot is actually running in rhizome successfully if we didn’t do it back in February.
The certificate expires [[2023-04-17]] so there’s plenty of time, but certbot is not running — certbot.service expects it in /usr/bin/certbot but it’s installed in /usr/local/bin/certbot
The certificate expires [[2023-04-17]] so there’s plenty of time, but certbot is not running — certbot.service expects it in /usr/bin/certbot but it’s installed in /usr/local/bin/certbot
Port wiki-dev to our shiny new coop cloud mediawiki instance wiki-alpha see issue for details.
[[Flancian]] I discussed migration paths with the [[wiki builders]] chat and the easiest way forward is probably to declare maintenance for our wiki (including registrations?), bring it down and bring up mediawiki on wiki.social.coop during that window.
Eduardo: originally from Argentina (near Buenos Aires), now in ZΓΌrich
Akshay: originally from India, now in Berlin
Tod: based in Utah
Cleanup:
[runko] Run certbot commands to revoke the previous certificates.
Not needed, actually could be harmful as we copied the certificates.
Instead we should check that certbot is successfully renewing, moved to future agenda.
check that backups work
They don’t :)
Trying to fix it now, got all the way to setting up timer and fixing pg-dump-to-s3.service (prune was failing as we have a new server which means a new path in s3).
Open question is whether we need the assets:precompile step or, as the release notes seem to imply, we can skip because we’re using the prebuilt images?
We removed the precompile in the end.
Done!
Always run --check -vv --diff before going live
This time it prevented us from having another DEEPL outage because of a mismatched collection version in [[ansible galaxy]]
Issues we ran into
Had to run export PASSWORD_STORE_DIR=~/.password-store/social.coop (this depends on where you put your password store)
Had to run ansible-galaxy collection install -r requirements.yml in ansible directory
Tod: what about mediawiki migration?
Next steps
Make sure that [[coop cloud]] is running successfully in server [[hypha]]
We agreed on 2022-01-23 to schedule a maintenance window to test the database restore on Hypha (our new Hetzner server). If all goes well with the restore and testing we may decide to leave it running there.
includes certbot and certbox specific nginx config.
moved datadog password from ansible vault to pass store, lookup('passwordstore'...). we can use this for everything, instead of needing 2 systems
social.coop/defaults/main.yml is where a lot of the parameters for our configuration now are, including everything mastodon (database, s3 buckets, etc.)
replaced ‘docker-compose’ with ‘docker compose’ invocations, as the earlier is deprecated/removed in new versions.
added datadog logs parameters to docker-compose, but they aren’t used by default (require further configuration)
created an env file for postgres (to pass the password)
a certificate is generated first, that is needed so nginx can start; this runs once as it’s set up to check for existence of a file
whether to use secrets the way the wiki has them in the rhizome playbook, or the other way (which risks people using prod secrets in their test environments)
whether to use PASSWORD_STORE_DIR or include a social.coop/ prefix for every pass.
how to do the actual migration to rhizome once it’s set up.
[[calix]] would any of these changes affect the mastodon upgrade process?
[[akshay]] we’d need to automate some of the migration steps; currently docker compose config doesn’t include this.
when we restart mastodon, we restart everything — including elastic search, postgres. we need a more granular way to restart things.
We wouldn’t be ‘locked in’ based on the recipes — and the recipes are high quality and we could contribute back!
[[calix]] in autonomic we were using ansible for provisioning services and users and using coop cloud for app deployment and maintenance, but there is a project to move away from the abra cli even: [[wiki cafe]] ~ https://wiki.cafe
[[calix]] how much do we need to consider the "beach factor" (how many people does it take to go to the beach before you forget how to do things) when thinking about the approach. there’s been some turn over in TWG people, and knowledge transfer has proven difficult.
David: running an experiment with the abra CLI should be straightforward; biggest issue was how abra installs everything in a single machine.
Attending: Dan, David Vasandani, Akshay, Eduardo (flancian), Ed (edsu), LibreEquity, Calix (3wordchant)
Dan: upstate New York
(round of introductions)
Check ins
Dan: wrote a draft on Mastodon Verification for the publication they work with. [[rel equals me]].
Akshay: been away for >1mo, was in vacation (to India!) and travelling (now in Italy!). One more week of holiday left! May have time to dedicate to social.coop.
Ed: been kind of sick / with the flu. Otherwise going well and into vacation mode until EOY! Open to do coworking.
Nextcloud calendar for coworking?
LibreEquity: doing well, very interested in the Twitter -> Mastodon move. Read Nathan Schneider’s book in 2014-2015, followed him back then and had this in the back of our mind.
David: life is good, kids are healthy :) Getting warmer (in US). Diving into monitoring. Grafana, using docker, learnt a lot.
Calix: doing alright :) Neat people starting at autonomic.
Eduardo: entering production freeze, feeling nice about it :) also re-re-finishing (THIS TIME FOR REAL (tm)) [[agora chapter]].
Hacked around for the last few weeks. Last experience with monitoring/debugging/logging was a few years back. Main intent was to explore modern options in the monitoring/debugging/logging space.
Loki: tool for logging.
Taskfile is like a Makefile but in YAML.
Prometheus: Borgmon-like :) Gathers data.
Grafana: UI layer? Frontend for Prometheus.
docker-compose files for the whole stack. Will add this to a repo in git.coop.
statsd, fluentd.
Open question: homegrown docker-compose long time or [[coop cloud]]/[[abra]]? Is this a choice or are they complementary?
Akshay: use the same stack pretty much at work. You can configure loki to store logs somewhere — like an S3 bucket — (and that way you do not need extra indexing infra?).
David: we pipe things through fluentd to consolidate logs.
David: [[datadog]] can get expensive with one wrong logging statement :)
Flancian: is this the kind of stack that we could run in Rhizome now?
Akshay: would recommend we run monitoring in a separate server, because.
Ed: is the goal of this to be able to detect emergencies?
David: Grafana supports setting up alerting.
Akshay: Alert Manager is provided by Prometheus as well as an alternative.
Next steps?
Flancian: could we install this monitoring stack in [[hypha]]? ;)
Seemingly yes, seems like a good idea.
David: any concerns with network traffic? Would like to have someone else try it out, and then work on deploying it. We would need something installed at the system level and/or docker level (on runko)
David: Decide if we want SSO, and if we want a stand alone SSO.
David: [[keycloak]] might provide a way to have a separate SSO
Akshay: have some friends who migrated from Mastodon to Keycloak :)
Calix: would like to pick your brain sometime :) Were password resets required?
Akshay: no.
Calix: screensharing.
Issue with the sign in dialog in Hometown/Mastodon when two login providers are enabled.
We should go all-in, no dual login.
Flancian: question about the issue we’re trying to work around.
Patch is one line to add the email field in responses, could be mounted as a volume patch in docker-compose.
But it would increase maintenance.
Calix: note that dokuwiki doesn’t use this email field except for sending notifications, could we poll folks on preferences between disabling that vs. login system churn.
It might indeed make sense to patch dokuwiki instead of patching xMastodon, given that the earlier is less critical (and a newer service).
David: does this maybe depend on which other services we want to run?
Calix: yes, not only the number but also whether they use the email field the same way, or if they are based on username.
LibreEquity: is social.coop running an email server?
Flancian: nope, using webarchitechts forwarders set up on request
NEXT EPISODE:
Nextcloud access for other [[working groups]] (flancian)
Possible default policy: we batch create accounts for all WG members proactively and DM them their passwords? ideally over Matrix
Experiments with coopcloud.tech (protean)
Chat log dump follows:
Dan says:I like jitsi better than bbb
David Vasandani says:( * ^ *) γγ·
calix (they/them) says:dang this is a pleasantly large group! π₯°
me says:
https://doc.anagora.org/social-coop-tech-group?edit for notes π
Dan says:I’m on Firefox!
Dan says:(I use the Nightly build)
calix (they/them) says:it’s -9Β°C here today π₯Ά
Dan says:wow that is v cold!
calix (they/them) says:π
David Vasandani says:π
Dan says:a previous Mastodon article I helped out with
https://themarkup.org/the-breakdown/2022/11/21/we-joined-mastodon-heres-what-we-learned-about-privacy-and-security
calix (they/them) says:
https://git.coopcloud.tech/coop-cloud/monitoring
calix (they/them) says:
https://git.coopcloud.tech/coop-cloud/monitoring-lite
calix (they/them) says:think it’s the same stack?
David Vasandani says:@calix it does look to be the same!
David Vasandani says:I believe the only difference is I put fluentbit between the service and loki.
calix (they/them) says:π
calix (they/them) says:π
me says:
rhizome.social.coop
edsu says:π
Dan says:unrelated but I have some affiliation with an NYC org
rhizome.org
LibreEquity says:RAID0?
edsu says:brb
edsu says:Here’s an issue where I wrote down some notes on the dokuwiki/mastodon sso experiment:
https://git.coop/social.coop/tech/operations/-/issues/53
Dan says:"hey are you talking about mastodon?" βmy 3yo
calix (they/them) says:π€£
edsu says:lol
LibreEquity says:π
edsu says:π
LibreEquity says:π
David Vasandani says:Great point!
edsu says:I disabled that dual login in the dokuwiki — agree
David Vasandani says:Thank you for that call out.
edsu says:π―
calix (they/them) says:oh and for completeness we hid the username/password fields in CSS π΅
calix (they/them) says:
https://github.com/cosmocode/dokuwiki-plugin-oauthdoorkeeper/compare/master…edsu:dokuwiki-plugin-oauthdoorkeeper:mastodon
David Vasandani says:Never be sorry!
David Vasandani says:Not everyone has an alias.
David Vasandani says:It by request.
running bonfire community; some upstream issues w/ bonfire (e.g. mail + mailgun)
good support in bonfire chat
goal is to find something that has more governance built into the same plaform; we have difficulty getting everyone on all the platforms (loomio, matrix, git, etc) would be good to have governance folded in.
there doesn’t appear to be voting built in, but since it is modular it should support writing an extension
if anybody wants to take an account feel free ; just look for confirmation link in your spam
@davidvasandani just created an account but didn’t get an email.
@protean shared and link and @davidvasandani was able to access.
participation rate is like 15%-30% of people who are loomio, and people who are on loomio is 50% of active mastodon
did a poll for translation feature, some people said thank you for doing it that way
jotaemmei: sometimes I don’t vote because I see that something is going to pass
if loomio is asking people to jump through too many hoops we should re-evaluate
how do we make sure we accomodate people’s attentional needs
this is a larger issue with the larger confederation of co-ops, but it is an issue even internal with social.coop
some people have signed up to do bridging work, but it’s unclear how to manage it going forward
[[flancian]] should we reboot runko at some point? :)
[ ] Yes we should! let’s schedule that for Friday 2022-12-10
- follow up on issues in [bug tracker](https://git.coop/social.coop/tech/operations/-/issues)
- [#52](https://git.coop/social.coop/tech/operations/-/issues/52) - Datadog Agent
- Do we have an account? Yes, info is in the pass.
- Plan size?
- Access?
- protean: there was no systemd service to start datadog,
- got it working again, and added logging
- not connecting to redis and elasticsearch because it’s not running in those docker containers
- @davidvasandani can help configure if we’d like to mornitor.
- we need to decide whether we want to keep using it, someone mentioned prometheus?
- @davidvasandani is testing Prom, Grafana, and Loki locally for monitoring Mastodon.
- We can discuss the pros and cons of build vs buy and what our objectives are without getting too technical.
- [#56](https://git.coop/social.coop/tech/operations/-/issues/56) - Experiments Server
- Currently setup on Hetzner (hypha)
- Is there interest in full E2E automation and testing of new versions and restoring from backups?
- Does the group use or have experience with Terraform?
- If there's interest @davidvasandani would like to run point on creating the automation and running some pairing sessions.
- [#53](https://git.coop/social.coop/tech/operations/-/issues/53) - Setup Dokuwiki
- Test setup on hypha? (Hetzner experiments server)
- Has someone picked up this ticket? Can @davidvasandani pick this up? (might want to touch base with @flancian)
new server!
we have the go ahead to use up to $250 per month
should select a server and put to a quick loomio checkin/vote
- protean: opened the log retention issue on loomio
- https://www.loomio.com/d/kXMdf0ZW/privacy-policy-log-retention-and-anonymization
- jota: what do we have now for a privacy policy?
- protean: it links to the code of conduct at the moment
- jota: there are questions of jurisdiction that I was hoping the legal working group could help address
- protean: i want the lawyer people to provide guidance here
- protean: i posted a draft text about what we are currently doing
- protean: there is .env configuration to scrub login information we could think about doing
- david: would there be interest in moving to using something like terraform for provisioning the Hetzner machines? that way we could bring up experiments and tear them down easily.
- there was general agreement that this sounded like a worthwhile thing to look at
- david is going to set up a time for the TWG to discuss how we want to evolve our infrastructure code
- schedule and announce the next meeting time
- moving to bi-weekly meetings, that meet a bit later for Bo, and hopefully are still doable by flancian?
- re: AWS/Azure 501c3 grants
- social.coop is not a 501c3 and the overhead of creating one or using someone elses didn’t seem feasible
these are not public to the internet — they are just on the ‘external’ docker network
[[david vasandani]] should we set up a separate meeting to talk about tech roadmap level things, or use this meeting?
this meeting should be a good forum.
[[ian]] also a lot of this discussion is advancing well in the matrix chat rooms.
[[edsu]] a lot of these are bound up with what social.coop (the larger group) wants to do, we should take care not to over-focus on the tech aspects only
[[david]] thinking mostly of incremental improvements to our current stack
[[david vasandani]] what’s the advantage of these experiments w.r.t. reliability? also, two servers might not affect reliability in the positive direction.
(…)
[[ian]] on the parallels between the fediverse now and email 20+ years ago. Things got hard with running your own email because of spam. There’s no reason to believe that this won’t happen with the fediverse, if we just let it go. People will start to demand more moderation policies, if people don’t want to be on your instance because people aren’t federating with you, there will be re-centralization.
on Ian’s experiment: if we’re going to combat the previous, we need some way of maintaining the distributed aspects of the network. This requires instances to agree on codes of conduct, standards for handling signups.
aside: could social.coop scale up to ~50k people with loomio?
-> we need an instance that can self-govern.
[[dphiffer]] proposed to union local that they run an instance.
but what does it mean for one coop to be a member of another coop?
[[ian]] it might or might not make sense depending on how the instances operate; for example higher level coops could vote on higher level rules on how associated instances federate.
lack of good tools for content moderation is a problem.
the fediverse currently has only hashtags to have distributed decision making on e.g. blocking
a coalition of federated instances could do better; they could agree for example
[[flancian]] we could also imagine extending the ActivityPub protocol to support sharing this information between instances; this is something the fediverse needs to solve if it’s going to scale
@Flancian: Happy after weekend playing with containers. Way fewer (late) meetings because of thanksgiving.
@Edsu: starting week happily.
@Protean: working on not taking down the site today ;)
@3wordchant / @3wc / Calix: doing well, happy to have an individual account in social.coop now :) ran in the rain yesterday (pleasantly) and have had an interesting day so far (ethical wise).
sudo touch /opt/social.coop/var/www/maintenance_mode_on
sudo docker-compose stop web streaming sidekiq sidekiq-scheduler sidekiq-default-q sidekiq-push-q sidekiq-pull-q es
sudo systemctl start pg-dump-to-s3.service
sudo docker-compose exec db vacuumdb -U postgres --all --full --analyze --verbose
sudo systemctl start pg-dump-to-s3.service
# validate that the backup is in the bucket and that we are able to retrieve it, otherwise move manually to [[hypha]]
sudo docker-compose exec db pg_dumpall -U postgres > ../../var/lib/postgresql/9.6.backup
sudo docker-compose down --remove-orphans
sudo git checkout 3_5_upgrade
sudo docker-compose up -d db
sudo cat ../../var/lib/postgresql/9.6.backup | docker-compose exec -T db psql -U postgres
sudo docker-compose logs -t db
sudo docker-compose run --rm -e SKIP_POST_DEPLOYMENT_MIGRATIONS=true web rails db:migrate
sudo docker-compose down --remove-orphans
sudo docker-compose up --scale sidekiq-pull-q=4 --scale sidekiq-default-q=12 --scale sidekiq-push-q=4 -d
sudo docker-compose run --rm web bin/tootctl cache clear
sudo docker-compose run --rm web rails db:migrate
sudo docker-compose down --remove-orphans
sudo docker-compose up --scale sidekiq-pull-q=4 --scale sidekiq-default-q=12 --scale sidekiq-push-q=4 -d
Done
==ROLLBACK==
sudo docker-compose down --remove-orphans
git checkout rebuild
sudo docker-compose up --scale sidekiq-pull-q=4 --scale sidekiq-default-q=12 --scale sidekiq-push-q=4 -d
option one: consolidate (move) twg.wiki (tech/operations repo), cwg.wiki (community/operations) into general.wiki, which gets processed and deployed from ansible?
option two: change [[ansible playbook]] to also check out the twg and cwg wikis and somehow integrate them at deployment step
pros/cons of these approaches
do we want to keep anything in the wiki secret?
[[ian]] there might be a grey area for things like management interfaces (URLs), but by and large the right default option is public
things that keep your logs clean
some documentation lives with the ansible repo as well
any pros to option two?
cons: makes linking between repos more difficult, [[edsu]]: it’s nice to think of it as one knowledge base
[[operations]] source of truth for our tech group work
nine repositories currently, counting three wiki
[[flancian]]: I’d like to designate one repository as ‘clone this first’ and add a ./bootstrap-local.sh script which clones all social.coop repos to begin with.
[[flancian]]: what’s the best repo for that? is it [[ansible]] because of how we want to use automation for this, in particular for ./bootstrap-prod.sh?
flancian: originally thought tech-operations but maybe ansible?
[[protean]] ~ [[ian]]: as he was doing the documentation revamp he leaned towards not putting everything in ansible
the setup we have may be partly influenced by the lack of group wikis in gitlab at the time of design
question: when you want to use the ansible repo, the way it’s intended to be used:
you have your ssh key in your local machine that you’re working on
you clone the ansible repo
you run the playbook locally and that connects and alters the server
we could make a playbook for things currently in the sauce
[[ian]] if we want to go to the server and clone something there, that workflow (which comes more natural to some) just doesn’t fit the above a lot
unclear if there is a big advantage of moving [[docker compose]] stuff to [[ansible]], it’s mostly one file
[[ansible repo]] should indeed install docker, docker compose though
[[flancian]] ansible did feel like a learning curve when starting
[[edsu]] should we use [[ansible]] to do upgrades? maybe we could tackle [[mastodon upgrade]] this way and learn from that?
[[ian]] even at work, ansible manages the stages from bare metal to deployment ready.
backup restores as a prereq for mastodon upgrade process? or is that too conservative?
second server for testing backup/restore
[[flancian]]: not comfortable doing upgrade without having done the restore; at some point we are going to have a problem and it will take us a few days to get back up; how was this done previously?
[[protean]]: when we moved from [[trunk]] (digitalocean vps) to [[runko]] (hetzner colocated) we had two servers
might be able to either
A. do a test restore in a droplet
B. run two compose instances on [[runko]], on different ports, there is a risk if you mess up.
π Node [[social coop tech group]]
π social coop tech group.md by @flancian οΈπ βοΈ
$ tootctl accounts create <user> --reattach --force --email=<email>docker exec -it docker-web-1 /bin/bashin [[rhizome]]π social-coop-tech-group.md by @anonymous@doc.anagora.org βοΈ
2024-04-13 — Tech Working Group Open House!
Here: [[Calix]], [[flancian]], Melissa (ansate), Jamie, Caitlin
2024-03-25
Here: [[Calix]] [[flancian]]
2024-02-27
Here: [[Calix]] [[edsu]] [[flancian]]
tootctlhave something?pass)2024-02-13
Here: [[flancian]] [[Calix]] [[andrewe]]
2023-12-18 .. [[2024-01-01]] .. [[2024-01-15]]
Here: [[flancian]] [[Calix]] :)
Wish you all a great holiday season! Some topics:
2023-12-04
Here: [[edsu]], [[Flancian]]
2023-10-09
Here: [[Akshay]], [[edsu]]I’ll b, [[Calix]], [[Flancian]]
[[2023-09-26]]
Here: [[Akshay]] [[Flancian]] [[Calix]]
Mastodon upgrade party!
[[2023-09-25]]
Here: [[Akshay]] [[Flancian]]
[[2023-09-11]]
Here: [[Flancian]]
[[2023-08-28]]
Here: [[Calix]] [[Flancian]]
[[2023-08-14]]
Here: [[Flancian]]
abra app backup.abra app backupworks for wiki.social.coop \o/[[2023-07-17]]
Here: [[Akshay]] [[Flancian]] [[Calix]]
[[2023-07-05]]
Here: [[Akshay]] [[Ed Summers]] [[Flancian]]
[[2023-06-19]]
Here: Akshay, Flancian, 3wc/calix
[[2023-06-05]]
Here: edsu, Akshay, Flancian, (a late Calix)
[[2023-05-22]]
Here: Akshay, Eduardo, Ed …
[[2023-05-15]]
Here: Akshay, Eduardo, …
[[2023-05-08]]
Here: Eduardo, …
[[2023-03-27]]
Here: Eduardo, Akskhay, Calix
[[2023-03-13]]
Here: Eduardo, Akskhay, Calix
[[2023-02-27]]
[[2023-02-13]]
--check -vv --diffbefore going liveexport PASSWORD_STORE_DIR=~/.password-store/social.coop(this depends on where you put your password store)ansible-galaxy collection install -r requirements.ymlin ansible directoryLocalSettings.phpfile)[[2023-01-30]]
We agreed on 2022-01-23 to schedule a maintenance window to test the database restore on Hypha (our new Hetzner server). If all goes well with the restore and testing we may decide to leave it running there.
This checklist was copied from a comment that Akshay put in a GitCoop issue.
Arecord forsocial.coopto something like 300.systemctl start pg-dump-to-s3.serviceArecord forsocial.coopto point to rhizome, make sure TTL is small to help us revert if needed.Cleanup:
log
[[2023-01-23]]
sneakers the ratis working on this[[2023-01-09]]
lookup('passwordstore'...). we can use this for everything, instead of needing 2 systemsgit clone --recurse-submodules)[[2022-12-19]]
addgroup wheel[[2022-12-05]]
@davidvasandani just created an account but didn’t get an email.[ ] Yes we should! let’s schedule that for Friday 2022-12-10 - follow up on issues in [bug tracker](https://git.coop/social.coop/tech/operations/-/issues) - [#52](https://git.coop/social.coop/tech/operations/-/issues/52) - Datadog Agent - Do we have an account? Yes, info is in the pass. - Plan size? - Access? - protean: there was no systemd service to start datadog, - got it working again, and added logging - not connecting to redis and elasticsearch because it’s not running in those docker containers - @davidvasandani can help configure if we’d like to mornitor.
we have the go ahead to use up to $250 per month
should select a server and put to a quick loomio checkin/vote - protean: opened the log retention issue on loomio - https://www.loomio.com/d/kXMdf0ZW/privacy-policy-log-retention-and-anonymization - jota: what do we have now for a privacy policy? - protean: it links to the code of conduct at the moment - jota: there are questions of jurisdiction that I was hoping the legal working group could help address - protean: i want the lawyer people to provide guidance here - protean: i posted a draft text about what we are currently doing - protean: there is .env configuration to scrub login information we could think about doing - david: would there be interest in moving to using something like terraform for provisioning the Hetzner machines? that way we could bring up experiments and tear them down easily. - there was general agreement that this sounded like a worthwhile thing to look at - david is going to set up a time for the TWG to discuss how we want to evolve our infrastructure code - schedule and announce the next meeting time - moving to bi-weekly meetings, that meet a bit later for Bo, and hopefully are still doable by flancian? - re: AWS/Azure 501c3 grants - social.coop is not a 501c3 and the overhead of creating one or using someone elses didn’t seem feasible
[[2022-11-28]]
[[2022-11-21 18:00 UTC]]
Location: https://socialcoop.meet.coop/dav-y3e-c21-sgv
Attending: 3wc, edsu, protean, flancian
Mastodon upgrade!
Check in
Meta
Open questions
Steps that @protean put together for the upgrade
[[2022-11-14 19:00 UTC]]
abrawhich uses [[docker swarm]] to do [[zero downtime deploy]]Several past instances had notes only on [[loomio]]
2022-05-27
2022-05-20
π social coop tech group.md by @agora@botsin.space
π social coop tech group.md by @an_agora@twitter.com
π social coop tech group.md by @anagora@matrix.org
π social-coop-tech-group.md by @anagora@matrix.org
Loading pushes...
Rendering context...