Allow orgs to use a custom domain

[Powersource]

Like Dreams. Requested by Kiez Burn.

Blocking issues:

• Refactor logic out of dreams (into plato core?) that handles automatically adding redirect urls to keycloak. @gustavlrsn is handling this Add a function that creates an org and handles related logic plato-core#6
• Maybe we need to switch to auth0-react instead of oidc-react at this point? Did you have to switch to nextjs-auth0 to get the subdomains to work @gustavlrsn ? Try auth0-react instead of oidc-react #195 see reply below
• Switch to putting the orgSlug in a subdomain instead of the normal slug. Like dreams has it, orgname.realities.platoproject.org instead of realities.platoproject.org/orgname

[gustavlrsn]

• Maybe we need to switch to auth0-react instead of oidc-react at this point? Did you have to switch to nextjs-auth0 to get the subdomains to work @gustavlrsn ? Try auth0-react instead of oidc-react #195

Switching to nextjs-auth0 was to get it to work with server-side rendering in Next.js, so unrelated to getting subdomains to work.

[hesslau]

just for clarification regarding dreams:
[1] would we be able to use dreams.kiezburn.org as our domain?
[2] would other orgs be visible at dreams.kiezburn.org? (like at http://dreams.blivande.com/)

[aerugo]

1) Yes 2) No. That’s only because before the ability to create new orgs was live we hosted a few other orgs on the Blivande org.

…

On Tue, 16 Feb 2021 at 15:20, Henrik Hesslau ***@***.***> wrote: just for clarification: [1] would we be able to use dreams.kiezburn.org as our domain? [2] would other orgs be visible at dreams.kiezburn.org? (like at http://dreams.blivande.com/) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#210 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3MU6JFPT4NAVLFD6NOI3TS7J5KHANCNFSM4XHOIZOA> .

[DiarmaidEverycs]

Hi, I would like to talk to someone about transferring accross the data from the old realities to the new one. I have written a script that will take the Excel file and import it into the neo4j database - but it is written in Python and interacts directly with the database, so I can see why you might not want to run it directly. But maybe there is someone I can collaborate with to get the data imported? I see that there is an organisation for us already at
https://realities.platoproject.org/kiezburn/c2a99a3a-7f2f-4a82-a920-75dde65d002e

[aerugo]

Hi Diarmaid, We decided with Henrik that the safer route would be to do this manually. Our database is more complex than the one you are used to since we host multiple orgs on the same instance, sharing the same database. It will take more hours for us to adapt the scripts than it will take for you guys to copy and paste the data into a new instance, especially if you split the task between a number of people.

…

On Mon, Feb 22, 2021 at 2:43 PM Diarmaid de Burca ***@***.***> wrote: Hi, I would like to talk to someone about transferring accross the data from the old realities to the new one. I have written a script that will take the Excel file and import it into the neo4j database - but it is written in Python and interacts directly with the database, so I can see why you might not want to run it directly. But maybe there is someone I can collaborate with to get the data imported? I see that there is an organisation for us already at https://realities.platoproject.org/kiezburn/c2a99a3a-7f2f-4a82-a920-75dde65d002e — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#210 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3MU6LP2TWRXYTUUMFWMP3TAJNRLANCNFSM4XHOIZOA> .

[DiarmaidEverycs]

I was using the master branch of your realities to do my development - I am pretty sure that the database that I would have would be the same as yours - from about 2 days ago.

You define organisations via a Org node and needs via a Need node, Relationships via Relationship node and have a HAS edge between them, as far as I can tell. There are other edges as well, I would need to check my script to see what they are. NodeIDs appear to be UUIDs, for example?

I wouldn't expect you to change the script, I just need a schema to work off if you have one. Trial and error is fine when I am hosting my own instance but probably not the best when working with others.

[aerugo]

Have you tried your script with multiple orgs set up already? If you can verify that it works, that might be good enough to consider. I will leave it up to @Powersource to make the call. If you put your script on GitHub we could have a look at it. No promises though. :)

[DiarmaidEverycs]

Yeah, it works with multiple orgs - you just need to use pass in the org ID. I just need to change it so that the identification is working with the node IDs instead of the title, title works fine when there are no possible conflicts but as we have and are importing the nodeIDs, we can just use them and it should never overlap with possible needs/responsibilities which have been defined for other orgs.

Edit: On a completely different topic, google tells me that having multiple subdomains on a React app is impossible. How do you solve that?

[Powersource]

To run a migration script against production we'd first want to merge that script into this codebase in a PR. And the code would have to be js to not add another language to the codebase and increase its complexity a lot.

And about the subdomains, I don't know but Gustav got it to work somehow :P

[aerugo]

Gotcha @Powersource. It seems like we're back to manual migration.

Like I said before: You have just around 152 responsibilites in the Kiez database. If you are 4 people, it should not take you more than an afternoon to copy and paste everything. On the other hand, we need to be very careful when working on our production database. I think that if we combine the time we will spend on getting the script right, on making extra backups before running it, and on potentially fixing things that might go wrong, the number of person-hours required for doing it with code seem to surpass the number of hours to do it manually.

Honestly, this is what I would do myself for an org I ran of this size - even though I technically have access to the DB.

[aerugo]

But since you already wrote the script @DiarmaidEverycs - perhaps put it on GitHub? Perhaps we are missing something that makes it obvious to you that this is risk-free and low-hassle?

[DiarmaidEverycs]

Yeah ok, I'm not a JS programmer so the chances of it being merged into your code base are approximitely nil. Was worth a shot though.

I just hate doing things manually, if it takes me 10 hours to automate a 1 hour task, is that not time well spent :P?

But I can host the script up on Github and link it to you when I get out of work. Maybe you will find it useful - its pretty fragile at the moment.

[aerugo]

if it takes me 10 hours to automate a 1 hour task, is that not time well spent

That, my friend, is quite subjective. ;)

[Diemo-zz]

I've left the script up here if you want to look at it.
https://github.com/Diemo-zz/RealitiesTransferScript

It is pretty simple, it loads up the excel file, gets each individual sheet and loads it into a pandas dataframe. I then iterate over all the rows and match nodes up (by nodeId) to the nodes already in the database. I also add in the edges as required (Need -> Guide, Responsibility -> Guide, Org - [HAS]-> Need/Responsibility.

It has a couple of issues for running it on a production database that I can see:

1. Identification of the users is just done by email - should be ok I think, but I read somewhere about them being identified by keycloak identifier instead.
2. When it is adding in the links, it doesnt do any checks to see whether or not the link already exists. If you run it twice, it will create a second link between the nodes.
3. It doesnt do any checking to see if the nodes already belong to an Org. If you run it on two different Orgs, it will happily connect up the Needs/Responsibilities to both of them
4. It commits the cardinal sin of using a for loop with a Pandas dataframe :-O. Also it can be cleaned up (e.g. pass the driver into the functions instead of using it as a global variable)

If you think that it will be useful for you in the future, I can fix most of the above problems when I have time, but it might not be worth it. It seems to me to be very fragile at the moment. I wanted to use a framework to access the database, but the one I tried had a broken tutorial, so I gave up and just calculated the Cypher commands manually. This is, uh, not very secure. I'm also not using the graphql framework that you guys have, which seems like a waste.

Let me know if you want to see the Excel template I was working off.